Merge remote-tracking branch 'upstream/7.x-3.x' into 7.x-3.x

docs/extensions.rst

@@ -8,6 +8,7 @@ The below modules are Tripal 3 compatible and grouped roughly by category:
 
    extensions/administrative
    extensions/analysis
+   extensions/breeding
    extensions/data_input
    extensions/developer
    extensions/3rd_party

docs/extensions/breeding.rst

@@ -0,0 +1,12 @@
+Breeding
+===================
+
+The following modules provide support for managing breeding data.
+
+BIMS
+--------------------------
+
+A module for an online breeding management system which allows breeders to store, manage, archive and analyze their public or private breeding program data.
+
+`Documentation <https://gitlab.com/mainlabwsu/bims/blob/master/README.md>`__
+`Repository <https://gitlab.com/mainlabwsu/bims>`__

docs/user_guide/example_genomics/func_annots/setup.rst

@@ -3,41 +3,37 @@ Module Setup
 .. note::
 
   Remember you must set the ``$DRUPAL_HOME`` environment variable if you want to cut-and-paste the commands below. See :doc:`../../install_tripal/drupal_home`
-  
-  
-For this example we will be load functional data for our gene. To do this we will use the Blast, KEGG, and InterPro extension modules. However, these extension modules are not part of the "core" Tripal package but are available as separate extensions.  Anyone may create extensions for Tripal.  These extensions are useful for genomic data and therefore are included in this tutorial. 
+
+
+For this example we will be load functional data for our gene. To do this we will use the Blast, KEGG, and InterPro extension modules. However, these extension modules are not part of the "core" Tripal package but are available as separate extensions.  Anyone may create extensions for Tripal.  These extensions are useful for genomic data and therefore are included in this tutorial.
 
 To download these modules:
 
   ::
-  
-    cd $DRUPAL_HOME    
+
+    cd $DRUPAL_HOME
     drush pm-download tripal_analysis_blast
-    drush pm-download tripal_analysis_kegg
     drush pm-download tripal_analysis_interpro
 
 Now, enable these extension modules:
 
   ::
-  
+
     drush pm-enable tripal_analysis_blast
     drush pm-enable tripal_analysis_interpro
-    drush pm-enable tripal_analysis_kegg
 
 For this example, we will use the following files which are available for downloading:
 
 - `Citrus_sinensis-orange1.1g015632m.g.iprscan.xml <http://www.gmod.org/mediawiki/images/0/0c/Citrus_sinensis-orange1.1g015632m.g.iprscan.xml>`_
-- `Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz <http://www.gmod.org/mediawiki/images/1/13/Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz>`_
 - `Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out <http://www.gmod.org/mediawiki/images/e/e8/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out>`_
 - `Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out <http://www.gmod.org/mediawiki/images/2/24/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out>`_
 
 Download these files to the ```$DRUPAL_HOME/sites/default/files``` directory. To do so quickly run these commands:
 
   ::
-  
+
     cd $DRUPAL_HOME/sites/default/files
     wget http://www.gmod.org/mediawiki/images/0/0c/Citrus_sinensis-orange1.1g015632m.g.iprscan.xml
-    wget http://www.gmod.org/mediawiki/images/1/13/Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz
     wget http://www.gmod.org/mediawiki/images/e/e8/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out
     wget http://www.gmod.org/mediawiki/images/2/24/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out
 
@@ -45,23 +41,22 @@ Each of these modules provides new fields for both the **gene** and **mRNA** con
 
 .. image:: setup1.png
 
-Next, we need to position the new field. Using the skills you learned in the :doc:`../../content_types/configuring_page_display` Create three new **Tripal Panes** named:
+Next, we need to position the new field. Using the skills you learned in the :doc:`../../content_types/configuring_page_display` Create two new **Tripal Panes** named:
 
 - Blast Results
 - Protein Domains
-- KEGG Pathways
 
 Be sure to:
 
-- Place the three new fields into each pane respectively 
-- Move the Panes out of the **disabled** section. 
+- Place the new fields into each pane respectively
+- Move the Panes out of the **disabled** section.
 - Set the label for each field to **Hidden**.
 
 The following shows an example of this layout:
 
 .. image:: setup2.png
 
-The fields are now ready for display once data is added!  
+The fields are now ready for display once data is added!
 
 .. note::
 
@@ -70,4 +65,3 @@ The fields are now ready for display once data is added!
 .. note::
 
   Anytime you install a Tripal v3 extension module you should check for new fields, and then place those fields in the layout.  Extension modules often will not do this for you because they do not assume you want these new fields.
-  

docs/user_guide/example_genomics/genomes_genes.rst

@@ -26,7 +26,7 @@ Enter the following:
 
   "File", "Upload the file name Citrus_sinensis-orange1.1g015632m.g.gff3"
   "Analysis", "Whole Genome Assembly and Annotation of Citrus sinensis"
-  "Organism", "Citrus sinensis"
+  "Existing Organism", "Citrus sinensis"
   "Landmark Type", "supercontig"
   "All other options", "leave as default"
 

docs/user_guide/example_genomics/pub_import.rst

@@ -3,7 +3,7 @@ Importing Publications
 .. note::
 
   Remember you must set the ``$DRUPAL_HOME`` environment variable if you want to cut-and-paste the commands below. See :doc:`../install_tripal/drupal_home`
-  
+
 Tripal provides an interface for automatically and manually adding publications.
 
 Manually Adding a Publication
@@ -64,7 +64,7 @@ Now, save this importer. You should see that we have one importer in the list:
 
 .. image:: pub_import.4.png
 
-We can use this importer to load all  publications related to Citrus sinensis from PubMed into our database (how to load these will be shown later). However, what if new publications are added? We would like this importer to be run monthly so that we can automatically add new publications as they become available. But we do not need to try to reload these 760 every time the loader runs each month. We will create a new importer that only finds publications within the last 30 days. To do this, click the link New Importer. Now, add the following criteria:
+We can use this importer to load all  publications related to <i>Citrus sinensis</i> from PubMed into our database (how to load these will be shown later). However, what if new publications are added? We would like this importer to be run monthly so that we can automatically add new publications as they become available. But we do not need to try to reload the same publications every time the loader runs each month. We will create a new importer that only finds publications within the last 30 days. To do this, click the link New Importer. Now, add the following criteria:
 
 .. csv-table::
   :header: "Field Name", "Value"
@@ -142,14 +142,14 @@ Import from the USDA National Agricultural Library
 The instructions for the Tripal publication importer described previously use the the NCBI PubMed database. However, you can also import publications from the USDA National Agriculture Library (AGRICOLA). However, to use this repository a few software dependences are required.  These include:
 
 - The `YAZ library <https://www.indexdata.com/resources/software/yaz/>`_
-- `PHP support for YAZ <https://www.php.net/manual/en/book.yaz.php>`_  
+- `PHP support for YAZ <https://www.php.net/manual/en/book.yaz.php>`_
 
-The following instructions are to install the necessary dependencies on an Ubuntu 18.04 LTS.   
+The following instructions are to install the necessary dependencies on an Ubuntu 18.04 LTS.
 
 First install yaz, the yaz development library and the php development library:
 
 .. code-block:: bash
- 
+
   sudo apt-get install yaz libyaz5-dev php-dev
 
 
@@ -157,10 +157,10 @@ Next update the PECL tool and install the PHP yaz library:
 
 
 .. code-block:: bash
-  
+
   sudo pecl channel-update pecl.php.net
   sudo pecl install yaz
-  
+
 Next, edit the `php.ini` files.  On Ubuntu 18.04 there are two PHP files:
 
 - `/etc/php/7.2/cli/php.ini`
@@ -175,8 +175,7 @@ Add the following line to each file:
 Finally, restart the web server so that it picks up the changes to the `php.ini` file.
 
 .. code-block:: bash
-  
+
   sudo service apache2 restart
 
 You can now import publications from Agricola using the same interface as with PubMed.
-

tripal/includes/tripal.importer.inc

@@ -80,7 +80,7 @@ function tripal_get_importer_form($form, &$form_state, $class) {
     $analyses = [];
     $analyses[''] = '';
     while ($analysis = $org_rset->fetchObject()) {
-      $analyses[$analysis->analysis_id] = "$analysis->name ($analysis->program $analysis->programversion, $analysis->sourcename)";
+      $analyses[$analysis->analysis_id] = $analysis->name;
     }
     $form['analysis_id'] = [
       '#title' => t('Analysis'),

tripal_chado/api/modules/tripal_chado.pub.api.inc

@@ -105,7 +105,7 @@ function chado_get_publication($identifiers, $options = []) {
     }
     else {
       tripal_report_error('tripal_pub_api', TRIPAL_ERROR,
-        "chado_get_publication: The dbxref identifier is not correctly formatted.",
+        "chado_get_publication: The dbxref identifier is not correctly formatted. Identifiers passed: %identifier.",
         ['%identifier' => print_r($identifiers, TRUE)]
       );
     }
@@ -1169,6 +1169,47 @@ function chado_pub_create_citation($pub) {
   return $citation;
 }
 
+/**
+ * Retrieves an array with all database cross references
+ *
+ * Implemented as SQL for performance reasons because chado_expand_var
+ * can take too long as it loads more information than needed
+ *
+ * @param $pub_id
+ *   A pub_id from the 'chado.pub' table
+ *
+ * @return
+ *   An array of records with the following keys:  'accession', 'version',
+ *   'description', 'name', 'url', 'urlprefix'.
+ *   These are the column names from the 'dbxref' and 'db' tables
+ *
+ * @ingroup tripal_pub_api
+ */
+function chado_get_pub_dbxrefs($pub_id) {
+  $fkey = 'pub_id';  // Should this be looked up in the schema?
+  $options = ['return_array' => 1];
+  $sql = "SELECT REF.accession, REF.version, REF.description, DB.name, DB.url, DB.urlprefix "
+       . "FROM {pub_dbxref} LINK "
+       . "INNER JOIN {dbxref} REF on LINK.dbxref_id = REF.dbxref_id "
+       . "INNER JOIN {db} DB on REF.db_id = DB.db_id "
+       . "WHERE LINK.$fkey = :pub_id";
+  $args = [':pub_id' => $pub_id];
+  $records = chado_query($sql, $args);
+
+  $results = [];
+  $delta = 0;
+  while($record = $records->fetchObject()) {
+    $results[$delta]['accession'] = $record->accession;
+    $results[$delta]['version'] = $record->version;
+    $results[$delta]['description'] = $record->description;
+    $results[$delta]['name'] = $record->name;
+    $results[$delta]['url'] = $record->url;
+    $results[$delta]['urlprefix'] = $record->urlprefix;
+    $delta++;
+  }
+  return $results;
+}
+
 /**
  * Retrieves the minimal information to uniquely describe any publication.
  *
@@ -1238,17 +1279,24 @@ function chado_get_minimal_pub_info($pub) {
     }
   }
 
+  // Load all database cross references.
+  $pub_dbxrefs = chado_get_pub_dbxrefs($pub->pub_id);
+
   // Get the first database cross-reference with a url.
-  $options = ['return_array' => 1];
-  $pub = chado_expand_var($pub, 'table', 'pub_dbxref', $options);
-  $dbxref = NULL;
-  if ($pub->pub_dbxref) {
-    foreach ($pub->pub_dbxref as $index => $pub_dbxref) {
-      if ($pub_dbxref->dbxref_id->db_id->urlprefix) {
-        $dbxref = $pub_dbxref->dbxref_id;
-      }
-    }
-  }
+// it is not clear what this was doing, it would have retrieved the last not the first
+// dbxref with a url, but the variable $dbxref is not referenced later. It could have
+// added information to $pub, but that does not appear to be referenced later.
+// chado_expand_var() can sometimes take a long time to execute, so just remove it?
+//  $options = ['return_array' => 1];
+//  $pub = chado_expand_var($pub, 'table', 'pub_dbxref', $options);
+//  $dbxref = NULL;
+//  if ($pub->pub_dbxref) {
+//    foreach ($pub->pub_dbxref as $index => $pub_dbxref) {
+//      if ($pub_dbxref->dbxref_id->db_id->urlprefix) {
+//        $dbxref = $pub_dbxref->dbxref_id;
+//      }
+//    }
+//  }
 
   // Get the URL.
   $values = [
@@ -1270,17 +1318,10 @@ function chado_get_minimal_pub_info($pub) {
     }
   }
 
-  // Get the list of database cross references.
-  $values = [
-    'pub_id' => $pub->pub_id,
-  ];
-  $options = [
-    'return_array' => 1,
-  ];
-  $pub_dbxrefs = chado_generate_var('pub_dbxref', $values, $options);
+  // Generate a list of database cross references formatted as "DB:accession".
   $dbxrefs = [];
   foreach ($pub_dbxrefs as $pub_dbxref) {
-    $dbxrefs[] = $pub_dbxref->dbxref_id->db_id->name . ':' . $pub_dbxref->dbxref_id->accession;
+    $dbxrefs[] = $pub_dbxref['name'] . ':' . $pub_dbxref['accession'];
   }
 
   // Get the citation.

tripal_chado/includes/TripalFields/chado_linker__prop/chado_linker__prop_widget.inc

@@ -143,6 +143,12 @@ class chado_linker__prop_widget extends ChadoFieldWidget {
     $value = $form_state['values'][$field_name]['und'][$delta]['chado-' . $field_table . '__value'];
     $form_state['values'][$field_name]['und'][$delta]['value'] = $value;
 
+    // A value of zero gets set to empty when sbumitted and some prop
+    // tables don't have a default value set (e.g. pubprop).
+    if (!$form_state['values'][$field_name]['und'][$delta]['chado-' . $field_table . '__rank']) {
+      $form_state['values'][$field_name]['und'][$delta]['chado-' . $field_table . '__rank'] = 0;
+    }
+
     // If the user removed the property then we want to clear out the other
     // fields except the pkey value. If the pkey field is present and the value
     // is present then the chado storage backend will delete the record.

tripal_chado/includes/TripalFields/data__sequence_record/data__sequence_record.inc

@@ -98,6 +98,12 @@ class data__sequence_record extends ChadoField {
     else {
       $this->addGenericReference($entity, $feature, $field_name);
     }
+
+    // The field should always return a 'value' key so even if we don't
+    // have a sequence we still need to add it
+    if (count(array_keys($entity->{$field_name}['und'])) == 0) {
+      $entity->{$field_name}['und'][0]['value'] = '';
+    }
   }
 
   /**

tripal_chado/includes/TripalFields/data__sequence_record/data__sequence_record_formatter.inc

@@ -28,10 +28,8 @@ class data__sequence_record_formatter extends ChadoFieldFormatter {
     $phase_term = chado_get_semweb_term('featureloc', 'phase');
     $fasta_defline = 'local:fasta_definition';
 
-    $content = [];
-
     foreach ($items as $delta => $item) {
-      if (empty($item['value'])) {
+      if (!$item['value']) {
         continue;
       }
 
@@ -41,24 +39,19 @@ class data__sequence_record_formatter extends ChadoFieldFormatter {
       $residues .= wordwrap($item['value'][$sequence_term], $num_bases, "<br>", TRUE);
       $residues .= '</pre>';
 
-      $content[] = [
+      $element[$delta] = [
         '#type' => 'item',
         '#title' => $item['value'][$label_term],
         '#description' => $item['value'][$description_term],
         '#markup' => $residues,
       ];
-
-
     }
 
-    if (empty($content)) {
+    if (count($element) == 0) {
       $element[0] = [
         '#type' => 'markup',
-        '#markup' => '',
+        '#markup' => 'There are no sequences for this feature.',
       ];
-      return;
     }
-
-    $element[0] = $content;
   }
 }

tripal_chado/includes/TripalFields/sbo__database_cross_reference/sbo__database_cross_reference.inc

@@ -37,6 +37,10 @@ class sbo__database_cross_reference extends ChadoField {
     // type. This will create form elements when editing the field instance
     // to allow the site admin to change the term settings above.
     'term_fixed' => FALSE,
+    // The number of items to show on a page.
+    'items_per_page' => 10,
+    // Limit to the number of items to show in cases of large number of cross references.
+    'max_items' => 10000,
   ];
 
   // The default widget for this field.
@@ -138,33 +142,39 @@ class sbo__database_cross_reference extends ChadoField {
 
     $linker_table = $base_table . '_dbxref';
     $options = ['return_array' => 1];
-    $record = chado_expand_var($record, 'table', $linker_table, $options);
-    if (property_exists($record, $linker_table) and is_array($record->$linker_table) and count($record->$linker_table) > 0) {
-      $i = 0;
-      foreach ($record->$linker_table as $index => $linker) {
-        $dbxref = $linker->dbxref_id;
-
-        // Ignore the GFF_source database. This is a weird thing required by
-        // GBrowse and is added by the GFF loader. We don't want to show it.
-        if ($dbxref->db_id->name == 'GFF_source') {
-          continue;
-        }
-
-        $URL = chado_get_dbxref_url($dbxref);
-        $entity->{$field_name}['und'][$i] = [
-          'value' => [
-            $dbname_term => $dbxref->db_id->name,
-            $accession_term => $dbxref->accession,
-            $dburl_term => $URL,
-          ],
-          'chado-' . $field_table . '__' . $pkey => $linker->$pkey,
-          'chado-' . $field_table . '__' . $fkey_lcolumn => $linker->$fkey_lcolumn->$fkey_lcolumn,
-          'chado-' . $field_table . '__dbxref_id' => $dbxref->dbxref_id,
-          'db_id' => $dbxref->db_id->db_id,
-          'accession' => $dbxref->accession,
-        ];
-        $i++;
+
+    // Build the SQL to find records associated with this publication.
+    $max_items = array_key_exists('max_items', $this->instance['settings']) ? $this->instance['settings']['max_items'] : 10000;
+    $sql = "SELECT REF.accession, DB.name, DB.urlprefix "
+         . "FROM {".$linker_table."} LINK "
+         . "INNER JOIN {dbxref} REF on LINK.dbxref_id = REF.dbxref_id "
+         . "INNER JOIN {db} DB on REF.db_id = DB.db_id "
+         . "WHERE LINK.$fkey_lcolumn = :id "
+         // Ignore the GFF_source database. This is a weird thing required by
+         // GBrowse and is added by the GFF loader. We don't want to show it.
+         . "AND NOT DB.name = 'GFF_source' "
+         . "ORDER BY REF.accession "  // if we hit the limit, the subset should be consistent
+         . "LIMIT :limit";
+    $args = [':id' => $entity->{'chado_record_id'},
+             ':limit' => $max_items + 1];
+    $records = chado_query($sql, $args);
+
+    // Store the query results
+    $delta = 0;
+    while($record = $records->fetchObject()) {
+      // Need this check to detect the case where the limit exactly equals the number of records
+      if ($delta < $max_items) {
+        $entity->{$field_name}['und'][$delta] = [];
+        $entity->{$field_name}['und'][$delta]['value'][$dbname_term] = $record->name;
+        $entity->{$field_name}['und'][$delta]['value'][$accession_term] = $record->accession;
+        $entity->{$field_name}['und'][$delta]['value'][$dburl_term] = $record->urlprefix;
       }
+      $delta++;
+    }
+    // Display a warning if we have exceeded the maximum number of cross references
+    if ( $delta > $max_items ) {
+      $entity->{$field_name}['und'][$delta]['value'][$dbname_term] = 'Note';
+      $entity->{$field_name}['und'][$delta]['value'][$accession_term] = "Only the first $max_items cross references are shown";
     }
   }
 

tripal_chado/includes/TripalFields/sbo__database_cross_reference/sbo__database_cross_reference_formatter.inc

@@ -15,6 +15,11 @@ class sbo__database_cross_reference_formatter extends ChadoFieldFormatter {
   public function view(&$element, $entity_type, $entity, $langcode, $items, $display) {
     $content = '';
 
+    // Do we have an empty list? If so, just return.
+    if (!$items[0]['value']) {
+      return;
+    }
+
     $field_name = $this->field['field_name'];
     $field_type = $this->field['type'];
     $field_table = $this->instance['settings']['chado_table'];
@@ -26,27 +31,83 @@ class sbo__database_cross_reference_formatter extends ChadoFieldFormatter {
     $accession_term = chado_get_semweb_term('dbxref', 'accession');
     $dburl_term = chado_get_semweb_term('db', 'url');
 
+    // First, organize the values by their databases.
+    $ordered_items = [];
     foreach ($items as $delta => $item) {
-      if (!$item['value']) {
-        continue;
+      $db = $item['value'][$dbname_term];
+      $accession = $item['value'][$accession_term];
+      // It is possible that a database does not have a url, in which case no link can be generated.
+      if (array_key_exists($dburl_term, $item['value']) and $item['value'][$dburl_term]) {
+        $url = $item['value'][$dburl_term];
+
+        // This emulates chado_get_dbxref_url() but implemented inline here for better performance.
+        $db_count = 0;
+        $acc_count = 0;
+        $url = preg_replace('/\{db\}/', $db, $url, -1, $acc_count);
+        $url = preg_replace('/\{accession\}/', $accession, $url, -1, $acc_count);
+
+        $content = l($accession, $url, ['attributes' => ['target' => '_blank']]);
       }
-      $content = $item['value'][$dbname_term] . ':' . $item['value'][$accession_term];
-      if ($item['value'][$dburl_term]) {
-        $dbxref = chado_get_dbxref(['dbxref_id' => $item['chado-' . $linker_table . '__dbxref_id']]);
-        $url = chado_get_dbxref_url($dbxref);
-        $content = l($content, $url, ['attributes' => ['target' => '_blank']]);
+      else {
+        $content = $accession;
       }
-      $element[$delta] = [
-        '#type' => 'markup',
-        '#markup' => $content,
-      ];
+      $ordered_items[ucfirst($db)][] = $content;
     }
 
-    if (count($element) == 0) {
-      $element[0] = [
-        '#type' => 'markup',
-        '#markup' => 'There are no cross references.',
+    // Reorder the list so it's compatible with theming a list.
+    ksort($ordered_items);
+
+    // Generate the pagers for each type.
+    $list_items = [];
+    $headers = [];
+    $rows = [];
+    foreach ($ordered_items as $type => $children) {
+      $items_per_page = array_key_exists('items_per_page', $this->instance['settings']) ? $this->instance['settings']['items_per_page'] : 10;
+      $total_records = count($children);
+      $total_pages = (int) ($total_records / $items_per_page) + 1;
+      $pelement = 0;
+      $current_page = pager_default_initialize($total_records, $items_per_page, $pelement);
+      $pager = theme('pager', [
+        'tags' => [],
+        'element' => $pelement,
+        'parameters' => [],
+        'quantity' => 5,
+      ]);
+      $pager = $this->ajaxifyPager($pager, $entity);
+      $page_items = array_chunk($children, $items_per_page);
+
+      $rows[] = [
+        [
+          'data' => ucfirst($type),
+          'header' => TRUE,
+          'width' => '20%',
+        ],
+        theme_item_list([
+          'items' => $page_items[$current_page],
+          'title' => '',
+          'type' => 'ul',
+          'attributes' => [],
+        ]) . $pager,
       ];
     }
+
+    $table = [
+      'header' => [],
+      'rows' => $rows,
+      'attributes' => [
+        'id' => 'sbo__database_cross_reference',
+        'class' => 'tripal-data-table',
+      ],
+      'sticky' => FALSE,
+      'caption' => "",
+      'colgroups' => [],
+      'empty' => 'There are no cross references.',
+    ];
+    $content = theme_table($table);
+    $element[0] = [
+      '#type' => 'markup',
+      '#markup' => $content,
+    ];
+
   }
 }

tripal_chado/includes/TripalFields/sbo__relationship/sbo__relationship.inc

@@ -433,14 +433,15 @@ class sbo__relationship extends ChadoField {
     $object_type = $entity->{$field_name}['und'][$delta]['value']['local:relationship_object']['rdfs:type'];
     $object_name = $entity->{$field_name}['und'][$delta]['value']['local:relationship_object']['schema:name'];
 
-
     // Remember the current entity could be either the subject or object!
     // Example: The genetic_marker, MARKER1 , derives from the sequence_variant, VARIANT1.
     // The above relationship will be shown both on marker and variant pages
     // and as such both subject and object names need to be shown.
+    // Keep the clause as an array for now so we can apply formatting
+    // and links in the formatter
     $clause = 'The ' . $subject_type . ', ' .
-      $subject_name . ', ' . $verb . ' ' . $rel_type_clean . ' ' .
-      $object_type . ', ' . $object_name . '.';
+        $subject_name . ', ' . $verb . ' ' . $rel_type_clean . ' ' .
+        $object_type . ', ' . $object_name . '.';
     $entity->{$field_name}['und'][$delta]['value']['SIO:000493'] = $clause;
 
     // Adding a label allows us to provide a single text value for the
@@ -524,7 +525,7 @@ class sbo__relationship extends ChadoField {
       // relationship type and the object and subject
       'include_fk' => [
         'type_id' => 1,
-          $object_id_key => [
+        $object_id_key => [
           'type_id' => 1,
           'organism_id' => 1,
         ],
@@ -645,7 +646,7 @@ class sbo__relationship extends ChadoField {
    * @return
    *   The verb to use when creating a sentence of the relationship.
    */
-  private function get_rel_verb($rel_type) {
+  public static function get_rel_verb($rel_type) {
     $rel_type_clean = lcfirst(preg_replace('/_/', ' ', $rel_type));
     $verb = '';
     switch ($rel_type_clean) {
@@ -658,6 +659,7 @@ class sbo__relationship extends ChadoField {
       case 'genome of':
       case 'part of':
         $verb = 'is a';
+        break;
       case 'position of':
       case 'sequence of':
       case 'variant of':
@@ -674,6 +676,7 @@ class sbo__relationship extends ChadoField {
       case 'is a':
       case 'is a maternal parent of':
       case 'is a paternal parent of':
+      case 'is a subproject of':
       case 'is consecutive sequence of':
       case 'maximally overlaps':
       case 'overlaps':

tripal_chado/includes/TripalFields/sbo__relationship/sbo__relationship_formatter.inc

@@ -59,6 +59,9 @@ class sbo__relationship_formatter extends ChadoFieldFormatter {
    */
   public function view(&$element, $entity_type, $entity, $langcode, $items, $display) {
 
+    // Load the Field class into scope for use below.
+    tripal_load_include_field_class('sbo__relationship');
+
     // Get the settings
     $settings = $display['settings'];
     $rows = [];
@@ -72,7 +75,11 @@ class sbo__relationship_formatter extends ChadoFieldFormatter {
       $subject_type = $item['value']['local:relationship_subject']['rdfs:type'];
       $object_name = $item['value']['local:relationship_object']['schema:name'];
       $object_type = $item['value']['local:relationship_object']['rdfs:type'];
-      $phrase = $item['value']['SIO:000493'];
+      $rel_type = $item['value']['local:relationship_type'];
+
+      // Get some better human-readable formats for the type and verb.
+      $verb = sbo__relationship::get_rel_verb($rel_type);
+      $rel_type_clean = lcfirst(preg_replace('/_/', ' ', $rel_type));
 
       // Handle some special cases.
       // For mRNA objects we don't want to show the CDS, exons, 5' UTR, etc.
@@ -87,31 +94,33 @@ class sbo__relationship_formatter extends ChadoFieldFormatter {
         continue;
       }
 
-      // Add bold font to the object and subject names.
-      // @todo add back in bolding...
-      // @todo Fix Current Bug: if type name is in the object name, weird bolding happens.
-      // $phrase = preg_replace("/$subject_type/", "<b>$subject_type</b>", $phrase);
-      // $phrase = preg_replace("/$object_type/", "<b>$object_type</b>", $phrase);
-
       // Convert the object/subject to a link if an entity exists for it.
       if (array_key_exists('entity', $item['value']['local:relationship_object'])) {
         list($entity_type, $object_entity_id) = explode(':', $item['value']['local:relationship_object']['entity']);
 
         if ($object_entity_id != $entity->id) {
-          $link = l($object_name, 'bio_data/' . $object_entity_id);
-          $phrase = str_replace($object_name, $link, $phrase);
+          $object_name = l($object_name, 'bio_data/' . $object_entity_id);
         }
       }
       if (array_key_exists('entity', $item['value']['local:relationship_subject'])) {
         list($entity_type, $subject_entity_id) = explode(':', $item['value']['local:relationship_subject']['entity']);
         if ($subject_entity_id != $entity->id) {
-          $link = l($subject_name, 'bio_data/' . $subject_entity_id);
-          $phrase = str_replace($subject_name, $link, $phrase);
+          $subject_name = l($subject_name, 'bio_data/' . $subject_entity_id);
         }
       }
 
+      // Add bold font to the subject and object names.
+      $subject_name = '<b>' . $subject_name . '</b>';
+      $object_name = '<b>' . $object_name . '</b>';
+
+      // The $clause text here will match what is already in $item['value']['SIO:000493']
+      // but we rebuild it here to incorporated links and formatting
+      $clause = 'The ' . $subject_type . ', ' .
+        $subject_name . ', ' . $verb . ' ' . $rel_type_clean . ' ' .
+        $object_type . ', ' . $object_name . '.';
+
       $rows[][] = [
-        'data' => $phrase,
+        'data' => $clause,
         //'class' => array('tripal-entity-unattached field-items')
       ];
     }
@@ -157,4 +166,3 @@ class sbo__relationship_formatter extends ChadoFieldFormatter {
   }
 }
 
-

tripal_chado/includes/TripalImporter/GFF3Importer.inc

@@ -306,17 +306,6 @@ class GFF3Importer extends TripalImporter {
       '#required' => TRUE,
       '#options' => $organisms,
     ];
-    $form['create_organism'] = [
-      '#type' => 'checkbox',
-      '#title' => t('Create organism'),
-      '#required' => FALSE,
-      '#description' => t('The Tripal GFF loader supports the "organism" attribute. This allows features of a
-       different organism to be aligned to the landmark sequence.  The format of the
-       attribute is "organism=[genus]:[species]", where [genus] is the organism\'s genus and [species] is the
-       species name. Check this box to automatically add the organism to the database if it does not already exists.
-       Otherwise lines with an organism attribute where the organism is not present in the database will be skipped.'),
-    ];
-
     $form['landmark_type'] = [
       '#title' => t('Landmark Type'),
       '#type' => 'textfield',
@@ -325,6 +314,7 @@ class GFF3Importer extends TripalImporter {
        the landmark features (first column of the GFF3 file) are not already in the database.."),
     ];
 
+
     $form['proteins'] = [
       '#type' => 'fieldset',
       '#title' => t('Proteins'),
@@ -410,7 +400,16 @@ class GFF3Importer extends TripalImporter {
       '#collapsible' => TRUE,
       '#collapsed' => FALSE,
     ];
-
+    $form['advanced']['create_organism'] = [
+      '#type' => 'checkbox',
+      '#title' => t('Create organism'),
+      '#required' => FALSE,
+      '#description' => t('The Tripal GFF loader supports the "organism" attribute. This allows features of a
+       different organism to be aligned to the landmark sequence.  The format of the
+       attribute is "organism=[genus]:[species]", where [genus] is the organism\'s genus and [species] is the
+       species name. Check this box to automatically add the organism to the database if it does not already exists.
+       Otherwise lines with an organism attribute where the organism is not present in the database will be skipped.'),
+    ];
 
     $form['advanced']['line_number'] = [
       '#type' => 'textfield',

tripal_ds/api/tripal_ds.pane.api.inc

@@ -39,7 +39,7 @@ function tripal_ds_create_field($field_label, $field_name, $bundle_name) {
   //Build the rest of the passes parameters.
   $group_field_name = 'gp_' . $field_name;
   //Create the field groups.
-  tripal_ds_additional_fields_field_group_info($bundle_name, $field_label, $group_field_name, $field_name);
+  tripal_ds_add_generic_field_group($bundle_name, $field_label, $group_field_name, $field_name);
   //Place the field groups in the layout.
   tripal_ds_update_ds_layout($bundle_name, $field_name, $group_field_name);
 }

tripal_ds/includes/tripal_ds.ds.inc

@@ -41,117 +41,113 @@ function _ds_layout_settings_info($bundle_name, $instances) {
   $region_left = [];
   $prop_fields = [];
   $summary_fields = [];
-  $data_sequence_fields = [];
-  $all_other_fields = [];
+  $unhandled_fields = [];
   $fields_with_regions = [];
   $i = 0;
-  $all_fields = [];
+
 
   try {
+
     // Get the bundle and term objects.
     $bundle = tripal_load_bundle_entity(['name' => $bundle_name]);
     $term = tripal_load_term_entity(['term_id' => $bundle->term_id]);
 
-    // Build one large multidimensional array of all instances to sort in alpha
-    // order to display fields in label alpha order.
+    // Get the list of fields and sort them by label.
+    $sorted_instances = [];
     foreach ($instances as $key => $instance) {
-      $all_fields[$i] = $instance;
+      $sorted_instances[$i] = $instance;
       $i++;
     }
-    usort($all_fields, tripal_ds_sort_array('label'));
+    usort($sorted_instances, tripal_ds_sort_array('label'));
 
-    // Iterate through the fields of this bundle.
-    foreach ($all_fields as $key => $instance) {
+    // Iterate through the fields of this bundle and put them into appropriate
+    // field groups.
+    foreach ($sorted_instances as $key => $instance) {
+      $base_table = array_key_exists('base_table', $instance['settings']) ? $instance['settings']['base_table'] : '';
+      $chado_table = array_key_exists('chado_table', $instance['settings']) ? $instance['settings']['chado_table'] : '';
       $instance_name = $instance['field_name'];
+
+      // Add the bundle type to the summary fields.
       if ($instance_name == "rdfs__type") {
         array_push($summary_fields, $instance_name);
+        continue;
       }
-      else {
-        //TODO: How do we handle non-chado dbs, placement of fields within
-        // tripal panes might need to be done in a hook.
-        $instance_base_table = array_key_exists('base_table', $instance['settings']) ? $instance['settings']['base_table'] : '';
-        $instance_base_chado = array_key_exists('chado_table', $instance['settings']) ? $instance['settings']['chado_table'] : '';
-        $prop_table = strpos($instance_base_chado, 'prop');
-        $data_sequence_record = strpos($instance_name, 'data__sequence_record');
-        $data_sequence = strpos($instance_name, 'data__sequence');
-        if ($instance_base_chado && $instance_base_table) {
-
-          if ($instance_base_chado == $instance_base_table) {
-            if ($data_sequence_record !== FALSE) {
-              array_push($data_sequence_fields, $instance_name);
-            }
-            elseif ($data_sequence !== FALSE or $instance_name == 'so__cds' or $instance_name == 'data__protein_sequence') {
-              // Do nothing as these chado column sequence fields
-              // should be hidden by default.
-            }
-            elseif ($prop_table !== FALSE) {
-              array_push($prop_fields, $instance_name);
-            }
-            else {
-              array_push($summary_fields, $instance_name);
-            }
 
-          }
-          elseif ($instance_base_chado != $instance_base_table) {
-            if ($data_sequence_record !== FALSE) {
-              array_push($data_sequence_fields, $instance_name);
-            }
-            elseif ($data_sequence !== FALSE or $instance_name == 'so__cds' or $instance_name == 'data__protein_sequence') {
-              // Do nothing as these chado column sequence fields
-              // should be hidden by default.
-            }
-            elseif ($prop_table !== FALSE) {
-              array_push($prop_fields, $instance_name);
-            }
-            else {
-              array_push($all_other_fields, $instance);
+      // The tripal_chado module adds an image to the organism content
+      // type so we want to make sure that image goes in the summary.
+      // It is not a TripalField so it won't have a chado table.
+      if ($instance_name == 'data__image' and $term->name == 'organism') {
+        array_push($summary_fields, $instance_name);
+        continue;
+      }
 
-              // Update the display settings so that the title is hidden.
-              $instance['display']['default']['label'] = 'hidden';
-              field_update_instance($instance);
-            }
-          }
-        }
-        else {
-          // The tripal_chado module adds an image to the organism content
-          // type so we want to make sure that image goes in the summary.
-          // It is not a TripalField so it won't have a chado table.
-          if ($instance_name == 'data__image' and $term->name == 'organism') {
-            array_push($summary_fields, $instance_name);
-          }
+      // For fields that have a data__sequence prefix we want to hide these
+      // unless it's the data__sequence_record field which displays all of the
+      // sequence data and should be in it's own fieldset.
+      if (preg_match('/data__sequence/', $instance_name)) {
+        if ($instance_name == 'data__sequence_record') {
+          $instance['display']['default']['label'] = 'hidden';
+          field_update_instance($instance);
+          array_push($unhandled_fields, $instance);
         }
+        continue;
+      }
+
+      // For the same reason, hide the CDS and protein fields
+      if ($instance_name == 'so__cds' or $instance_name == 'data__protein_sequence'){
+        continue;
+      }
+
+      // If this field holds a value from the prop table then add it to the
+      // properties field group.
+      if (preg_match('/_prop$/', $chado_table)) {
+        array_push($prop_fields, $instance_name);
+        continue;
       }
-    }
 
-    // Consolidate the field sets.
+      // Put all fields from the base table in the summary and all other
+      // fields that were not handled above into a different array for
+      // handling later.
+      if ($base_table == $chado_table) {
+        array_push($summary_fields, $instance_name);
+        continue;
+      }
+      else {
+        // Update the display settings so that the title is hidden.
+        $instance['display']['default']['label'] = 'hidden';
+        field_update_instance($instance);
+        array_push($unhandled_fields, $instance);
+      }
+    } // end looping over fields.
+
+    // Create the summary and properties field groups.
     if (!empty($summary_fields)) {
-      _summary_field_group_info($bundle_name, $summary_fields);
+      tripal_ds_add_summary_field_group($bundle_name, $summary_fields);
     }
     if (!empty($prop_fields)) {
-      _prop_field_group_info($bundle_name, $prop_fields);
-    }
-    if (!empty($data_sequence_fields)) {
-      _data_sequence_field_group_info($bundle_name, $data_sequence_fields);
+      tripal_ds_add_prop_field_group($bundle_name, $prop_fields);
     }
-    if (!empty($all_other_fields)) {
-      foreach ($all_other_fields as $key => $other_field) {
-        $group_field_name = 'gp_' . $other_field['field_name'];
-
-        // Need to truncate the names because of database field size restrictions,
-        // updating fields here to ensure name consistency.
-        $group_field_name = substr($group_field_name, 0, 27);
-
-        // Add random numbers to ensure the field name is unique within the 32
-        // character limit of the field.
-        $group_field_name = $group_field_name . rand(0, 99999);
-        tripal_ds_additional_fields_field_group_info($bundle_name, $other_field['label'], $group_field_name, $other_field['field_name']);
-      }
+
+    // All other fields get their own field group (i.e. Tripal Pane).
+    foreach ($unhandled_fields as $key => $other_field) {
+      $group_field_name = 'gp_' . $other_field['field_name'];
+
+      // Need to truncate the names because of database field size restrictions,
+      // updating fields here to ensure name consistency.
+      $group_field_name = substr($group_field_name, 0, 27);
+
+      // Add random numbers to ensure the field name is unique within the 32
+      // character limit of the field.
+      $group_field_name = $group_field_name . rand(0, 99999);
+
+      // Now add a generic field group for this field.
+      tripal_ds_add_generic_field_group($bundle_name, $other_field['label'], $group_field_name, $other_field['field_name']);
     }
 
-    // Build one large multidimensional array of all instances to sort in alpha
-    // order to display fields in label alpha order.
+    // Build one large multidimensional array of all field groups to sort
+    // in alpha order to display fields in label alpha order.
     $right_fields = [];
-    $all_field_groups = field_group_info_groups('TripalEntity', $bundle_name);
+    $all_field_groups = field_group_info_groups('TripalEntity', $bundle_name, NULL, TRUE);
 
     if (!empty($all_field_groups)) {
       if (is_array($all_field_groups)) {
@@ -164,31 +160,6 @@ function _ds_layout_settings_info($bundle_name, $instances) {
         usort($right_fields, tripal_ds_sort_object('label'));
       }
     }
-    elseif (empty($all_field_groups)) {
-      //Add the original instances that were passed and the field_groups that
-      //were created.
-      $field_group_fields = db_select('field_group', 'fg')
-        ->fields('fg', ['group_name', 'data'])
-        ->condition('bundle', $bundle_name, '=')
-        ->execute()
-        ->fetchAll();
-
-      $instance_names = [];
-      $field_group_names = [];
-      foreach ($all_fields as $key => $instance) {
-        $instance_names[$key]['field_name'] = $instance['field_name'];
-        $instance_names[$key]['label'] = $instance['label'];
-
-      }
-      foreach ($field_group_fields as $key => $field_group_name) {
-        $data = unserialize($field_group_name->data);
-        $field_group_names[$key]['field_name'] = $field_group_name->group_name;
-        $field_group_names[$key]['label'] = $data['format_settings']['label'];
-
-      }
-      $all_field_groups = array_merge($instance_names, $field_group_names);
-      usort($all_field_groups, tripal_ds_sort_array('label'));
-    }
 
     // Now build the $region_right array and the fields array.
     $i = 0;
@@ -337,7 +308,7 @@ function _ds_layout_pub_settings_info($bundle_name, $instances) {
       $group_field_name = $group_field_name . rand(0, 99999);
 
       // Build the field group.
-      tripal_ds_additional_fields_field_group_info($bundle_name, $other_field['label'], $group_field_name, $other_field['field_name']);
+      tripal_ds_add_generic_field_group($bundle_name, $other_field['label'], $group_field_name, $other_field['field_name']);
 
       // Update arrays.
       array_push($temporary_field, $group_field_name, $other_field['field_name']);

tripal_ds/includes/tripal_ds.field_group.inc

@@ -7,7 +7,7 @@
  * @param $fields
  *  Array of the machine names of the children of the field group being created.
  */
-function _summary_field_group_info($bundle_name, $fields) {
+function tripal_ds_add_summary_field_group($bundle_name, $fields) {
   //Tripal pane to nest the summary fieldset within.
   $field_group_tripalpane = new stdClass();
   $field_group_tripalpane->disabled = FALSE; /* Edit this to true to make a default field_group disabled initially*/
@@ -80,7 +80,7 @@ function _summary_field_group_info($bundle_name, $fields) {
  * @param $fields
  *  Array of the machine names of the children of the field group being created.
  */
-function _prop_field_group_info($bundle_name, $fields) {
+function tripal_ds_add_prop_field_group($bundle_name, $fields) {
   //Tripal pane  to nest the fieldset within.
   $field_group_tripalpane = new stdClass();
   $field_group_tripalpane->disabled = FALSE; /* Edit this to true to make a default field_group disabled initially*/
@@ -152,7 +152,7 @@ function _prop_field_group_info($bundle_name, $fields) {
  * @param $fields
  *  Array of the machine names of the children of the field group being created.
  */
-function _data_sequence_field_group_info($bundle_name, $fields) {
+function tripal_ds_add_sequences_field_group($bundle_name, $fields) {
   //Tripal pane  to nest the fieldset within.
   $field_group_tripalpane = new stdClass();
   $field_group_tripalpane->disabled = FALSE; /* Edit this to true to make a default field_group disabled initially*/
@@ -182,7 +182,7 @@ function _data_sequence_field_group_info($bundle_name, $fields) {
   ];
   drupal_write_record('field_group', $field_group_tripalpane);
   //Write to the tripal_ds table to record the new tripal pane.
-  tripal_ds_bundle_menu_item($bundle_name, 'Sequence', 'group_sequence_tripalpane', 'TripalEntity');
+  tripal_ds_bundle_menu_item($bundle_name, 'Sequences', 'group_sequence_tripalpane', 'TripalEntity');
 
   //Table of fields.
   $field_group = new stdClass();
@@ -230,7 +230,7 @@ function _data_sequence_field_group_info($bundle_name, $fields) {
  * @param $field_name
  *  Machine name of the child element.
  */
-function tripal_ds_additional_fields_field_group_info($bundle_name, $field_label, $group_field_name, $field_name) {
+function tripal_ds_add_generic_field_group($bundle_name, $field_label, $group_field_name, $field_name) {
   //Write to the tripal_ds table to record the new tripal pane.
   tripal_ds_bundle_menu_item($bundle_name, $field_label, $group_field_name, 'TripalEntity');