Explorar el Código

Merge pull request #2 from spficklin/dsenalik-agl_restoration

Dsenalik agl restoration
Douglas Senalik hace 5 años
padre
commit
19869affed

+ 6 - 6
docs/dev_guide/custom_field/manual_field_creation.rst

@@ -12,18 +12,18 @@ Before we create our class we must first create a proper directory structure.  T
 
 .. code-block:: bash
 
-  /sites/all/modules/[your_module]/includes/TripalField/[field_name]/[field_name].inc
-  /sites/all/modules/[your_module]/includes/TripalField/[field_name]/[field_name]_widget.inc
-  /sites/all/modules/[your_module]/includes/TripalField/[field_name]/[field_name]_formatter.inc
+  /sites/all/modules/[your_module]/includes/TripalFields/[field_name]/[field_name].inc
+  /sites/all/modules/[your_module]/includes/TripalFields/[field_name]/[field_name]_widget.inc
+  /sites/all/modules/[your_module]/includes/TripalFields/[field_name]/[field_name]_formatter.inc
 
 
 In the directories above the token [your_module] can be substituted with the name of your module and [field_name] is the name of your field.  You can name your field whatever you like, but you must use this name consistently in other locations throughout the modules.  Because all fields are defined by vocabulary terms, it is custom to name your fields with the vocabulary **short name** followed by two underscores followed by the **term name**, hence:  obi__organism.  Here the ChadoField implementation goes in the [field_name].inc file, the ChadoFieldWidget in the [field_name]_widget.inc file and the ChadoFieldFormatter in the [field_name]_formatter.inc.   All new fields must implement all three classes.   in the case of our obi__organism field the directory structure is as follows:
 
 .. code-block:: bash
 
-  /sites/all/modules/tripal/tripal_chado/includes/TripalField/obi__organism/obi__organism.inc
-  /sites/all/modules/tripal/tripal_chado/includes/TripalField/obi__organism/obi__organism_widget.inc
-  /sites/all/modules/tripal/tripal_chado/includes/TripalField/obi__organism/obi__organism_formatter.inc
+  /sites/all/modules/tripal/tripal_chado/includes/TripalFields/obi__organism/obi__organism.inc
+  /sites/all/modules/tripal/tripal_chado/includes/TripalFields/obi__organism/obi__organism_widget.inc
+  /sites/all/modules/tripal/tripal_chado/includes/TripalFields/obi__organism/obi__organism_formatter.inc
 
 Anatomy of the ChadoField Class
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ 4 - 3
docs/user_guide/bulk_loader.rst

@@ -9,10 +9,11 @@ The bulk loader is a tool that Tripal provides for loading of data contained in
 
 The following commands can be executed to install the Tripal Bulk Loader using Drush:
 
-.. code-block bash
+.. code-block:: bash
 
   cd /var/www/
   drush pm-enable tripal_bulk_loader
+  
 
 Plan How to Store Data
 ----------------------
@@ -21,7 +22,7 @@ To demonstrate use of the Bulk Loader, a brief example that imports a list of or
 
 * `Fragaria.txt <http://tripal.info/sites/default/files/book_pages/Fragaria_0.txt>`_
 
-.. code-block bash
+.. code-block:: bash
 
   cd $DRUPAL_HOME/sites/default/files
   wget http://tripal.info/sites/default/files/book_pages/Fragaria_0.txt
@@ -29,7 +30,7 @@ To demonstrate use of the Bulk Loader, a brief example that imports a list of or
 
 This file has three columns: NCBI taxonomy ID, genus and species:
 
-.. .. csv-table:: Fragaria sample file
+.. csv-table:: Fragaria sample file
 
   3747    "Fragaria"        "x ananassa"
   57918   "Fragaria"        "vesca"

+ 56 - 9
docs/user_guide/example_genomics/pub_import.rst

@@ -17,17 +17,19 @@ We will add information about the Tripal publication. Enter the following values
 .. csv-table::
   :header: "Field Name", "Value"
 
-  "Title", "Tripal v1.1: a standards-based toolkit for construction of online genetic and genomic databases."
+  "Title", "Tripal v3: an ontology-based toolkit for construction of FAIR biological community databases."
+  "Volume", "baz077"
   "Series Name", "Database"
-  "Publication Year", "2013"
-  "Unique Local Identifier", "Tripal v1.1: a standards-based toolkit for construction of online genetic and genomic databases."
-  "Type	Journal", "Article"
-  "Publication Date", "2013 Oct 25"
+  "Publication Year", "2019"
+  "Unique Local Identifier", "Tripal v3: an ontology-based toolkit for construction of FAIR biological community databases."
+  "Publication Type", "Journal"
   "Cross Reference", "Database: PMID"
-  "Accession", "24163125"
-  "Authors", "Sanderson LA, Ficklin SP, Cheng CH, Jung S, Feltus FA, Bett KE, Main D"
-  "Citation", "Sanderson LA, Ficklin SP, Cheng CH, Jung S, Feltus FA, Bett KE, Main D. Tripal: a construction Toolkit for Online Genome Databases. Database, Oct 25 2013. bat075"
-  "Abstract", "Tripal is an open-source freely available toolkit for construction of online genomic and genetic databases. It aims to facilitate development of community-driven biological websites by integrating the GMOD Chado database schema with Drupal, a popular website creation and content management software. Tripal provides a suite of tools for interaction with a Chado database and display of content therein. The tools are designed to be generic to support the various ways in which data may be stored in Chado. Previous releases of Tripal have supported organisms, genomic libraries, biological stocks, stock collections and genomic features, their alignments and annotations. Also, Tripal and its extension modules provided loaders for commonly used file formats such as FASTA, GFF, OBO, GAF, BLAST XML, KEGG heir files and InterProScan XML. Default generic templates were provided for common views of biological data, which could be customized using an open Application Programming Interface to change the way data are displayed. Here, we report additional tools and functionality that are part of release v1.1 of Tripal. These include (i) a new bulk loader that allows a site curator to import data stored in a custom tab delimited format; (ii) full support of every Chado table for Drupal Views (a powerful tool allowing site developers to construct novel displays and search pages); (iii) new modules including ‘Feature Map’, ‘Genetic’, ‘Publication’, ‘Project’, ‘Contact’ and the ‘Natural Diversity’ modules. Tutorials, mailing lists, download and set-up instructions, extension modules and other documentation can be found at the Tripal website located at http://tripal.info."
+  "Accession", "31328773"
+  "Publication Date", "2019 Jul 22"
+  "Citation", "Spoor S, Cheng CH, Sanderson LA, Condon B, Almsaeed A, Chen M, Bretaudeau A, Rasche H, Jung S, Main D, Bett K, Staton M, Wegrzyn JL, Feltus FA, Ficklin SP. Tripal v3: an ontology-based toolkit for construction of FAIR biological community databases.  Database. July 2019, 2019: baz077"
+  "Authors", "Spoor S, Cheng CH, Sanderson LA, Condon B, Almsaeed A, Chen M, Bretaudeau A, Rasche H, Jung S, Main D, Bett K, Staton M, Wegrzyn JL, Feltus FA, Ficklin SP"
+  "Abstract", "Community biological databases provide an important online resource for both public and private data, analysis tools and community engagement. These sites house genomic, transcriptomic, genetic, breeding and ancillary data for specific species, families or clades. Due to the complexity and increasing quantities of these data, construction of online resources is increasingly difficult especially with limited funding and access to technical expertise. Furthermore, online repositories are expected to promote FAIR data principles (findable, accessible, interoperable and reusable) that presents additional challenges. The open-source Tripal database toolkit seeks to mitigate these challenges by creating both the software and an interactive community of developers for construction of online community databases. Additionally, through coordinated, distributed co-development, Tripal sites encourage community-wide sustainability. Here, we report the release of Tripal version 3 that improves data accessibility and data sharing through systematic use of controlled vocabularies (CVs). Tripal uses the community-developed Chado database as a default data store, but now provides tools to support other data stores, while ensuring that CVs remain the central organizational structure for the data. A new site developer can use Tripal to develop a basic site with little to no programming, with the ability to integrate other data types using extension modules and the Tripal application programming interface. A thorough online User’s Guide and Developer’s Handbook are available at http://tripal.info, providing download, installation and step-by-step setup instructions."
+
 
 To complete the page click the **Save** button at the bottom
 
@@ -133,3 +135,48 @@ Where
 - [DRUPAL_HOME] is the directory where Drupal is installed
 
 The cron entry above will launch the importer at 8:30am on the first and fifteenth days of the month. We will run this importer twice a month in the event it fails to run (e.g. server is down) at least one time during the month.
+
+
+Import from the USDA National Agricultural Library
+--------------------------------------------------
+The instructions for the Tripal publication importer described previously use the the NCBI PubMed database. However, you can also import publications from the USDA National Agriculture Library (AGRICOLA). However, to use this repository a few software dependences are required.  These include:
+
+- The `YAZ library <https://www.indexdata.com/resources/software/yaz/>`_
+- `PHP support for YAZ <https://www.php.net/manual/en/book.yaz.php>`_  
+
+The following instructions are to install the necessary dependencies on an Ubuntu 18.04 LTS.   
+
+First install yaz, the yaz development library and the php development library:
+
+.. code-block:: bash
+ 
+  sudo apt-get install yaz libyaz5-dev php-dev
+
+
+Next update the PECL tool and install the PHP yaz library:
+
+
+.. code-block:: bash
+  
+  sudo pecl channel-update pecl.php.net
+  sudo pecl install yaz
+  
+Next, edit the `php.ini` files.  On Ubuntu 18.04 there are two PHP files:
+
+- `/etc/php/7.2/cli/php.ini`
+- `/etc/php/7.2/apache2/php.ini`
+
+Add the following line to each file:
+
+::
+
+  extension=yaz.so
+
+Finally, restart the web server so that it picks up the changes to the `php.ini` file.
+
+.. code-block:: bash
+  
+  sudo service apache2 restart
+
+You can now import publications from Agricola using the same interface as with PubMed.
+

+ 6 - 6
docs/user_guide/install_tripal/manual_install/install_drupal.rst

@@ -52,26 +52,26 @@ Substitute [user] for the name of the user that will own the web files.
 
   The apache web server runs as the user 'www-data'.  For security reasons you should chose a user other than 'www-data' to be the owner of the Drupal root directory.
 
-Tripal 3.x requires version 7.x of Drupal. Drupal can be freely downloaded from the http://www.drupal.org website. At the writing of this Tutorial the most recent version of Drupal 7 is version 7.64. The software can be downloaded manually from the Drupal website through a web browser or we can use the ``wget`` command to retrieve it:
+Tripal 3.x requires version 7.x of Drupal. Drupal can be freely downloaded from the http://www.drupal.org website. At the writing of this Tutorial the most recent version of Drupal 7 is version 7.69. The software can be downloaded manually from the Drupal website through a web browser or we can use the ``wget`` command to retrieve it:
 
 .. code-block:: bash
 
   cd $DRUPAL_HOME
-  wget http://ftp.drupal.org/files/projects/drupal-7.64.tar.gz
+  wget http://ftp.drupal.org/files/projects/drupal-7.69.tar.gz
 
 
 Next, we want to install Drupal. We will use the tar command to uncompress the software:
 
 .. code-block:: bash
 
-  tar -zxvf drupal-7.64.tar.gz
+  tar -zxvf drupal-7.69.tar.gz
 
-Notice that we now have a drupal-7.64 directory with all of the Drupal files. We want the Drupal files to be in our document root, not in a 'drupal-7.64' subdirectory. So, we'll move the contents of the directory up one level:
+Notice that we now have a drupal-7.69 directory with all of the Drupal files. We want the Drupal files to be in our document root, not in a 'drupal-7.69' subdirectory. So, we'll move the contents of the directory up one level:
 
 .. code-block:: bash
 
-  mv drupal-7.64/* ./
-  mv drupal-7.64/.htaccess ./
+  mv drupal-7.69/* ./
+  mv drupal-7.69/.htaccess ./
 
 If an index.html file is present (as is the case with Ubuntu installations) you can move it out of the way so that it does not interfere with Drupal by executing the following:
 

+ 109 - 0
tripal/includes/TripalBundleController.inc

@@ -162,6 +162,52 @@ class TripalBundleController extends EntityAPIControllerExportable {
     }
   }
 
+  /**
+   * Finds any orphaned entities associated with all bundles.
+   *
+   * An orphaned entity can occur if the module that created the entity
+   * unknowingly lost its underlying record in its data store.  Such a case
+   * could happen if someone directly removed the record from the data store
+   * outside of the module's control. This function allows each module
+   * to report if any orphans are missing for a given bundle type.
+   *
+   * @param bool $count
+   *   Set to TRUE to return only a count of orphans.
+   * @param integer $offset
+   *   For paging of entities set this to the offset within the total count.
+   * @param integer $limit
+   *   For paging of entities set this to the total number to return.
+   *
+   * @return array|integer
+   *  If $count == FALSE then an array of all entity IDs that are orphaned. If
+   *  $count == TRUE then a single integer count value is returned.
+   */
+  public function findAllOrphans($count = FALSE, $offset = 0, $limit = 0) {
+    $results = db_select('tripal_bundle', 'tb')
+      ->fields('tb')
+      ->orderBy('label')
+      ->execute();
+    if ($count) {
+      $response = 0;
+    }
+    else {
+      $response = "";
+    }
+    while (($bundle_record = $results->fetchObject())) {
+      $bid = $bundle_record->id;
+      $bundle_response = $this->findOrphans($bid, $count, $offset, $limit);
+      if (is_array($bundle_response)) {
+        foreach ($bundle_response as $key => $value) {
+          $response += $value;
+        }
+      }
+      else {
+        $response += $bundle_response;
+      }
+    }
+    return $response;
+  }
+
 
   /**
    * Deletes orphaned entities.
@@ -187,6 +233,9 @@ class TripalBundleController extends EntityAPIControllerExportable {
     $num_deleted = 0;
     $transaction = db_transaction();
     try {
+      if ($id === 0) {
+        return $this->deleteAllOrphans($job);
+      }
 
       // Get the list of entities that need cleanup.
       $eids = $this->findOrphans($id, FALSE, 0, 0);
@@ -227,4 +276,64 @@ class TripalBundleController extends EntityAPIControllerExportable {
     return $num_deleted;
 
   }
+
+  /**
+   * Deletes orphaned entities from all bundles.
+   *
+   * An orphaned entity can occur if the module that created the entity
+   * unknowingly lost its underlying record in its data store.  Such a case
+   * could happen if someone directly removed the record from the data store
+   * outside of the module's control. This function allows each module
+   * to report if any orphans are missing for a given bundle type.
+   *
+   * @param TripalJob $job
+   *   An optional Tripal Job object. This is provided when this function is
+   *   called using the Tripal Jobs system.
+   *
+   * @return integer
+   *   The number of entitites that were cleaned up.
+   */
+  public function deleteAllOrphans(TripalJob $job = NULL) {
+    $num_deleted = 0;
+    $transaction = db_transaction();
+
+    try {
+      $results = db_select('tripal_bundle', 'tb')
+        ->fields('tb')
+        ->orderBy('label')
+        ->execute();
+      $eids = $this->findAllOrphans(FALSE, 0, 0);
+      $num_entities = count($eids);
+      if ($job) {
+        $job->logMessage('Found !num orphaned entities.', ['!num' => $num_entities]);
+        $job->setInterval(1);
+        $job->setTotalItems($num_entities);
+      }
+
+      if ($num_entities == 0) {
+        return 0;
+      }
+
+      while (($bundle_record = $results->fetchObject())) {
+        $num_bundle_del = $this->deleteOrphans($bundle_record->id);
+        if ($job) {
+          $job->addItemsHandled($num_bundle_del);
+          $job->logMessage("Removed !num orphaned !label entities.", ['!num' => $num_bundle_del, '!label' => $bundle_record->label]);
+        }
+        $num_deleted += $num_bundle_del;
+      }
+    }
+    catch (Exception $e) {
+      $transaction->rollback();
+      $err_msg = "Failed to remove orphans: " . $e->getMessage();
+      if ($job) {
+        $job->logMessage($err_msg, [], 'error');
+      }
+      else {
+        drupal_set_message($err_msg, 'error');
+      }
+      return 0;
+    }
+    return $num_deleted;
+  }
 }

+ 1 - 0
tripal/includes/TripalEntityCollection.inc

@@ -527,6 +527,7 @@ class TripalEntityCollection {
     }
     if ($job) {
       $job->setTotalItems($total_entities);
+      $job->setInterval(1);
     }
 
     // Iterate through the bundles in this collection and get the entities.

+ 1 - 1
tripal/includes/TripalJob.inc

@@ -28,7 +28,7 @@ class TripalJob {
 
   /**
    * The interval when the job progress should be updated. Updating the job
-   * progress incurrs a database write which takes time and if it occurs to
+   * progress incurrs a database write which takes time and if it occurs too
    * frequently can slow down the loader.  This should be a value between
    * 0 and 100 to indicate a percent interval (e.g. 1 means update the
    * progress every time the num_handled increases by 1%).

+ 129 - 92
tripal/includes/tripal.registration.inc

@@ -13,9 +13,6 @@
 function tripal_registration_form($form, &$form_state) {
   $form_data = unserialize(variable_get('tripal_site_registration', NULL));
 
-  $form_state['details']['funding'] =
-    isset($form_state['details']['funding']) ? $form_state['details']['funding'] : 1;
-
   $form['description'] = [
     '#title' => 'Why Register your Site?',
     '#type' => 'item',
@@ -43,12 +40,35 @@ function tripal_registration_form($form, &$form_state) {
     '#title' => t('Do not register this site (opt-out).'),
     '#default_value' => isset($form_data['values']['disable_tripal_reporting']) ? $form_data['values']['disable_tripal_reporting'] : NULL,
     '#description' => "If you do not want to register your site please check
-      this box as it will stop the reminder notifications.  You can return later and register at any time.",
+      this box. You will no longer be notified to register and no details will be submitted.  You can return later and register at any time.",
     '#ajax' => array(
-      'callback' => '_tripal_form_disable_reg_callback',
-      'event' => 'click',
+      'callback' => 'custom_registration_ajax_disable_reg_callback',
+      'wrapper' => 'reg-details',
     ),
   );
+  $opt_out = $form_data['values']['disable_tripal_reporting'];
+  if (array_key_exists('values', $form_state) and
+      array_key_exists('disable_tripal_reporting', $form_state['values'])){
+        $opt_out = $form_state['values']['disable_tripal_reporting'] ? TRUE : FALSE;
+  }
+
+  if ($opt_out) {
+    $form['details'] = [
+        '#type' => 'markup',
+        '#prefix' => '<div id="reg-details">',
+        '#suffix' => '</div>',
+    ];
+    return $form;
+  }
+  $form['details'] = [
+    '#type' => 'fieldset',
+    '#title' => t('Registration Details'),
+    '#collapsible' => TRUE,
+    '#collapsed' => FALSE,
+    '#disabled' => $opt_out,
+    '#prefix' => '<div id="reg-details">',
+    '#suffix' => '</div>',
+  ];
 
   $purpose = array(0 => t('Production'), 1 => t('Development'), 2 => t('Experimental'));
   $form['details']['tripal_reg_site_purpose'] = array(
@@ -56,10 +76,10 @@ function tripal_registration_form($form, &$form_state) {
       '#title' => t('Site Status'),
       '#default_value' => isset($form_data['values']['tripal_reg_site_purpose']) ? $form_data['values']['tripal_reg_site_purpose'] : NULL,
       '#options' => $purpose,
-      '#required' => FALSE,
-      '#description' => t('Please register your site regardless if it is experimental (to explore tripal), 
-       for development of a future site (or updates to an existing site), or a site currently 
-       in production. For funding, it is important to know how many sites are active for each category.  If your site changes 
+      '#required' => TRUE,
+      '#description' => t('Please register your site regardless if it is experimental (to explore tripal),
+       for development of a future site (or updates to an existing site), or a site currently
+       in production. For funding, it is important to know how many sites are active for each category.  If your site changes
        status, such as from development to production, please remember to return and update the purpose.')
   );
 
@@ -74,7 +94,7 @@ function tripal_registration_form($form, &$form_state) {
       '#type' => 'textarea',
       '#title' => t('Description of the site'),
       '#default_value' => isset($form_data['values']['tripal_reg_site_description']) ? $form_data['values']['tripal_reg_site_description'] : NULL,
-      '#required' => FALSE,
+      '#required' => TRUE,
       '#description' => t('Please provide a brief description of this site.  Consider including
      details such as its purpose, the primary data types your site provides, and the
      research community your site serves.')
@@ -139,76 +159,76 @@ function tripal_registration_form($form, &$form_state) {
     '#suffix' => '</div>',
   );
 
-  $count = count($form_data['values']['funding']);
-  if ($form_state['details']['funding'] < $count) {
-    $form_state['details']['funding'] = $form_state['details']['funding'] + $count;
+  $num_sources = 1;
+  if (isset($form_data['values']['funding'])) {
+    $num_sources = count(array_keys($form_data['values']['funding'])) + 1;
   }
-  else {
-    $form_state['details']['funding'] = $form_state['details']['funding'] + 1;
+  if (array_key_exists('triggering_element', $form_state) and
+      $form_state['triggering_element']['#name'] == 'add_funding') {
+    $num_sources++;
   }
 
-  for ($i = 1; $i <= $form_state['details']['funding']; $i++) {
-    if ($i === $form_state['details']['funding']) {
-      $form['details']['funding'][$i] = array(
-        '#type' => 'fieldset',
-        '#title' => t("Funding Source $i"),
-        '#tree' => TRUE,
-        '#collapsible' => TRUE,
-        '#collapsed' => FALSE,
-        '#description' => t('When requesting funds for additional Tripal development,
-       it is important to report the breadth of of funding sources for Tripal sites.
-       Please consider sharing this information by providing the granting
-       agency, and funding periods.')
-      );
+  for ($i = 1; $i <= $num_sources; $i++) {
+    $collapsed = TRUE;
+    $tripal_reg_site_agency = '';
+    $tripal_reg_site_grant = '';
+    $tripal_reg_site_amount = '';
+    $tripal_reg_site_start = date('Y', time());
+    $tripal_reg_site_end = date('Y', time());
+    if ($i == $num_sources) {
+      $collapsed = FALSE;
     }
-    else {
-      $form['details']['funding'][$i] = array(
-        '#type' => 'fieldset',
-        '#title' => t("Funding Source $i"),
-        '#tree' => TRUE,
-        '#collapsible' => TRUE,
-        '#collapsed' => TRUE,
-        '#description' => t('When requesting funds for additional Tripal development,
-       it is important to report the breadth of of funding sources for Tripal sites.
-       Please consider sharing this information by providing the granting
-       agency, and funding periods.')
-      );
+    if ($i < $num_sources) {
+      $source = $form_data['values']['funding'][$i-1];
+      $tripal_reg_site_agency = $source['tripal_reg_site_agency'];
+      $tripal_reg_site_grant = $source['tripal_reg_site_grant'];
+      $tripal_reg_site_amount = $source['tripal_reg_site_amount'];
+      $tripal_reg_site_start = $source['funding_period']['tripal_reg_site_start'];
+      $tripal_reg_site_end = $source['funding_period']['tripal_reg_site_end'];
     }
+    $form['details']['funding'][$i] = array(
+      '#type' => 'fieldset',
+      '#title' => t("Funding Source %index, %name", ['%index' => $i, '%name' => $tripal_reg_site_agency]),
+      '#tree' => TRUE,
+      '#collapsible' => TRUE,
+      '#collapsed' => $collapsed,
+      '#description' => t('When requesting funds for additional Tripal development,
+     it is important to report the breadth of of funding sources for Tripal sites.
+     Please consider sharing this information by providing the granting
+     agency, and funding periods.')
+    );
+
     $form['details']['funding'][$i]['tripal_reg_site_agency'] = array(
       '#type' => 'textfield',
-      '#default_value' => isset($form_data['values']['funding'][$i]['tripal_reg_site_agency']) ? $form_data['values']['funding'][$i]['tripal_reg_site_agency'] : NULL,
+        '#default_value' => $tripal_reg_site_agency,
       '#title' => t('Funding Agency'),
     );
-
     $form['details']['funding'][$i]['tripal_reg_site_grant'] = array(
       '#type' => 'textfield',
-      '#default_value' => isset($form_data['values']['funding'][$i]['tripal_reg_site_grant']) ? $form_data['values']['funding'][$i]['tripal_reg_site_grant'] : NULL,
-      '#title' => t('Grant Number'),
+      '#default_value' => $tripal_reg_site_grant,
+      '#title' => t('Grant/Award Number'),
     );
     $form['details']['funding'][$i]['tripal_reg_site_amount'] = array(
       '#type' => 'textfield',
-      '#default_value' => isset($form_data['values']['funding'][$i]['tripal_reg_site_amount']) ? $form_data['values']['funding'][$i]['tripal_reg_site_amount'] : NULL,
+        '#default_value' => $tripal_reg_site_amount,
       '#title' => t('Funding Amount'),
     );
-
     $form['details']['funding'][$i]['funding_period'] = array(
       '#type' => 'fieldset',
       '#title' => t('Funding Period'),
       '#tree' => TRUE,
     );
-
     $form['details']['funding'][$i]['funding_period']['tripal_reg_site_start'] = array(
       '#type' => 'date_select',
       '#title' => t("Start"),
-      '#default_value' => isset($form_data['values']['funding'][$i]['funding_period']['tripal_reg_site_start']) ? $form_data['values']['funding'][$i]['funding_period']['tripal_reg_site_start'] : date('Y', time()),
+      '#default_value' => $tripal_reg_site_start,
       '#date_year_range' => '-20:+20',
       '#date_format' => 'Y',
     );
-
     $form['details']['funding'][$i]['funding_period']['tripal_reg_site_end'] = array(
       '#type' => 'date_select',
       '#title' => t('End'),
-      '#default_value' => isset($form_data['values']['funding'][$i]['funding_period']['tripal_reg_site_end']) ? $form_data['values']['funding'][$i]['funding_period']['tripal_reg_site_end'] : date('Y', time()),
+      '#default_value' => $tripal_reg_site_end,
       '#date_year_range' => '-20:+20',
       '#date_format' => 'Y',
     );
@@ -216,39 +236,42 @@ function tripal_registration_form($form, &$form_state) {
 
   $form['details']['funding']['add_funding'] = array(
     '#type' => 'button',
+    '#name' => 'add_funding',
     '#value' => t('Add additional funding sources'),
     '#href' => '',
+    '#limit_validation_errors' => [],
     '#ajax' => array(
       'callback' => 'custom_registration_ajax_add_funding',
       'wrapper' => 'funding',
     ),
   );
 
-  // Provide a submit button.
-  if (!empty($form_data)) {
-    $form['submit'] = array(
-      '#type' => 'submit',
-      '#value' => 'Update registration information',
-    );
-  }
-  else {
-    $form['submit'] = array(
-      '#type' => 'submit',
-      '#value' => 'Register Site',
-    );
-  }
-
+  $form['details']['submit'] = array(
+    '#type' => 'submit',
+    '#value' => 'Submit',
+  );
 
   return $form;
 
 }
 
 function custom_registration_ajax_add_funding($form, $form_state) {
+  // Save the current state of the registration form.
+  tripal_registration_form_save_registration_info($form_state);
+
   return $form['details']['funding'];
 }
 
-function _tripal_form_disable_reg_callback($form, &$form_state) {
+function custom_registration_ajax_disable_reg_callback($form, &$form_state) {
+  // Save the current state of the registration form.
+  tripal_registration_form_save_registration_info($form_state);
+
+  // Turn off the notice to register the site.
   variable_set('disable_tripal_reporting', TRUE);
+  if ($form_state['values']['disable_tripal_reporting']) {
+    drupal_set_message('You have opted-out of registation.');
+  }
+  return $form['details'];
 }
 
 /**
@@ -261,18 +284,43 @@ function _tripal_form_disable_reg_callback($form, &$form_state) {
  *   during processing.
  */
 function tripal_registration_form_validate($form, &$form_state){
-  $mail_pi = $form_state['values']['principal_investigator_email'];
-  $mail_sa = $form_state['values']['tripal_reg_site_admin_email'];
-  if ($form_state['values']['disable_tripal_reporting'] != TRUE) {
-    if (!empty($mail_pi) && !valid_email_address($mail_pi)) {
-      form_set_error('[details][principal_investigator][principal_investigator_email]', t('The email address for the principal investigator appears to be invalid.'));
-    }
-    if(!empty($mail_sa) && !valid_email_address($mail_sa)) {
-      form_set_error('[details][site_admin][tripal_reg_site_admin_email]', t("The email address for the site administrator appears to be invalid."));
-    }
+  $mail_pi = array_key_exists('principal_investigator_email', $form_state['values']) ? $form_state['values']['principal_investigator_email'] : '';
+  $mail_sa = array_key_exists('tripal_reg_site_admin_email', $form_state['values']) ? $form_state['values']['tripal_reg_site_admin_email'] : '';
+
+  if ($form_state['values']['disable_tripal_reporting']) {
+    return;
+  }
+
+  if (!$mail_pi and !$mail_sa) {
+    form_set_error('principal_investigator_email', t('Please provide an email address for the principal investigator or the site admin.'));
+  }
+
+  if (!empty($mail_pi) && !valid_email_address($mail_pi)) {
+    form_set_error('principal_investigator_email', t('The email address for the principal investigator appears to be invalid.'));
   }
+  if(!empty($mail_sa) && !valid_email_address($mail_sa)) {
+    form_set_error('tripal_reg_site_admin_email', t("The email address for the site administrator appears to be invalid."));
+  }
+
 }
 
+/**
+ * Saves the current state of the registration form as a drupal variable.
+ */
+function tripal_registration_form_save_registration_info($form_state) {
+  // Check for empty funding periods and remove them.
+  $sources = [];
+  foreach ($form_state['values']['funding'] as $funding_source) {
+    if (!empty($funding_source['tripal_reg_site_agency'])) {
+      $sources[] = $funding_source;
+    }
+  }
+  $form_state['values']['funding'] = $sources;
+
+  // Save the registration details.
+  $registration = serialize($form_state);
+  variable_set('tripal_site_registration', $registration);
+}
 /**
  * Implements submit from the Form API.
  *
@@ -284,33 +332,22 @@ function tripal_registration_form_validate($form, &$form_state){
  */
 function tripal_registration_form_submit($form, &$form_state) {
   variable_set('disable_tripal_reporting', TRUE);
+  tripal_registration_form_save_registration_info($form_state);
 
-  //Check for empty funding periods and remove them.
-  $j = 1;
-  foreach ($form_state['values']['funding'] as $funding_source) {
-    if (!empty($funding_source['tripal_reg_site_agency']) && !empty($funding_source['tripal_reg_site_grant'])) {
-      $form_state['values']['fundings'][$j] = $funding_source;
-      $j++;
-    }
-  }
-  $form_state['values']['funding'] = $form_state['values']['fundings'] ?? [];
-  $registration = serialize($form_state);
-  variable_set('tripal_site_registration', $registration);
-
-  //Now send the updated info to the Tripal Site.
+  // Now send the updated info to the Tripal Site.
   // Only register with tripal.info if the user has not opt'd out.
   if ($form_state['values']['disable_tripal_reporting'] == FALSE) {
+
+    $registration = variable_get('tripal_site_registration');
     tripal_registration_remote_submit($registration);
     drupal_set_message(t('Registration sent to tripal.info'), 'status');
     drupal_set_message(t('Thank you for registering. You can update your details at any time.'), 'status');
   }
   else {
-    drupal_set_message(t('You are not registered with tripal.info. You can 
-            change this at any time by unchecking the opt out checkbox and 
+    drupal_set_message(t('You are not registered with tripal.info. You can
+            change this at any time by unchecking the opt out checkbox and
             submitting the form.'), 'status');
   }
-
-
 }
 
 function tripal_registration_remote_submit($data) {

+ 20 - 1
tripal/includes/tripal.unpublish_orphans.inc

@@ -23,6 +23,7 @@ function tripal_unpublish_orphans_form($form, &$form_state) {
   while ($bundle = $results->fetchObject()) {
     $bundles[$bundle->id] = $bundle->label;
   }
+  $bundles[0] = "Delete all orphaned content";
   drupal_set_title('Unpublish Orphaned Content');
 
   $form['description'] = [
@@ -91,6 +92,22 @@ function tripal_unpublish_orphans_form($form, &$form_state) {
     }
   }
 
+  if ($selected_bundle_id === '0') {
+    $bundlec = entity_get_controller('TripalBundle');
+    $count = $bundlec->findAllOrphans(TRUE);
+    $form['bundle_info_fieldset']['message'] = [
+      '#type' => 'markup',
+      '#markup' => t('<p><strong>There are ' . $count . ' orphaned entities in all bundles.</strong></p>'),
+    ];
+
+    if ($count > 0) {
+      $form['bundle_info_fieldset']['submit'] = [
+        '#type' => 'submit',
+        '#value' => 'Unpublish Orphaned Entities',
+      ];
+    }
+  }
+
   return $form;
 }
 
@@ -104,7 +121,9 @@ function tripal_unpublish_orphans_form_validate($form, &$form_state) {
   $bundle_id = isset($form_state['values']['bundles']) ? $form_state['values']['bundles'] : NULL;
 
   if (empty($bundle_id) || !is_numeric($bundle_id)) {
-    form_set_error('bundles', t('Please select a valid bundle.'));
+    if (!isset($bundle_id) || $bundle_id !== '0') {
+      form_set_error('bundles', t('Please select a valid bundle.'));
+    }
   }
 }
 

+ 8 - 6
tripal/tripal.module

@@ -84,15 +84,17 @@ function tripal_init() {
   // otherwise PostgreSQL version that may have a different datestyle setting
   // will fail when inserting or updating a date column in a table.
   db_query("SET DATESTYLE TO :style", array(':style' => 'MDY'));
-  
+
   //Ask users to do the registration form
   if (user_access('administer tripal')) {
     if (empty(variable_get('tripal_site_registration', FALSE)) || !(variable_get('disable_tripal_reporting', FALSE))) {
-      drupal_set_message('Please register your Tripal Site. Registering provides important
-      information that will help secure funding for continued improvements to Tripal. ' .
-        l('Click to register now or opt out.', 'admin/tripal/register'), 'warning');
+      if (current_path() != 'admin/tripal/register' and current_path() != 'system/ajax') {
+        drupal_set_message('Please register your Tripal Site. Registering provides important
+        information that will help secure funding for continued improvements to Tripal. ' .
+          l('Click to register now or opt out.', 'admin/tripal/register'), 'warning');
+      }
     }
-  } 
+  }
 }
 
 function tripal_menu_alter(&$items) {
@@ -1382,7 +1384,7 @@ function tripal_cron() {
   tripal_add_job('Cron: Checking expired files', 'tripal',
     'tripal_expire_files', $args, 1, 1, $includes, TRUE);
 
-  // Update the registration information every month.  
+  // Update the registration information every month.
   if (variable_get('tripal_site_registration', FALSE)) {
     $last_submit = variable_get('tripal_site_registration_last_update');
     if ($last_submit) {

+ 8 - 6
tripal_chado/api/modules/tripal_chado.pub.api.inc

@@ -368,7 +368,7 @@ function chado_import_pub_by_dbxref($pub_dbxref, $do_contact = FALSE,
 
   $message = "Importing of publications is performed using a database transaction. " .
     "If the load fails or is terminated prematurely then the entire set of " .
-    "deletions is rolled back and will not be found in the database";
+    "insertions/updates is rolled back and will not be found in the database";
   tripal_report_error($message_type, TRIPAL_INFO, $message, [], $message_opts);
 
 
@@ -505,7 +505,7 @@ function chado_execute_pub_importer($import_id, $publish = TRUE,
 
   $message = "Importing of publications for this importer is performed using a database transaction. " .
     "If the load fails or is terminated prematurely then the entire set of " .
-    "deletions is rolled back and will not be found in the database";
+    "insertions/updates is rolled back and will not be found in the database";
   tripal_report_error($message_type, TRIPAL_INFO, $message, [], $message_opts);
 
 
@@ -532,9 +532,10 @@ function chado_execute_pub_importer($import_id, $publish = TRUE,
     // Loop until we have a $pubs array that does not have
     // our requested numer of records.  This means we've hit the end
     do {
-      // retrieve the pubs for this page. We'll retreive 100 at a time
+      // retrieve the pubs for this page. We'll retrieve 100 at a time
+      $npages = isset($num_pubs)?(intval($num_pubs/$num_to_retrieve)+1):'?';  // will be 0 to 99 in last page
       tripal_report_error($message_type, TRIPAL_INFO,
-        "Querying !remote_db for up to !num pubs that match the criteria.",
+        "Page ".($page+1)." of $npages. Querying !remote_db for up to !num pubs that match the criteria.",
         [
           '!num' => $num_to_retrieve,
           '!remote_db' => $remote_db,
@@ -548,11 +549,12 @@ function chado_execute_pub_importer($import_id, $publish = TRUE,
         ['%num' => $num_pubs], $message_opts);
 
       $subset_report = tripal_pub_add_publications($pubs, $import->do_contact, $do_update, $job);
+      $countpubs = count($pubs);  // the following merge resets count($pubs) so save it
       foreach ($subset_report as $action => $pubs) {
         $report[$action] = array_merge($report[$action], $pubs);
       }
       $page++;
-    } while (count($pubs) == $num_to_retrieve);
+    } while ($countpubs == $num_to_retrieve);
 
     // Publish as requested by the caller.
     _chado_execute_pub_importer_publish($publish, $job, $message_type, $message_opts);
@@ -668,7 +670,7 @@ function chado_reimport_publications($do_contact = FALSE, $dbxref = NULL,
 
   $message = "Importing of publications for this importer is performed using a database transaction. " .
     "If the load fails or is terminated prematurely then the entire set of " .
-    "deletions is rolled back and will not be found in the database";
+    "insertions/updates is rolled back and will not be found in the database";
   tripal_report_error($message_type, TRIPAL_INFO, $message, [], $message_opts);
 
   $transaction = db_transaction();

+ 2 - 0
tripal_chado/includes/TripalFields/sio__references/sio__references.inc

@@ -98,6 +98,7 @@ class sio__references extends ChadoField {
       $matches = [];
       if (preg_match('/^(.+?)_pub$/', $chado_table, $matches)) {
         $reference_table = $matches[1];
+
         // Find the base table this links to and get the fk columns that map it.
         $schema = chado_get_schema($chado_table);
         $fkeys = $schema['foreign keys'];
@@ -106,6 +107,7 @@ class sio__references extends ChadoField {
             $fkleft = array_keys($fk_details['columns'])[0];
             $fkright = $fk_details['columns'][$fkleft];
           }
+
         }
         // Iterate through all of the records in the linker table that
         // match the given pub ID.

+ 5 - 0
tripal_chado/includes/TripalFields/sio__references/sio__references_formatter.inc

@@ -16,6 +16,11 @@ class sio__references_formatter extends ChadoFieldFormatter {
     $field_name = $this->field['field_name'];
     $chado_table = $this->instance['settings']['chado_table'];
 
+    // Do we have an empty list? If so, just return.
+    if (!$items[0]['value']) {
+      return;
+    }
+
     // First, organize the values by their types.
     $ordered_items = [];
     foreach ($items as $delta => $item) {

+ 80 - 30
tripal_chado/includes/loaders/tripal_chado.pub_importer_AGL.inc

@@ -30,8 +30,8 @@ function tripal_pub_remote_alter_form_AGL($form, $form_state, $num_criteria = 1)
   // So far we haven't been able to get AGL to filter results to only
   // include pubs by the XX number days in the past.  So, we will
   // change the 'days' element to be the year to query
-  $form['themed_element']['days']['#title'] = t('Year');
-  $form['themed_element']['days']['#description'] = t('Please enter a year to limit records by the year they were published, created or modified in the database.');
+  $form['themed_element']['days']['#title'] = t('Earliest year of publication');
+  $form['themed_element']['days']['#description'] = t('Filter returned publications for those that have been published no earlier than this year.');
 
   // The Journal Name filter doesn't seem to work, so remove it
   for ($i = 1; $i <= $num_criteria; $i++) {
@@ -55,11 +55,15 @@ function tripal_pub_remote_alter_form_AGL($form, $form_state, $num_criteria = 1)
  */
 function tripal_pub_remote_validate_form_AGL($form, $form_state) {
   $days = trim($form_state['values']["days"]);
+  $latestyear = trim($form_state['values']["latestyear"]);
   $num_criteria = $form_state['values']['num_criteria'];
 
   if ($days and !preg_match('/^\d\d\d\d$/', $days)) {
     form_set_error("days", "Please enter a four digit year.");
   }
+  if ($latestyear and !preg_match('/^\d\d\d\d$/', $latestyear)) {
+    form_set_error("latestyear", "Please enter a four digit year.");
+  }
 
   $num_ids = 0;
   for ($i = 1; $i <= $num_criteria; $i++) {
@@ -96,9 +100,10 @@ function tripal_pub_remote_validate_form_AGL($form, $form_state) {
  * @ingroup tripal_pub
  */
 function tripal_pub_remote_search_AGL($search_array, $num_to_retrieve, $page) {
-  // get some values from the serach array
+  // get some values from the search array
   $num_criteria = $search_array['num_criteria'];
   $days = array_key_exists('days', $search_array) ? $search_array['days'] : '';
+  $latestyear = array_key_exists('latestyear', $search_array) ? $search_array['latestyear'] : '';
 
   // set some defaults
   $search_array['limit'] = $num_to_retrieve;
@@ -342,10 +347,10 @@ function tripal_pub_remote_search_AGL($search_array, $num_to_retrieve, $page) {
     }
   }
 
-  // for AGL the 'days' form element was converted to represent the year
-  if ($days) {
-    $ccl .= "and year=($days)";
-  }
+  // for AGL the 'days' form element was converted to represent the earliest year
+  // ! these search terms do not work for AGL, disable here and filter the results returned
+  // if ($days) { $ccl .= "and year>=($days) "; }
+  // if ($latestyear) { $ccl .= "and year<=($latestyear) "; }
 
   // remove any preceeding 'and' or 'or'
   $ccl = preg_replace('/^\s*(and|or)/', '', $ccl);
@@ -364,12 +369,17 @@ function tripal_pub_remote_search_AGL($search_array, $num_to_retrieve, $page) {
   // the search query is built using CCL, we need to first
   // configure it so it can map the attributes to defined identifiers
   // The attribute set used by AGL can be found at the bottom of this page:
-  // http://agricola.nal.usda.gov/help/z3950.html
+  // https://agricola.nal.usda.gov/help/z3950.html
+  //   Boolean searching (AND, OR, NOT) is supported on search types with
+  //   a position attribute of 3 only.
   //
   // More in depth details:  http://www.loc.gov/z3950/agency/bib1.html
   //
-  // CCL Syntax: http://www.indexdata.com/yaz/doc/tools.html#CCL
+  // CCL Syntax: https://www.indexdata.com/yaz/doc/tools.html#CCL
   //
+  // the abstract field u=62, year u=30, and journal u=1033 are not in the documented
+  // list of supported values at https://agricola.nal.usda.gov/help/z3950.html
+  // publisherdate u=31 is listed, but if used returns zero results
   $fields = [
     "title" => "u=4",
     "author" => "u=1003",
@@ -394,9 +404,9 @@ function tripal_pub_remote_search_AGL($search_array, $num_to_retrieve, $page) {
   // get the total number of records
   $total_records = tripal_pub_AGL_count($yazc, $search_str);
 
-  // get the pubs in the specified rang
+  // get the pubs in the specified range
   $start = $page * $num_to_retrieve;
-  $results = tripal_pub_AGL_range($yazc, $search_str, $start, $num_to_retrieve, $total_records);
+  $results = tripal_pub_AGL_range($yazc, $search_str, $start, $num_to_retrieve, $total_records, $days, $latestyear);
 
   // close the connection
   yaz_close($yazc);
@@ -425,8 +435,18 @@ function tripal_pub_remote_search_AGL($search_array, $num_to_retrieve, $page) {
  *
  * @ingroup tripal_pub
  */
-function tripal_pub_AGL_range($yazc, $search_str, $start, $num_to_retrieve, $total_records) {
-  yaz_range($yazc, 1, $total_records);
+function tripal_pub_AGL_range($yazc, $search_str, $start, $num_to_retrieve, $total_records, $startyear, $endyear) {
+  // The original code was: yaz_range($yazc, 1, $total_records);
+  // and for large queries we received an error: ERROR retrieving records from AGL: (10007) Timeout
+  // or for an intermediate size query: ERROR retrieving records from AGL: (10004) Connection lost
+  // Empirical testing shows errors started somewhere between 1550 and 1575 records
+  // If we use just the appropriate values for this range to the call to yaz_range then it works,
+  // but we can't exceed $total_records
+  $local_to_retrieve = $num_to_retrieve;
+  if (($start + $num_to_retrieve) > $total_records) {
+    $local_to_retrieve = $total_records - $start;
+  }
+  yaz_range($yazc, $start, $local_to_retrieve); // $start is 0-based
   if (!yaz_present($yazc)) {
     $error_no = yaz_errno($yazc);
     $error_msg = yaz_error($yazc);
@@ -463,8 +483,21 @@ function tripal_pub_AGL_range($yazc, $search_str, $start, $num_to_retrieve, $tot
         'pubs' => [],
       ];
     }
+
     // parse the pub XML
     $pub = tripal_pub_AGL_parse_pubxml($pub_xml);
+
+    // since year limits don't work when searching, implement them here for the
+    // returned publications. This is a very ugly solution, because the count of
+    // publications is now wrong, the xml for every pub must still be downloaded,
+    // and the Test Importer will display pubs that will be later filtered out.
+    $pass = 1;
+    if (array_key_exists('Year', $pub)) {
+      if ( ($startyear) and ($pub['Year']<$startyear) ) { $pass = 0; }
+      if ( ($endyear) and ($pub['Year']>$endyear) ) { $pass = 0; }
+    }
+    $pub['passfilter'] = $pass;
+
     $pubs[] = $pub;
   }
   return [
@@ -583,17 +616,24 @@ function tripal_pub_AGL_parse_pubxml($pub_xml) {
             '11' => 'Nov',
             '12' => 'Dec',
           ];
-          $date0 = substr($value, 0, 6);  // date entered on file
-          $date1 = substr($value, 7, 4);  // year of publication
-          $date2 = substr($value, 11, 4); // month of publication
-          $place = substr($value, 15, 3);
-          $lang = substr($value, 35, 3);
+          $date0 = mb_substr($value, 0, 6);  // date entered on file
+          $typeofdate = mb_substr($value, 6, 1);  // e = detailed date
+          $date1 = mb_substr($value, 7, 4);  // year of publication
+          $date2 = mb_substr($value, 11, 2); // month of publication
+          $date3 = mb_substr($value, 13, 2); // day of publication
+          $place = mb_substr($value, 15, 3);
+          $lang = mb_substr($value, 35, 3);
           if (preg_match('/\d\d\d\d/', $date1)) {
             $pub['Year'] = $date1;
             $pub['Publication Date'] = $date1;
           }
-          if (preg_match('/\d\d/', $date2)) {
-            $pub['Publication Date'] = $date1 . " " . $month[substr($date2, 0, 2)] . " " . substr($date2, 3, 2);
+          if (($typeofdate == 'e') and (preg_match('/\d\d/', $date2))) {
+            if ( ( $date2 >= 1 ) and ( $date2 <= 12 ) ) {
+              $pub['Publication Date'] = $date1 . " " . $month[$date2] . " " . $date3;
+            }
+            else {
+              drupal_set_message("Invalid month value \"$date2\" extracted from \"$value\"", "warning");
+            }
           }
           if (!preg_match('/\s+/', $place)) {
             $pub['Published Location'] = $place;
@@ -892,7 +932,12 @@ function tripal_pub_AGL_parse_pubxml($pub_xml) {
   }
 
   // build the full authors list
-  if (is_array($pub['Author List'])) {
+  if (!array_key_exists('Author List', $pub)) {
+    // there is a constraint in chado.pubprop against value being null 
+    $pub['Author List'] = array(['Surname' => 'anonymous']);
+    $pub['Authors'] = 'anonymous';
+  }
+  else if (is_array($pub['Author List'])) {
     $authors = '';
     foreach ($pub['Author List'] as $author) {
       if (array_key_exists('valid', $author) and $author['valid'] == 'N') {
@@ -913,7 +958,7 @@ function tripal_pub_AGL_parse_pubxml($pub_xml) {
         }
       }
     }
-    $authors = substr($authors, 0, -2);
+    $authors = mb_substr($authors, 0, -2);
     $pub['Authors'] = $authors;
   }
   else {
@@ -926,13 +971,18 @@ function tripal_pub_AGL_parse_pubxml($pub_xml) {
     $pub['Abstract'] = preg_replace('/[\p{So}]/u', '', mb_convert_encoding($pub['Abstract'], 'UTF-8', 'HTML-ENTITIES'));
   }
   $newauths = [];
-  foreach ($pub['Author List'] AS $auth) {
-    foreach ($auth AS $k => $v) {
-      $auth[$k] = preg_replace('/[\p{So}]/u', '', mb_convert_encoding($v, 'UTF-8', 'HTML-ENTITIES'));
+  if (array_key_exists('Author List', $pub)) {
+    foreach ($pub['Author List'] AS $auth) {
+      foreach ($auth AS $k => $v) {
+        $auth[$k] = preg_replace('/[\p{So}]/u', '', mb_convert_encoding($v, 'UTF-8', 'HTML-ENTITIES'));
+      }
+      array_push($newauths, $auth);
     }
-    array_push($newauths, $auth);
+    $pub['Author List'] = $newauths;
+  }
+  else {
+    $pub['Author List'] = array(['Surname' => 'anonymous']);
   }
-  $pub['Author List'] = $newauths;
 
   // build the citation
   $pub['Citation'] = chado_pub_create_citation($pub);
@@ -1009,11 +1059,11 @@ function tripal_pub_remote_search_AGL_get_author($xml, $ind1) {
           $first_names = explode(' ', $author['Given Name']);
           $author['First Initials'] = '';
           foreach ($first_names as $index => $name) {
-            $author['First Initials'] .= substr($name, 0, 1);
+            $author['First Initials'] .= mb_substr($name, 0, 1);
           }
         }
-        if ($ind1 == 3) { // A family name
-
+        if ($ind1 == 3) { // A family name, occurs rarely
+          $author['Surname'] = $value;
         }
         break;
     }

+ 20 - 4
tripal_chado/includes/loaders/tripal_chado.pub_importer_PMID.inc

@@ -208,12 +208,20 @@ function tripal_pub_PMID_search_init($search_str, $retmax) {
   // do a search for a single result so that we can establish a history, and get
   // the number of records. Once we have the number of records we can retrieve
   // those requested in the range.
-  $query_url = "http://www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?" .
+  $query_url = "https://www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?" .
     "db=Pubmed" .
     "&retmax=$retmax" .
     "&usehistory=y" .
     "&term=" . urlencode($search_str);
 
+  $api_key = variable_get('tripal_pub_importer_ncbi_api_key', NULL);
+  $sleep_time = 333334;
+  if (!empty($api_key)) {
+    $query_url .= "&api_key=" . $api_key;
+    $sleep_time = 100000;
+  }
+
+  usleep($sleep_time);  // 1/3 of a second delay, NCBI limits requests to 3 / second without API key
   $rfh = fopen($query_url, "r");
   if (!$rfh) {
     drupal_set_message('Could not perform Pubmed query. Cannot connect to Entrez.', 'error');
@@ -292,7 +300,7 @@ function tripal_pub_PMID_fetch($query_key, $web_env, $rettype = 'null',
 
   // repeat the search performed previously (using WebEnv & QueryKey) to retrieve
   // the PMID's within the range specied.  The PMIDs will be returned as a text list
-  $fetch_url = "http://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?" .
+  $fetch_url = "https://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?" .
     "rettype=$rettype" .
     "&retmode=$retmod" .
     "&retstart=$start" .
@@ -301,6 +309,13 @@ function tripal_pub_PMID_fetch($query_key, $web_env, $rettype = 'null',
     "&query_key=$query_key" .
     "&WebEnv=$web_env";
 
+  $api_key = variable_get('tripal_pub_importer_ncbi_api_key', NULL);
+  $sleep_time = 333334;
+  if (!empty($api_key)) {
+    $fetch_url .= "&api_key=" . $api_key;
+    $sleep_time = 100000;
+  }
+
   foreach ($args as $key => $value) {
     if (is_array($value)) {
       $fetch_url .= "&$key=";
@@ -313,6 +328,7 @@ function tripal_pub_PMID_fetch($query_key, $web_env, $rettype = 'null',
       $fetch_url .= "&$key=$value";
     }
   }
+  usleep($sleep_time);  // 1/3 of a second delay, NCBI limits requests to 3 / second without API key
   $rfh = fopen($fetch_url, "r");
   if (!$rfh) {
     drupal_set_message('ERROR: Could not perform PubMed query.', 'error');
@@ -338,10 +354,10 @@ function tripal_pub_PMID_fetch($query_key, $web_env, $rettype = 'null',
  * XML should contain only a single publication record.
  *
  * Information about the valid elements in the PubMed XML can be found here:
- * http://www.nlm.nih.gov/bsd/licensee/elements_descriptions.html
+ * https://www.nlm.nih.gov/bsd/licensee/elements_descriptions.html
  *
  * Information about PubMed's citation format can be found here
- * http://www.nlm.nih.gov/bsd/policy/cit_format.html
+ * https://www.nlm.nih.gov/bsd/policy/cit_format.html
  *
  * @param $pub_xml
  *  An XML string describing a single publication

+ 181 - 77
tripal_chado/includes/loaders/tripal_chado.pub_importers.inc

@@ -82,9 +82,11 @@ function tripal_pub_importers_list() {
       <li>Create a new importer by clicking the 'New Importer' link above, and after saving it should appear in the list below.  Click the
           link labeled 'Import Pubs' to schedule a job to import the publications</li>
       <li>The first method only performs the import once.  However, you can schedule the
-          importer to run peridically by adding a cron job. </li> 
+          importer to run peridically by adding a cron job. </li>
      </ol><br>");
 
+  $form = drupal_get_form('tripal_pub_importer_ncbi_api_key_form');
+  $page .= drupal_render($form);
 
   $table = [
     'header' => $headers,
@@ -131,13 +133,9 @@ function tripal_pub_importer_setup_page($action = 'new', $pub_import_id = NULL)
     drupal_set_message(t('If you want to create contact pages for authors, you must first ') . l(t('load the Tripal Contact Ontology'), 'admin/tripal/loaders/chado_vocabs/obo_loader'), 'error');
   }
 
-  /**
-   * Commenting the AGL message out until we get AGL working again.
-   * spf 6/22/2018
-   * if(!extension_loaded ('yaz')){
-   * drupal_set_message(t('<b>Note:</b> In order to create an importer using the USDA National Agricultural Library (AGL) you must install the yaz libraries. See the ') . l(t('Pub Module help page'), 'admin/tripal/legacy/tripal_pub/help') . ' for assistance.  If you do not want to use AGL you can ignore this warning.', 'warning');
-   * }
-   */
+  if(!extension_loaded ('yaz')){
+    drupal_set_message(t('<b>Note:</b> In order to create an importer using the USDA National Agricultural Library (AGL) you must install the yaz libraries. See the ') . l(t('Users Guide for Instructions'), 'https://tripal.readthedocs.io/en/latest/user_guide/example_genomics/pub_import.html#import-from-the-usda-national-agricultural-library') . ' for assistance.  If you do not want to use AGL you can ignore this warning.', 'warning');
+  }
 
   // generate the search form
   $form = drupal_get_form('tripal_pub_importer_setup_form', $pub_import_id, $action);
@@ -150,11 +148,13 @@ function tripal_pub_importer_setup_page($action = 'new', $pub_import_id = NULL)
     $remote_db = array_key_exists('remote_db', $_SESSION['tripal_pub_import']) ? $_SESSION['tripal_pub_import']['remote_db'] : '';
     $num_criteria = array_key_exists('num_criteria', $_SESSION['tripal_pub_import']) ? $_SESSION['tripal_pub_import']['num_criteria'] : '';
     $days = array_key_exists('days', $_SESSION['tripal_pub_import']) ? $_SESSION['tripal_pub_import']['days'] : '';
+    $latestyear = array_key_exists('latestyear', $_SESSION['tripal_pub_import']) ? $_SESSION['tripal_pub_import']['latestyear'] : '';
 
     $search_array = [];
     $search_array['remote_db'] = $remote_db;
     $search_array['num_criteria'] = $num_criteria;
     $search_array['days'] = $days;
+    $search_array['latestyear'] = $latestyear;
     for ($i = 1; $i <= $num_criteria; $i++) {
       $search_array['criteria'][$i]['search_terms'] = $_SESSION['tripal_pub_import']['criteria'][$i]['search_terms'];
       $search_array['criteria'][$i]['scope'] = $_SESSION['tripal_pub_import']['criteria'][$i]['scope'];
@@ -184,6 +184,10 @@ function tripal_pub_importer_setup_page($action = 'new', $pub_import_id = NULL)
           if (array_key_exists('Publication Dbxref', $pub) and $pub['Publication Dbxref']) {
             $raw_link = l('raw', 'admin/tripal/loaders/pub/raw/' . $pub['Publication Dbxref'], ['attributes' => ['target' => '_blank']]);
           }
+          // indicate those that will be excluded by AGL year filtering parameters
+          if ((array_key_exists('passfilter', $pub)) and ($pub['passfilter'] == 0 )) {
+            $citation = '<span style="text-decoration: line-through;">' . $citation . '</span>';
+          }
           $rows[] = [
             number_format($i),
             $citation,
@@ -277,12 +281,13 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
   $criteria = NULL;
   $remote_db = '';
   $days = '';
+  $latestyear = '';
   $disabled = '';
   $do_contact = '';
   $num_criteria = 1;
   $loader_name = '';
 
-  // if this is an edit the we are pulling an import object from the database
+  // if this is an edit then we are pulling an import object from the database
   if ($action == "edit") {
     $sql = "SELECT * FROM {tripal_pub_import} WHERE pub_import_id = :pub_import_id";
     $importer = db_query($sql, [':pub_import_id' => $pub_import_id])->fetchObject();
@@ -290,6 +295,7 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
     $criteria = unserialize($importer->criteria);
     $remote_db = $criteria['remote_db'];
     $days = $criteria['days'];
+    $latestyear = $criteria['latestyear'];
     $disabled = $criteria['disabled'];
     $do_contact = $criteria['do_contact'];
     $num_criteria = $criteria['num_criteria'];
@@ -300,6 +306,7 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
   if (array_key_exists('tripal_pub_import', $_SESSION)) {
     $remote_db = $_SESSION['tripal_pub_import']['remote_db'];
     $days = $_SESSION['tripal_pub_import']['days'];
+    $latestyear = $_SESSION['tripal_pub_import']['latestyear'];
     $disabled = $_SESSION['tripal_pub_import']['disabled'];
     $do_contact = $_SESSION['tripal_pub_import']['do_contact'];
     $num_criteria = $_SESSION['tripal_pub_import']['num_criteria'];
@@ -315,21 +322,23 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
   // if we are re constructing the form from a failed validation or ajax callback
   // then use the $form_state['values'] values
   if (array_key_exists('values', $form_state)) {
-    $remote_db = $form_state['values']['remote_db'];
-    $days = $form_state['values']['days'];
-    $disabled = $form_state['values']['disabled'];
-    $do_contact = $form_state['values']['do_contact'];
-    $num_criteria = $form_state['values']['num_criteria'];
-    $loader_name = $form_state['values']['loader_name'];
+    $remote_db = $form_state['values']['remote_db'] ?? null;
+    $days = $form_state['values']['days'] ?? null;
+    $latestyear = $form_state['values']['latestyear'] ?? null;
+    $disabled = $form_state['values']['disabled'] ?? null;
+    $do_contact = $form_state['values']['do_contact'] ?? null;
+    $num_criteria = $form_state['values']['num_criteria'] ?? null;
+    $loader_name = $form_state['values']['loader_name'] ?? null;
   }
   // if we are re building the form from after submission (from ajax call) then
   // the values are in the $form_state['input'] array
   if (array_key_exists('input', $form_state) and !empty($form_state['input'])) {
-    $remote_db = $form_state['input']['remote_db'];
-    $days = $form_state['input']['days'];
-    $disabled = $form_state['input']['disabled'];
-    $do_contact = $form_state['input']['do_contact'];
-    $loader_name = $form_state['input']['loader_name'];
+    $remote_db = $form_state['input']['remote_db'] ?? null;
+    $days = $form_state['input']['days'] ?? null;
+    $latestyear = $form_state['input']['latestyear'] ?? null;
+    $disabled = $form_state['input']['disabled'] ?? null;
+    $do_contact = $form_state['input']['do_contact'] ?? null;
+    $loader_name = $form_state['input']['loader_name'] ?? null;
 
     // because the num_criteria is a value and not a visible or hidden form
     // element it is not part of the ['input'] array, so we need to get it from the form
@@ -381,11 +390,7 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
   if (!$remote_db) {
     $remote_db = 'PMID';
   }
-  /**
-   * Removing AGL option until we get it fixed
-   * spf 6/22/2018
-   */
-  unset($remote_dbs['AGL']);
+
   $form['themed_element']['remote_db'] = [
     '#title' => t('Source'),
     '#type' => 'select',
@@ -405,6 +410,14 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
     '#default_value' => $days,
     '#size' => 5,
   ];
+  // AGL only, this field will be removed for the pubmed loader
+  $form['themed_element']['latestyear'] = [
+    '#type' => 'textfield',
+    '#title' => t('Latest year of publication'),
+    '#description' => t('Filter returned publications for those that have been published no later than this year.'),
+    '#default_value' => $latestyear,
+    '#size' => 5,
+  ];
   $form['themed_element']['disabled'] = [
     '#type' => 'checkbox',
     '#title' => t('Disabled'),
@@ -451,6 +464,70 @@ function tripal_pub_importer_setup_form($form, &$form_state = NULL, $pub_import_
   return $form;
 }
 
+/**
+ * The form used for setting the optional NCBI API key.
+ *
+ * @param $form
+ *   The form element to be populated.
+ * @param $form_state
+ *   The state of the form element to be populated.
+ *
+ * @return array
+ *   The populated form element.
+ *
+ * @ingroup tripal_pub
+ */
+function tripal_pub_importer_ncbi_api_key_form($form, $form_state) {
+  $description = t('Tripal imports publications using NCBI\'s ')
+    . l('EUtils API', 'https://www.ncbi.nlm.nih.gov/books/NBK25500/')
+    . t(', which limits users and programs to a maximum of 3 requests per second without an API key. '
+        . 'However, NCBI allows users and programs to an increased maximum of 10 requests per second if '
+        . 'they provide a valid API key. This is particularly useful in speeding up large publication imports. '
+        . 'For more information on NCBI API keys, please ')
+    . l('see here', 'https://www.ncbi.nlm.nih.gov/books/NBK25497/#chapter2.Coming_in_December_2018_API_Key', array(
+      'attributes' => array(
+        'target' => 'blank',
+      ),
+    )) . '.';
+
+  $form['ncbi_api_key'] = array(
+    '#type' => 'textfield',
+    '#title' => t('(Optional) NCBI API key:'),
+    '#description' => $description,
+    '#default_value' => variable_get('tripal_pub_importer_ncbi_api_key', NULL),
+    '#ajax' => array(
+      'callback' => 'tripal_pub_importer_set_ncbi_api_key',
+      'wrapper' => 'ncbi_api_key',
+    ),
+    '#prefix' => '<div id="ncbi_api_key">',
+    '#suffix' => '</div>',
+  );
+
+  return $form;
+}
+
+/**
+ * This function saves the NCBI API key to the database.
+ *
+ * It is called when the user makes a change to the NCBI API key field and then
+ * moves their cursor out of the field.
+ *
+ * @param $form
+ *   The new form element.
+ * @param $form_state
+ *   The state of the new form element.
+ *
+ * @return array
+ *   The new api key field.
+ *
+ * @ingroup tripal_pub
+ */
+function tripal_pub_importer_set_ncbi_api_key($form, $form_state) {
+  variable_set('tripal_pub_importer_ncbi_api_key', check_plain($form_state['values']['ncbi_api_key']));
+  drupal_set_message('NCBI API key has been saved successfully!');
+  return $form['ncbi_api_key'];
+}
+
 /**
  * A helper function for the importer setup form that adds the criteria to
  * the form that belong to the importer.
@@ -501,10 +578,14 @@ function tripal_pub_importer_setup_add_criteria_fields(&$form, &$form_state, $nu
 
     // if we have criteria supplied from the database then use that as the initial defaults
     if ($criteria) {
-      $search_terms = $criteria['criteria'][$i]['search_terms'];
-      $scope = $criteria['criteria'][$i]['scope'];
-      $is_phrase = $criteria['criteria'][$i]['is_phrase'];
-      $operation = $criteria['criteria'][$i]['operation'];
+      if (array_key_exists('criteria', $criteria)) {
+        if (array_key_exists($i, $criteria['criteria'])) {
+          $search_terms = $criteria['criteria'][$i]['search_terms'];
+          $scope = $criteria['criteria'][$i]['scope'];
+          $is_phrase = $criteria['criteria'][$i]['is_phrase'];
+          $operation = $criteria['criteria'][$i]['operation'];
+        }
+      }
     }
 
     // if the criteria comes the session
@@ -516,12 +597,12 @@ function tripal_pub_importer_setup_add_criteria_fields(&$form, &$form_state, $nu
     }
 
     // If the form_state has variables then use those.  This happens when an error occurs on the form or the
-    // form is resbumitted using AJAX
+    // form is resubmitted using AJAX
     if (array_key_exists('values', $form_state)) {
-      $search_terms = $form_state['values']["search_terms-$i"];
-      $scope = $form_state['values']["scope-$i"];
-      $is_phrase = $form_state['values']["is_phrase-$i"];
-      $operation = $form_state['values']["operation-$i"];
+      $search_terms = $form_state['values']["search_terms-$i"] ?? null;
+      $scope = $form_state['values']["scope-$i"] ?? null;
+      $is_phrase = $form_state['values']["is_phrase-$i"] ?? null;
+      $operation = $form_state['values']["operation-$i"] ?? null;
     }
     $form['themed_element']['criteria'][$i]["scope-$i"] = [
       '#type' => 'select',
@@ -641,6 +722,7 @@ function tripal_pub_importer_setup_form_validate($form, &$form_state) {
   $num_criteria = $form_state['values']['num_criteria'];
   $remote_db = $form_state['values']["remote_db"];
   $days = trim($form_state['values']["days"]);
+  $latestyear = trim($form_state['values']["latestyear"]);
   $disabled = $form_state['values']["disabled"];
   $do_contact = $form_state['values']["do_contact"];
   $loader_name = trim($form_state['values']["loader_name"]);
@@ -666,6 +748,10 @@ function tripal_pub_importer_setup_form_validate($form, &$form_state) {
     form_set_error("days", "Please enter a numeric, non decimal value, for the number of days.");
     $_SESSION['tripal_pub_import']['perform_search'] = 0;
   }
+  if ($latestyear and !is_numeric($latestyear) or preg_match('/\./', $latestyear)) {
+    form_set_error("latestyear", "Please enter a numeric, non decimal value, for latestyear.");
+    $_SESSION['tripal_pub_import']['perform_search'] = 0;
+  }
   // allow the selected remote database to validate any changes to the form if needed
   $callback = "tripal_pub_remote_validate_form_$remote_db";
   $form = call_user_func($callback, $form, $form_state);
@@ -682,6 +768,7 @@ function tripal_pub_importer_setup_form_submit($form, &$form_state) {
   $num_criteria = $form_state['values']['num_criteria'];
   $remote_db = $form_state['values']["remote_db"];
   $days = trim($form_state['values']["days"]);
+  $latestyear = trim($form_state['values']["latestyear"]);
   $loader_name = trim($form_state['values']["loader_name"]);
   $disabled = $form_state['values']["disabled"];
   $do_contact = $form_state['values']["do_contact"];
@@ -689,6 +776,7 @@ function tripal_pub_importer_setup_form_submit($form, &$form_state) {
   // set the session variables
   $_SESSION['tripal_pub_import']['remote_db'] = $remote_db;
   $_SESSION['tripal_pub_import']['days'] = $days;
+  $_SESSION['tripal_pub_import']['latestyear'] = $latestyear;
   $_SESSION['tripal_pub_import']['num_criteria'] = $num_criteria;
   $_SESSION['tripal_pub_import']['loader_name'] = $loader_name;
   $_SESSION['tripal_pub_import']['disabled'] = $disabled;
@@ -795,6 +883,10 @@ function theme_tripal_pub_importer_setup_form_elements($variables) {
   $markup .= '</div>';
   $markup .= '<div id="pub-search-form-row1" style="clear:both">';
   $markup .= '  <div id="pub-search-form-row1-col1">' . drupal_render($form['days']) . '</div>';
+  // latest year field is used ony for AGL importer
+  if ($variables['form']['remote_db']['#value'] == 'AGL') {
+    $markup .= '  <div id="pub-search-form-row1-col2">' . drupal_render($form['latestyear']) . '</div>';
+  }
   $markup .= '</div>';
   $markup .= '<div id="pub-search-form-row2">' . drupal_render($form['disabled']) . '</div>';
   $markup .= '<div id="pub-search-form-row3">' . drupal_render($form['do_contact']) . '</div>';
@@ -930,46 +1022,53 @@ function tripal_pub_add_publications($pubs, $do_contact, $update = FALSE, $job =
     $memory = number_format(memory_get_usage()) . " bytes";
     print "Processing $i of $total_pubs. Memory usage: $memory.\r";
 
-    // add the publication to Chado
-    $action = '';
-    $pub_id = tripal_pub_add_publication($pub, $action, $do_contact, $update, $job);
-    if ($pub_id) {
-      // add the publication cross reference (e.g. to PubMed)
-      if ($pub_id and $pub['Publication Dbxref']) {
-        $dbxref = [];
-        if (preg_match('/^(.*?):(.*?)$/', trim($pub['Publication Dbxref']), $matches)) {
-          $dbxref['db_name'] = $matches[1];
-          $dbxref['accession'] = $matches[2];
-        }
-        else {
-          tripal_report_error($message_type, TRIPAL_ERROR,
-            'Unable to extract the dbxref to be associated with the publication (pub ID=@pub_id) from @dbxref. This reference should be [database-name]:[accession]',
-            [
-              '@pub_id' => $pub_id,
-              '@dbxref' => $pub['Publication Dbxref'],
-              $message_opts,
-            ]
-          );
+    // implementation of year limits for AGL uses a 'passfilter' flag
+    if ((!array_key_exists('passfilter', $pub)) or ($pub['passfilter'] == 1 )) {
+
+      // add the publication to Chado
+      $action = '';
+      $pub_id = tripal_pub_add_publication($pub, $action, $do_contact, $update, $job);
+      // $pub_id will be null if publication already existed
+      if ($pub_id) {
+        // add the publication cross reference (e.g. to PubMed)
+        if ($pub_id and $pub['Publication Dbxref']) {
+          $dbxref = [];
+          if (preg_match('/^(.*?):(.*?)$/', trim($pub['Publication Dbxref']), $matches)) {
+            $dbxref['db_name'] = $matches[1];
+            $dbxref['accession'] = $matches[2];
+          }
+          else {
+            tripal_report_error($message_type, TRIPAL_ERROR,
+              'Unable to extract the dbxref to be associated with the publication (pub ID=@pub_id) from @dbxref. This reference should be [database-name]:[accession]',
+              [
+                '@pub_id' => $pub_id,
+                '@dbxref' => $pub['Publication Dbxref'],
+                $message_opts,
+              ]
+            );
+          }
+          $pub_dbxref = tripal_associate_dbxref('pub', $pub_id, $dbxref);
         }
-        $pub_dbxref = tripal_associate_dbxref('pub', $pub_id, $dbxref);
+        $pub['pub_id'] = $pub_id;
       }
-      $pub['pub_id'] = $pub_id;
-    }
 
-    switch ($action) {
-      case 'error':
-        $report['error'][] = $pub['Citation'];
-        break;
-      case 'inserted':
-        $report['inserted'][] = $pub['Citation'];
-        break;
-      case 'updated':
-        $report['updated'][] = $pub['Citation'];
-        break;
-      case 'skipped':
-        $report['skipped'][] = $pub['Citation'];
-        break;
+      switch ($action) {
+        case 'error':
+          $report['error'][] = $pub['Citation'];
+          break;
+        case 'inserted':
+          $report['inserted'][] = $pub['Citation'];
+          break;
+        case 'updated':
+          $report['updated'][] = $pub['Citation'];
+          break;
+        case 'skipped':
+          $report['skipped'][] = $pub['Citation'];
+          break;
+      }
     }
+    // else pub failed AGL year filter
+    else { $report['skipped'][] = $pub['Citation']; }
     $i++;
   }
   return $report;
@@ -1017,7 +1116,7 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
   $pub_id = 0;
 
   // These are options for the tripal_report_error function. We do not
-  // want to log messages to the watchdog except for errors and to the job and 
+  // want to log messages to the watchdog except for errors and to the job and
   // to the terminal
   $message_type = 'pub_import';
   $message_opts = [
@@ -1035,7 +1134,7 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
     return FALSE;
   }
 
-  // Before proceeding check to see if the publication already exists. If there 
+  // Before proceeding check to see if the publication already exists. If there
   // is only one match and the $update_if_exists is NOT set then return FALSE.
   $pub_ids = chado_publication_exists($pub_details);
 
@@ -1053,7 +1152,7 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
     return FALSE;
   }
 
-  // If we have more than one matching pub then return an error as we don't 
+  // If we have more than one matching pub then return an error as we don't
   // know which to update even if update_if_exists is set to TRUE.
   if (count($pub_ids) > 1) {
     tripal_report_error($message_type, TRIPAL_NOTICE,
@@ -1104,7 +1203,7 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
     return FALSE;
   }
 
-  // The series name field in the pub table is only 255 characters, so we 
+  // The series name field in the pub table is only 255 characters, so we
   // should trim just in case.
   $series_name = '';
   if (array_key_exists('Series_Name', $pub_details)) {
@@ -1158,8 +1257,8 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
     }
   }
 
-  // Before we add any new properties we need to remove those that are there 
-  // if this is an update.  The only thing we don't want to remove are the 
+  // Before we add any new properties we need to remove those that are there
+  // if this is an update.  The only thing we don't want to remove are the
   // 'Publication Dbxref'.
   if ($update_if_exists) {
     $sql = "
@@ -1178,12 +1277,17 @@ function tripal_pub_add_publication($pub_details, &$action, $do_contact = FALSE,
   // Iterate through the properties and add them.
   foreach ($pub_details as $key => $value) {
 
-    // The pub_details may have the raw search data (e.g. in XML from PubMed. 
+    // The pub_details may have the raw search data (e.g. in XML from PubMed.
     // We'll irgnore this for now.
     if ($key == 'raw') {
       continue;
     }
 
+    // Filtering flag for AGL, not a property to add here
+    if ($key == 'passfilter') {
+      continue;
+    }
+
     // Since we're not updating the 'Publication Dbxref' on an update
     // skip this property.
     if ($update_if_exists and $key == 'Publication Dbxref') {

+ 1 - 1
tripal_ws/tripal_ws.module

@@ -39,7 +39,7 @@ require_once "includes/TripalFields/WebServicesFieldFormatter.inc";
 function tripal_ws_init() {
   global $base_url;
 
-  $api_url = $base_url . '/web-sevices/';
+  $api_url = $base_url . '/web-services/';
 
   $vocab = tripal_get_vocabulary_details('hydra');