Merge pull request #697 from tripal/689-tv3-docs

Functional Docs for the User's Guide:  Setup and Blast.

docs/user_guide.rst

@@ -10,9 +10,9 @@ User's Guide
    user_guide/what_is_tripal
    user_guide/install_tripal
    user_guide/drupal_overview
-   user_guide/example_genomics
    user_guide/learn_chado
    user_guide/content_types
+   user_guide/example_genomics
    user_guide/mviews
    user_guide/searching
    user_guide/job_management

docs/user_guide/example_genomics.rst

@@ -14,3 +14,4 @@ The following tutorial will walk you through creating content and loading genomi
    ./example_genomics/controlled_vocabs
    ./example_genomics/genomes_genes
    ./example_genomics/pub_import
+   ./example_genomics/func_annots

docs/user_guide/example_genomics/func_annots.rst

@@ -0,0 +1,14 @@
+Functional Annotations
+======================
+
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Table of Contents
+   :glob:
+
+   func_annots/setup
+   func_annots/blast
+   func_annots/interpro
+   func_annots/kegg
+   func_annots/go

docs/user_guide/example_genomics/func_annots/blast.rst

@@ -0,0 +1,175 @@
+Adding BLAST Results
+====================
+.. note::
+
+  Remember you must set the ``$DRUPAL_HOME`` environment variable if you want to cut-and-paste the commands below. See :doc:`../../install_tripal/drupal_home`
+
+
+Adding BLAST Databases
+----------------------
+
+Before we load our BLAST results we want to add some external databases.  For this tutorial we have protein BLAST results against NCBI nr and ExPASy SwissProt.  We would like the BLAST hits to be clickable such that they link back to their respective databases. To do this, we must add some additional databases.  Navigate to **Tripal → Data Loaders → Chado Databases** and click the link titled **Add a Database**. The resulting page provides fields for adding a new database.  Add two new databases, one for NCBI nr and the other for ExPASy SwissProt.
+
+Use these values for adding the NCBI nr database:
+
+.. csv-table::
+  :header: "Field Name", "Value"
+
+  "Name", "NCBI nr"
+  "Description", "NCBI's non-redundant protein database"
+  "URL", "http://www.ncbi.nlm.nih.gov/"
+  "URL Prefix", "http://www.ncbi.nlm.nih.gov/protein/"
+
+Use these values for adding the SwssProt database:
+
+.. csv-table::
+  :header: "Field Name", "Value"
+
+  "Name", "ExPASy Swiss-Prot"
+  "Description", "A curated protein sequence database which strives to provide a high level of annotation, a minimal level of redundancy and high level of integration with other databases"
+  "URL", "http://expasy.org/sprot/"
+  "URL prefix", "http://www.uniprot.org/uniprot/"
+
+
+Configure Parsing of BLAST Results
+----------------------------------
+First, we need to ensure that the BLAST module can properly parse the BLAST hits. To do this, navigate to **Tripal → Extensions → Tripal Blast Analyses**. On this page are configuration settings for the Tripal BLAST Analysis extension module.
+
+Within the section titled BLAST Parsing, you can specify a different, more meaningful name for the sequence library file (a.k.a. database) used for BLASTing. This name will be displayed with BLAST results. You can also provide regular expressions for parsing BLAST hits. For example, the following is a line for a match from SwissProt:
+
+  ::
+
+    sp|P43288|KSG1_ARATH Shaggy-related protein kinase alpha OS=Arabidopsis thaliana GN=ASK1 PE=2 SV=3
+
+
+Here the hit name is ``KSG1_ARATH``, the accession is ``P43288``, the hit description is ``Shaggy-related protein kinase alpha OS=Arabidopsis thaliana`` and the organism is ``Arabidopsis thaliana``. We need regular expressions to tell Tripal how to extract these unique parts from the match text. Because Tripal is a PHP application, the syntax for regular expressions follows the PHP method. Documentation for regular expressions used in PHP can be found here. The following regular expressions can be used to extract the hit name, the accession, hit description and organism for the example SwissProt line above:
+
+.. csv-table::
+  :header: "Element", "Regular Expression"
+
+  "Hit Name", ``^sp\|.*?\|(.*?)\s.*?$``
+  "Hit Description", ``^sp\|.*?\|.*?\s(.*)$``
+  "Hit Accession", ``^sp\|(.*?)\|.*?\s.*?$``
+  "Hit Organism", ``^.*?OS=(.*?)\s\w\w=.*$``
+
+In this tutorial, we will be adding BLAST results for the two databases we just created: ExPASy SwissProt and NCBI nr. First, select ExPASy SwissProt from the drop-down menu. A form will appear:
+
+.. image:: blast1.png
+
+In the form fields, add the following values:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+  "Title for the BLAST analysis", "(leave blank)"
+  "Regular expression for Hit Name", ``^sp\|.*?\|(.*?)\s.*?$``
+  "Regular expression for Hit Description", ``^sp\|.*?\|.*?\s(.*)$``
+  "Regular expression for Hit Accession:", ``^sp\|(.*?)\|.*?\s.*?$``
+  "Regular expression for Organism", ``^.*?OS=(.*?)\s\w\w=.*$``
+  "Organism Name", "(leave blank)"
+
+Click **Save Settings**.
+
+The match accession will be used for building web links to the external database. The accession will be appended to the URL Prefix set earlier when the database record was first created.
+
+Now select the **NCBI nr** database from the drop-down. NCBI databases use a format that is compatible with BLAST. Therefore, the hit name, accession and description are handled differently in the BLAST XML results. To correctly parse results from an NCBI database click the **Use Genbank style parser** checkbox. This should disable all other fields and is all we need for this database.  Clikc the Save Settings button.
+
+Create the Analysis Page
+------------------------
+
+.. note::
+
+  It is always recommended to create an analysis page anytime you import data. The purpose of the analysis page is to describe how the data being added was derived or collected.
+
+Now we can create our analysis page. Navigate to **Content → Tripal Content** and click the **Add Tripal Content** link. This page contains a variety of content types that the site supports.  Scroll to the **Other** section and find the content type named **Blast Results**:
+
+.. image:: blast2.png
+
+Here we can save details about the analysis used to create the BLAST results.  Enter the following in the fields that appear on the page:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+    "Name", "blastx Citrus sinensis v1.0 genes vs ExPASy SwissProt"
+    "Description", "Citrus sinensis mRNA sequences were BLAST'ed against the ExPASy SwissProt protein database using a local installation of BLAST on in-house linux server. Expectation value was set at 1e-6"
+    "BLAST Program", "blastx"
+    "BLAST Version", "2.2.25"
+    "Data Source Name ", "Citrus sinensis mRNA vs ExPASy SwissProt"
+    "Date Performed", "(today's date)"
+
+Click the **Save** button. You can now see our new BLAST analysis page.
+
+.. image:: blast3.png
+
+Create a second Analysis page for the results of the NCBI nr BLAST analysis. Use the following values:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+    "Name", "blastx Citrus sinensis v1.0 genes vs NCBI nr"
+    "Description", "Citrus sinensis mRNA sequences were BLAST'ed against the NCBI non-redundant protein database using a local installation of BLAST on in-house linux server. Expectation value was set at 1e-6"
+    "BLAST Program", "blastx"
+    "BLAST Version", "2.2.25"
+    "Data Source Name ", "Citrus sinensis mRNA vs NCBI nr"
+    "Date Performed", "(today's date)"
+
+
+Import the BLAST XML results
+----------------------------
+First, we will load BLAST results for our citrus gene vs ExPASy SwissProt.  Now that we have our database records setup and configured and we have our analysis record created, we are ready to import the blast results.  To do this, navigate to **Tripal > Data Loaders > Chado BLAST XML results loader**.  The following page will be presented:
+
+.. image:: blast4.png
+
+The top section of this page provides multiple methods for providing results file: via an upload interface, specifying a remote URL or a file path that is local to the server.  Most likely, you will always upload or provide a remote URL.  However, we download the file earlier, and stored them here: ``$DRUPAL_HOME/sites/default/files``.  So, in this case we can use the path on the local server.  Provide the following value for this form:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+  "Server path", "sites/default/files/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out"
+  "Analysis", "blastx Citrus sinensis v1.0 genes vs ExPASy SwissProt (blastall 2.2.25, Citrus sinensis mRNA vs ExPASy SwissProt)"
+  "Database", "ExPASy SwissProt"
+  "BLAST XML File Extension", "out"
+  "Query Type", "mRNA"
+
+.. note::
+
+  For the **Server path** we need not give the full path.  Because we downloaded the files into the Drupal directory we can leave off any preceding path and Tripal will resolve the path.  Otherwise we could provide the full path.
+
+.. note::
+
+  Specifying **ExPASy SwissProt** as the database will allow the importer to use the database configuration settings we entered earlier.
+
+Clicking the **Import BLAST file** will add a job which we can manually execute with the following command:
+
+::
+
+  drush trp-run-jobs --username=administrator --root=$DRUPAL_HOME
+
+The results should now be loaded. Now we want to add the results for NCBI nr. Repeat the steps above to add a new analysis with the following details:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+  "Server path", "sites/default/files/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out"
+  "Analysis", "blastx Citrus sinensis v1.0 genes vs ExPASy SwissProt (blastall 2.2.25, Citrus sinensis mRNA vs NCBI nr)"
+  "Database", "ExPASy SwissProt"
+  "BLAST XML File Extension", "out"
+  "Query Type", "mRNA"
+
+Click the Save button and manually run the job:
+
+::
+
+  drush trp-run-jobs --username=administrator --root=$DRUPAL_HOME
+
+To view results we must find the mRNA that has BLAST hits.  For this example, click on the **mRNA Search** link in the **Data Search** block.  Search for the mRNA named `orange1.1g015615m`.  Viewing the page, we should now see BLAST results by clicking the 'BLAST results' link in the left table of contents.
+
+.. image:: blast5.png
+
+Notice, that when viewing the results, the SwissProt matches are links.  When clicked they redirect the user to the SwissProt website where users can find more information about that protein.
+
+.. image:: blast6.png
+
+.. note::
+
+  The match links are able to link out to SwissProt and NCBI because of the initial setup where we added the database settings and we set regular expressions for parsing the match accessions.

docs/user_guide/example_genomics/func_annots/go.rst

@@ -0,0 +1,2 @@
+Managing Gene Ontology Annotations
+=================================

docs/user_guide/example_genomics/func_annots/interpro.rst

@@ -0,0 +1,80 @@
+Adding InterProScan Results
+===========================
+
+.. note::
+
+  Remember you must set the ``$DRUPAL_HOME`` environment variable if you want to cut-and-paste the commands below. See :doc:`../../install_tripal/drupal_home`
+
+For this tutorial, these results were obtained by using a local installation of InterProScan installed on a computational cluster. However, you may choose to use Blast2GO or the online InterProScan utility. Results should be saved in ``XML`` format.
+
+
+What is InterProScan?
+---------------------
+To learn more about InterProScan, please visit https://www.ebi.ac.uk/interpro/
+
+
+Create the Analysis Page
+-------------------------
+
+  .. note::
+
+    It is always recommended to create an analysis page anytime you import data. The purpose of the analysis page is to describe how the data being added was derived or collected.
+
+Tripal defines the **InterPro Results** Bundle, which is a specific type of Chado analysis.  Create a new record by going to ``Content -> Tripal Content -> Add Tripal Content --> InterPro Results``.
+
+Fill out the fields as described in the table below.
+
+
+.. csv-table::
+  :header: "Field", "Value"
+
+  "Name", "InterPro Annotations of C. sinensis v1.0"
+  "InterPro Program", "InterProScan"
+  "InterPro Version", "4.8"
+  "Date Performed", "Current Date"
+  "Data Source Name", "C. sinensis v1.0 mRNA"
+  "Data Source Version", "v1.0"
+  "Data Source URI", "n/a"
+  "Description", "Materials & Methods: C. sinensis mRNA sequences were mapped to IPR domains and GO terms using a local installation of InterProScan executed on a computational cluster. InterProScan date files used were MATCH_DATA_v32, DATA_v32.0 and PTHR_DATA v31.0."
+
+Press the **Save** button.
+
+Import the InterProScan XML results
+------------------------------------
+
+
+Next, we will load InterProScan results for our citrus gene.  To do this, navigate to **Tripal > Data Loaders > Chado InterProScan XML results loader**.  The following page will be presented:
+
+.. image:: interpro1.png
+
+The top section of this page provides multiple methods for providing results file: via an upload interface, specifying a remote URL or a file path that is local to the server.  Most likely, you will always upload or provide a remote URL.  However, we downloaded the files earlier, and stored them here: ``$DRUPAL_HOME/sites/default/files``.  So, in this case we can use the path on the local server.  Provide the following value for this form:
+
+.. csv-table::
+  :header: "Field", "Value"
+
+  "Server path", "sites/default/files/Citrus_sinensis-orange1.1g015632m.g.iprscan.xml"
+  "Analysis", "InterPro Annotations of C. sinensis v1.0"
+  'Load GO terms to the database', 'unchecked'
+  "Query Name RE", ""
+  "Use Unique Name", "unchecked"
+  "Query Type", "mRNA"
+
+In order for GO terms to be imported, the Gene Ontology must be loaded on your site: for this tutorial, we leave the box unchecked.
+
+
+.. note::
+
+  For the **Server path** we need not give the full path.  Because we downloaded the files into the Drupal directory we can leave off any preceding path and Tripal will resolve the path.  Otherwise we could provide the full path.
+
+
+
+Clicking the **Import InterProScan file** will add a job which we can manually execute with the following command:
+
+::
+
+    drush trp-run-jobs --username=administrator --root=$DRUPAL_HOME
+
+
+After the job is run, our InterPro field will be populated on the mRNA page with an annotation diagram:
+
+.. image:: interpro2.png

docs/user_guide/example_genomics/func_annots/kegg.rst

@@ -0,0 +1,2 @@
+Adding KEGG Results
+===================

docs/user_guide/example_genomics/func_annots/setup.rst

@@ -0,0 +1,73 @@
+Module Setup
+============
+.. note::
+
+  Remember you must set the ``$DRUPAL_HOME`` environment variable if you want to cut-and-paste the commands below. See :doc:`../../install_tripal/drupal_home`
+  
+  
+For this example we will be load functional data for our gene. To do this we will use the Blast, KEGG, and InterPro extension modules. However, these extension modules are not part of the "core" Tripal package but are available as separate extensions.  Anyone may create extensions for Tripal.  These extensions are useful for genomic data and therefore are included in this tutorial. 
+
+To download these modules:
+
+  ::
+  
+    cd $DRUPAL_HOME    
+    drush pm-download tripal_analysis_blast
+    drush pm-download tripal_analysis_kegg
+    drush pm-download tripal_analysis_interpro
+
+Now, enable these extension modules:
+
+  ::
+  
+    drush pm-enable tripal_analysis_blast
+    drush pm-enable tripal_analysis_interpro
+    drush pm-enable tripal_analysis_kegg
+
+For this example, we will use the following files which are available for downloading:
+
+- `Citrus_sinensis-orange1.1g015632m.g.iprscan.xml <http://www.gmod.org/mediawiki/images/0/0c/Citrus_sinensis-orange1.1g015632m.g.iprscan.xml>`_
+- `Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz <http://www.gmod.org/mediawiki/images/1/13/Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz>`_
+- `Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out <http://www.gmod.org/mediawiki/images/e/e8/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out>`_
+- `Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out <http://www.gmod.org/mediawiki/images/2/24/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out>`_
+
+Download these files to the ```$DRUPAL_HOME/sites/default/files``` directory. To do so quickly run these commands:
+
+  ::
+  
+    cd $DRUPAL_HOME/sites/default/files
+    wget http://www.gmod.org/mediawiki/images/0/0c/Citrus_sinensis-orange1.1g015632m.g.iprscan.xml
+    wget http://www.gmod.org/mediawiki/images/1/13/Citrus_sinensis-orange1.1g015632m.g.KEGG.heir.tar.gz
+    wget http://www.gmod.org/mediawiki/images/e/e8/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_uniprot_sprot.fasta.out
+    wget http://www.gmod.org/mediawiki/images/2/24/Blastx_citrus_sinensis-orange1.1g015632m.g.fasta.0_vs_nr.out
+
+Each of these modules provides new fields for both the **gene** and **mRNA** content types.  To add these new field to those content types, navigate to **Structure > Tripal Content Types** and click the **manage fields** link in the row for the **mRNA** content type.  Click the link titled **Check for new fields**.  After a few moments the page will refresh and you will be notified that new fields have been added.
+
+.. image:: setup1.png
+
+Next, we need to position the new field. Using the skills you learned in the :doc:`../../content_types/configuring_page_display` Create three new **Tripal Panes** named:
+
+- Blast Results
+- Protein Domains
+- KEGG Pathways
+
+Be sure to:
+
+- Place the three new fields into each pane respectively 
+- Move the Panes out of the **disabled** section. 
+- Set the label for each field to **Hidden**.
+
+The following shows an example of this layout:
+
+.. image:: setup2.png
+
+The fields are now ready for display once data is added!  
+
+.. note::
+
+   If you want both the **Gene** and **mRNA** content type to have BLAST, InterPro and KEGG result fields you must repeat the steps above for both.
+
+.. note::
+
+  Anytime you install a Tripal v3 extension module you should check for new fields, and then place those fields in the layout.  Extension modules often will not do this for you because they do not assume you want these new fields.
+  

docs/user_guide/example_genomics/organisms.rst

@@ -48,7 +48,7 @@ Click the checbox beside the 'Import taxonomy for existing species' and click Su
 
 ::
 
-  drush trp-run-jobs --username=administrator --root=/var/www/html
+  drush trp-run-jobs --username=administrator --root=$DRUPAL_HOME
 
 You will see the following output: