GenePattern provides access to a broad array of computational methods used to analyze genomic data. Its extendable architecture makes it easy for computational biologists to add analysis and visualization modules, which ensures that GenePattern users have access to new computational methods on a regular basis.
If you are new to GenePattern, begin with the basics:
This guide assumes that you are a programmer and familiar with GenePattern. As a programmer, you generally work with GenePattern in one of two ways:
Only the GenePattern team can create, edit, or install modules on the GenePattern public server. Therefore, to create a module, you must have a local GenePattern server installed (see Starting Your Own GenePattern Server).
Creating a GenePattern module is a multi-step process:
Malicious code: By adding a module, a user can execute arbitrary code on the GenePattern server. Because arbitrary code may include malicious code, take precautions to protect your server: for example, employ virus scanner software and restrict access to appropriately privileged (non-root) users. For more information about securing your server, see Securing the Server.
To create a module in GenePattern:
If no errors are found, GenePattern copies the support files to the server and makes the module available to the GenePattern clients.
The in-depth article Creating a GenePattern Module provides a hands-on tutorial that walks you through the process of creating a GenePattern module.
Modules can have end-user license agreements (EULAs) attached to them. In order to run a licensed module (either standalone or within a pipeline), users must view and accept the license terms; GenePattern will display the license and prompt the user to accept it. After a user accepts a licensed module's terms, that user can run the module and GenePattern will not prompt the user to re-accept the license on subsequent runs. However, a user will need to re-accept the license whenever the licensed module is updated to a new version. The GenePattern Server records each acceptance of a licensed module's terms and reports these records to a database application hosted at the Broad Institute. If this module will only be on your GenePattern server and you need to maintain your own database for the terms agreements, contact the GenePattern development team for assistance.
Academic users can agree to the license and use the module. Commercial users, however, need to contact the GenePattern team to work out terms for use, particularly if commercial users are establishing their own GenePattern server on which to host the licensed module.
The EULA terms need to be in a text document (.txt). You can add a EULA to your module in the module integrator by clicking Add License File.
To edit a module:
When you create or edit a module, GenePattern displays its properties in the module integrator. Click the Help icon to display descriptions of each property and its valid values. The help text is provided here for your convenience.
When you create or edit a module, GenePattern displays its properties in the module integrator. Click the Help icon to display the following descriptions of each property and its valid values.
Note: Only the GenePattern team can create, edit or install modules on the GenePattern public server. Therefore, to create a module, you must have a local GenePattern server installed.
Creating a GenePattern module is a multi-step process:
When you save your changes, the module properties that you have entered are validated as follows:
If everything checks out, the uploaded files are saved in the GenePattern module library and the module registered in the module database. The module and its uploaded files are indexed in the background so that they are available for searching. You can run the module immediately and can share it with others.
The following sections describe each module property in detail:
An example for each property is given based on the Consensus Clustering module, which may be uploaded from the module repository if you haven't already installed it.
The name of the module will be used in the drop-down module catalog lists and as a directory name on the server with this name. It should be a short but descriptive name, without spaces or punctuation, and may be mixed upper- and lower-case.
ConsensusClustering example: ConsensusClustering
Each time you update a module, you create a new version of the module. Typically, you want to edit the most recent version of a module. If you want to edit an earlier version, select that version from the drop-down list of versions.
Click Help to display this text.
Click Save to save your changes, creating a new version of the module, and remain in the module integrator.
Click Save and Run to save your changes, creating a new version of the module, exit from the module integrator and run the module.
The Life Science Identifier (LSID) used to uniquely identify a GenePattern module. You cannot create or edit LSIDS. They are created automatically by the GenePattern server when a module is saved.
ConsensusClustering example: urn:lsid:broad.mit.edu:cancer.software.genepattern.module.analysis:00030:5
The description is where to explain what your module does, and why someone would want to use it. It can be anywhere from a sentence to a short paragraph in length. The description, sometimes in abridged form, is displayed in the pipeline designer module choice list, in generated code when creating scripts from pipelines, and in the web client. It's a very good way for you to document succinctly why your module exists.
ConsensusClustering example: Resampling-based clustering method
Enter the author's name. If you share this module with others, they will know how to give the author credit and whom to contact with questions, suggestions, or enhancement ideas.
ConsensusClustering example: Stefano Monti
Enter the author's affiliation (company or academic institution). If you share this module with others, they will know how to give the author credit and whom to contact with questions, suggestions, or enhancement ideas.
ConsensusClustering example: Broad Institute
Upload a text file containing the End-User license agreement. Users will be prompted to accept this license before running the module.
Enter a brief description of the changes that you have made to the module. When GenePattern clients display a drop-down list of versions, the comments for each version are visible in the drop-down list.
ConsensusClustering example: Added ability to create heatmap images of clusters
On the GenePattern home page, modules and pipelines are organized by categories. Pipelines are always assigned to the category name pipeline. When you create/update a module, you can choose an existing category name or create a new category name. If your module fits into an existing category, such as Preprocess & Utilities, select that category from the drop-down list; otherwise, click the New button to add a new category. GenePattern creates the drop-down list of categories dynamically based on the categories of the modules installed on your GenePattern server. If you delete the last module in a given category, that category is removed from the drop-down list. ConsensusClustering example: Clustering
Modules may be marked as either public or private. When a module is first created, the default is to mark it private. When a module is first created, the default is to mark it private.
ConsensusClustering example: public
The quality level is a simple three-level classification that lets the user know what level of confidence the author has in the robustness of the module. In increasing order of quality expectations, they are: are "development", "preproduction", and "production". Although these terms have no strict definitions, they are useful for setting user expectations. If you make this module public, set the quality level appropriately.
ConsensusClustering example: production
If your module is compiled for a specific platform (Intel, Alpha, PowerPC, etc.), indicate that here. CPU requirements are enforced when the module is run.
ConsensusClustering example: any
If your module requires a specific operating system (Windows, Linux, MacOS, etc.), indicate that here. Operating system requirements are enforced when the module is run.
ConsensusClustering example: any
There is no specific language support or requirement enforcement at this time. However, by describing the primary language that a module is implemented in, you give some hints to the prospective user about their system requirements.
ConsensusClustering example: Java
If your module requires at least a certain revision of the language runtime environment(eg. 1.3.1_07), indicate that here. This is not currently enforced, but provides useful information to the prospective module user.
ConsensusClustering example: none specified
Select the file formats of the output files generated by your module. If your module generates an output file format not included in the list, click New to add that format to the list.
Any files required by your module, such as scripts, libraries, property files, DLLs, executable programs, etc. must be uploaded to the server. These files may be referenced in the command line field using the <libdir>filename nomenclature. There is no upper limit on the number of files which may be uploaded, assuming there is enough space.
Files that have been uploaded appear as links in this section. You may view or download them by clicking appropriately in your browser.
Help Files: Public modules should always include a help file that provides instructions for using the module, a detailed description of each input parameter, a detailed description of each output file (both its format and content), and either an explanation of the algorithm or a reference to the paper, journal or book that explains it.
When a user selects your module, GenePattern displays a form that includes the module parameters and a Help button. When the user clicks the Help button, GenePattern examines the list of support files for the module and displays the first file that has a standard documentation file extension. If no documentation file was provided, GenePattern displays a message indicating that no information is available. (By default, the standard documentation file extensions are html, htm, xhtml, pdf, rtf, and txt. You can modify this list of extensions by editing the files.doc property in the GenePattern /resources/genepattern.properties file.)
ConsensusClustering example: Current files: Acme.jar archiver.jar common_cmdline.jar ConsensusClustering.pdf file_support.jar geneweaver.jar gp-common.jar ineq_0.2-2.tar.gz ineq_0.2-2.tgz jaxb-rt-1.0-ea.jar my.local.install.r RunSomAlg.jar trove.jar version.txt
The crux of adding a module to the GenePattern server is to provide the command line that will be used to launch the module, including substitutions for settings that will be specified differently for each invocation. In the command line field, you will provide a combination of the fixed text and the dynamically-changed text which together constitute the command line for an invocation of the module.
Perhaps the trickiest thing about specifying a command line is making it truly platform-independent. Sure, it works fine for your computer, right now. But if you zip it and send it to an associate, are they running a Mac? Windows? Unix? You may not know, and you shouldn't need to care. By carefully describing the command line using substitution variables, you can pretty well ensure that your module will run anywhere.
Parameters: Parameters that require substitution should be enclosed in brackets (ie. <filename>). Every parameter listed in the parameters section must be mentioned in the command line unless its optional field is checked. A default value may be provided and will be used if the user fails to specify a value when invoking the module.
Click the View Argument List button to display a list of the parameters mentioned in the command line. You can change the order of the parameters by dragging them to a new position in the list or by editing the text of the command line.
Substitution properties: In addition to parameter names, you may also use environment variables, Java system properties, and any properties defined in the %GenePatternInstallDir%/resources/genepattern.properties file. In particular, there are predefined values for <java>, <perl>, and <R>, three languages that are used within various modules that may be downloaded from the module catalog at the public GenePattern website. Useful substitution properties include:
| <java> | path to Java, the same one running the GenePattern server |
| <perl> | path to Perl, installed with GenePattern server on Windows, otherwise the one already installed on your system |
| <R> | path to a program that runs R and takes as input a script of R commands. R is installed with GenePatternserver on Windows and MacOS |
| <java_flags> | memory size and other Java JVM settings from the GenePattern/resources/genepattern.properties file |
| <libdir> | directory where the module's support files are stored |
| <job_id> | job number |
| <name> | name of the module being run |
| <filename_basename> | for each input file parameter, the filename without directory |
| <filename_extension> | for each input file parameter, the extension without filename or directory |
| <filename_file> | for each input file parameter, the input filename without directory |
| <path.separator> | Java classpath delimiters (: or ;), useful for specifying a classpath for Java-based modules |
| <file.separator> | / or \ for directory delimiter |
| <line.separator> | newline, carriage return, or both for line endings |
| <user.dir> | current directory where the job is executing |
| <user.home> | user's home directory |
Rather than having to customize your module's command line for the exact location of the language runtime on each computer, you can use the substitution properties. For example,
<java> -cp <libdir>mymodule.jar com.foo.MyModule <arg1>
GenePattern will then take care of locating the Java runtime, asking it to begin execution at the MyModule class using code from the uploaded file mymodule.jar.
Standard input/output: If your module is designed to accept a standard input stream and/or write to a standard output stream, you can use redirection syntax when describing the command line. To redirect a file to the input stream, enter the text \< followed by the input file parameter. To redirect the standard output or standard error streams to a named file, enter the text \> or \\>& followed by the name of the output file. In the following example, the LogTransform module reads its input from the standard input stream and writes its output to the standard output stream:
<perl> <libdir>log_transform.pl \< <input.filename> \> <output.file>
ConsensusClustering example (actually all on one line):
<java> <java_flags> -DR_HOME=<R_HOME> -cp <libdir>geneweaver.jar edu.mit.wi.genome.geneweaver.clustering.ConsensusClustering <input.filename> <kmax> <niter> <normalize.type> -N <norm.iter> -S <resample> -t <algo> -L <merge.type> -i <descent.iter> -o <out.stub> -s -d <create.heat.map> -z <heat.map.size> -l1 -v
The input parameters section of the form appears perhaps to be the most daunting. And yet there is little that is required to make a working module declaration. Each parameter in the command line that comes from a user input must have an entry in this section. Otherwise the clients would know nothing about how to prompt the user for input nor could they explain to the user what type of input is expected.
To add one or more parameters, enter the number of parameters to add and click the Add Parameter button.
Each parameter has a name, which can be whatever you like, using letters, numbers, and period as a separator character between "words". It can be of mixed upper- and lower-case. The name is used inside <brackets> within the command line to indicate that the value of that variable should be substituted at that position within the command line. The name is also used as a label within the web client to prompt the user for the value for that field. And the name is used as a way of identifying which parameter is which for the scripting clients.
ConsensusClustering examples: kmax, input.filename
Some parameters are not required on the command line. These parameters, when left blank by the user when the module is invoked, result in nothing being added to the command line for that parameter.
The description field is optional, but is very useful. It allows the module author to provide a more detailed description than the name itself. What is the "kmax" parameter used for? Does it interact with any other parameters? Do you have any advice about what is a reasonable range of settings for it? The description is displayed by the GenePattern clients when they prompt for input for each field.
ConsensusClustering example: Type of clustering algorithm
Some parameters should have a default value which will be supplied on the module's command line if no setting is supplied by the user when invoking the module. This is not the same as the defaults defined in the program invoked by the module. Instead, this allows the module author to create a default, even when none exists in the program being invoked by the module.
The default value may use substitution variables, just like the rest of the command line. So a valid default for an output file might be <input.filename_basename>.foo, meaning that the output file will have the same stem as the input.filename parameter, but will have a .foo extension.
Default values for parameters that have a choice list must be either blank or one of the values from the choice list. Any other setting will result in an error message. If no default for a choice list is provided, the first entry on the list will be the default.
ConsensusClustering examples: NMF, 5, <input.filename_basename>
Some parameters need to have extra text prefixing them on the command line when they are specified. For example, you might need to write "-F filename" to pass in a filename. The prefix text "-F" or "-F " would be specified here. To insert a space between the flag and the parameter, add the space to the prefix text.
example (with space): -F inputfile
example (without space): -Finputfile
Declaration of the type of an input parameter allows the client to make a smarter presentation of the input to the user. (As of GenePattern 1.2, all parameters are being treated as either text or input file types). Parameter type choices are:
When you select a parameter type of input file, a drop-down list of file formats appears in the file format column. Select the valid file format(s) for this parameter. If your module requires an input file format not included in the list, scroll back to the Output Description field and click New to add that format to the list. For this type of parameter, when the user enters the name of the file, the GenePattern clients pass along the entire file rather than just the file name.
Some parameters are best represented as a drop-down list of choices. By constraining input to those from the list, the user is saved typing and cannot make a mistake by choosing an invalid setting (unless there is a dependency on some other parameter). To enter the choices, click the Edit Choices link and enter the choices in the Edit Choice List window.
For each choice enter the value required by the program (Value) and, optionally, a more human-readable value (Display Value). When you exit from the Edit Choice List window, the choices you entered are displayed as a semi-colon delimited set of choices. For example:
hierarchical=Hierarchical clustering;SOM=Self-organizing map;NMF=Non-negative Matrix Factorization;3.14159265=pi
Creating a GenePattern module is a two-step process:
When writing a program that will be run as a GenePattern analysis module, keep in mind the following:
perl myscript" rather than including "#!/usr/bin/perl" as the first line of the myscript.pl file.Visualization modules are similar to analysis modules. The only difference between analysis and visualization modules is that analysis modules run on the server machine and visualization modules run on the client machine. Each module is launched in a separate process. An applet is used to launch the visualization module.
If you are writing R code to be invoked as a GenePattern module, follow the guidelines in Writing Modules for GenePattern. This section provides additional information for modules written in R:
When you create a module in GenePattern, you specify the command line that invokes the program that performs the desired function. Generally, the command line includes arguments, such as the parameters for the algorithm and the data file to analyze. For example, the following command line invokes the myfunc() function in the R script named myscript.R, passing a single parameter, input.filename:
<R> <libdir>myscript.R myfunc <input.filename> |
Calling R script from a command line is possible, but generally not useful because you cannot pass arguments to the script. To pass arguments to your R code, create a function. For example:
myfunc <- function ( input.filename ) |
When designing the program and its command line, use parameter flags (for example, -f input_file) rather than relying on parameter positions. Parameter flags allow users to build command lines with variable numbers of arguments, which makes it easy to omit optional parameters. When writing R code, if you have optional input parameters on the command line, you must use named rather than positional parameters in the command line definition.
For example, to write an R function that takes a filename as input, the main function might be:
myfunction <- |
The command line for the module would be:
<R2.5> <libdir> myscript.R myfunction -i<input.file> |
In order to facilitate the use of R libraries in GenePattern modules, the GenePattern team maintains a list of R libraries and makes them available in the module repository. To make use of these libraries, you need only make an edit (to be described below) to the module manifest, which causes the necessary libraries to be downloaded and installed, if the library does not already exist on that server. Once installed, the R libraries will be available to any other module which requires the same library.
This section describes how to do this using the RLibraryDemo module as an example.
With this method of installing R libraries there is no need to include all of the libraries with the module package; nor do you write the extra code to install those libraries when the module is first run. This also means that, should any problems with the libraries arise during the installation of the module, an error will be thrown at that time, rather than seeming to install correctly and failing at run time. Lastly, but no less importantly, this allows you to install the libraries only one time per server and to share those libraries between modules, rather than installing for each individual module which needs them.
The easiest way is to use one the GenePattern team has already provided and/or use our demo module, RLibraryDemo, as a template for your own.
For a list of available R libraries from the GenePattern team, please contact us at gp-help AT broadinstitute DOT org. The current list is available at ftp://ftp.broadinstitute.org/pub/genepattern/plugins/rlib/ in folders organized by R version.
The RLibraryDemo module is available here. Once you have downloaded the module, open the zip file and take a look at the module's "manifest".
Notice the "requiredPatchLSIDs" and "requiredPatchURLs" parameters. For each library required by your module, you must specify both the LSID of the library, as well as the URL to the location where the library can be accessed.
Note that in order to use this example module, you will need to install R 2.10. For information on how to do so please see Using Different Versions of R in GenePattern.
If you are writing MATLAB code to be invoked as a GenePattern module, follow the guidelines in Writing Modules for GenePattern. In addition, for MATLAB code, you must address licensing and distribution issues, as described in this section:
You can invoke a MATLAB executable from a GenePattern module using one of two approaches: the direct approach or the compiled approach. Following are brief descriptions of each approach, including its advantages and disadvantages:
If you are simply using your M-code on your standalone GenePattern server, the direct approach is simpler; however, if you want to give copies of your M-code to other people or deploy your M-code on a shared GenePattern server, the compiled approach is preferred. The compiled approach may provide slightly better performance for fast running modules since the startup delay will be shorter, but the actual execution time will be approximately the same for either approach.
The instructions in this section are based on the following MATLAB versions:
When you create a module in GenePattern, you specify the command line that invokes the program that performs the desired function. Generally, the command line includes arguments, such as the parameters for the algorithm and the data file to analyze.
Calling script M-code from a command line is possible, but generally not useful because you cannot pass arguments to the script. To pass arguments to your M-code, create a no-return entry function to serve as the top level call into MATLAB. The following example defines a no-return entry function that accepts two parameters:
function analyzeThis ( filename, whatToWrite ) |
Writing Modules for GenePattern provides additional guidelines for writing code that will run as a GenePattern module.
If you do not plan to use the compiled M-code approach, skip this section and continue with Distributing Your MATLAB Code.
Compiling your MATLAB M-code into a standalone executable is described in the MATLAB Compiler Documentation. Please refer to this documentation to understand all of the options available to you. To summarize the simplest case, from within MATLAB, at the MATLAB prompt, execute the following command:
mcc -m analyzeThis |
where analyzeThis is the name of your entry function. This command generates the following files in your $MATLAB_ROOT/work directory:
|
analyzeThis (Linux, Mac OS X) or analyzeThis.exe (Windows) |
Executable file |
| analyzeThis.ctf | Component Framework file |
| analyzeThis.c (Linux, Windows) | C language Source Code |
| analyzeThis.h (Linux, Windows | C Language Header file |
| analyzeThis_main.c | C language Source Code |
| analyzeThis_mcc_component_data.c | C language Source Code |
Note: To use the MATLAB compiler on Mac OS X, you must have Xcode 2.2 installed; minimally, the Developer Tools, gcc 4.0, gcc 3.3, Mac OS X SDK, and BSD SDK. These instructions were tested using Xcode 2.2.1.
After writing your MATLAB code, create a GenePattern module that invokes the code that you have written. Creating Modules in GenePattern describes how to create a GenePattern module. This section provides supplemental information for MATLAB:
Creating Modules in GenePattern describes how to create a GenePattern module that invokes the code that you have written. This section provides additional information that applies when you are directly calling the MATLAB executable from the GenePattern module:
On Windows, your GenePattern module definition form can contain a simple command line that calls MATLAB with the -r flag to execute your function; for example:
|
|
This example invokes MATLAB without the splash screen (-nosplash) and directs it to execute the quoted command, where p1 and p2 are parameters that you specify in the GenePattern module definition form and that are passed to the MATLAB command line as Strings. MATLAB looks for the function analyzeThis on the MATLAB path; therefore, it is not necessary to upload the function as a support file, although it is recommended.
To ensure that the GenePattern server can call the MATLAB executable, you typically add the MATLAB directory to your PATH system environment variable. (Alternatively, you can enter the full path to the MATLAB executable on the command line, but this makes it more difficult to deploy the module on other GenePattern servers.)
To check that MATLAB is on your path:
matlab and press Enter.If the MATLAB application starts, MATLAB is on your path.
If MATLAB is not on your path, add it:
$MATLAB_ROOT/bin directory to the path.Open a new DOS window and check again that MATLAB is on your path.
On platforms other than Windows, the execution of the command line differs slightly due to variations in the Java Virtual Machines (VMs) that GenePattern is running. If you use the simple matlab command, as described for Windows, the Java VMs on these platforms attempt to parse and quote the command line resulting in MATLAB generating errors in its eval function.
On these platforms, you must use a wrapper Java class to launch MATLAB. This wrapper class also works on Windows and does not rely on the PATH variable, which makes it the preferred method for implementing the direct approach on any platform.
To use the wrapper Java class:
runmatlab.jar file as a support file. To request a copy of this file, send e-mail to gp-help (at) broadinstitute.org; alternatively, the java source code for the RunMatlab wrapper class is included here: RunMatlab.java.
<java> -cp <libdir>runmatlab.jar RunMatlab <libdir> analyzeThis <p1> <p2> |
Where analyzeThis is the name of your MATLAB entry function name and <p1> and <p2> are the arguments to the function. The RunMatlab class ensures that the arguments are correctly written out and calls MATLAB with the -nosplash and -nodisplay arguments.
Creating Modules in GenePattern describes how to create a GenePattern module that invokes the code that you have written. This section provides additional information that applies when you are compiling your M-code into a standalone executable and invoking that executable from the GenePattern module:
To run a standalone executable generated by the MATLAB Compiler, the GenePattern server must have the MATLAB Component Runtime (MCR) installed. This is a collection of shared libraries, which contains the runtime code for MATLAB, that is used by the standalone application. If the GenePattern server has MATLAB installed, you do not need to install the MCR; it is already installed.
Full details for installing the MCR can be found in the MATLAB Compiler documentation, in the section titled "Deploying Components to Other Machines". To summarize this documentation, on the GenePattern server machine, you need to run the MCRInstaller:
On Windows, to run the MCRInstaller:
<matlabroot>\toolbox\compiler\deploy\win32\MRCInstaller.exe to the server machine.MCRInstaller.exe.On Linux, to run the MCRInstaller:
buildmcr.<matlabroot>/toolbox/compiler/deploy/MCRInstaller.zip to the server machine.MCRInstaller.zip into a directory (<mcr_root>).|
|
On Mac OS X, to run the MCRInstaller:
buildmcr.<matlabroot>/toolbox/compiler/deploy/MCRInstaller.zip to the server machine.MCRInstaller.zip into a directory (<mcr_root>).|
|
When the MATLAB Compiler generates a standalone executable, it also generates a Component Framework (.ctf) file. The .ctf file must be on the path when you run the standalone executable. The easiest way to address this requirement is to create a launcher script (.bat or .sh file) that adds the .ctf file to the PATH or LIBPATH and then runs the standalone executable.
On Windows, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.bat, that contains the following lines:
|
|
On Linux, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.sh, that contains the following lines:
|
|
The chmod line sets the executable permission on the executable file; by default, the GenePattern server does not set this permission for uploaded files.
On Mac OS X, for example, to launch the MATLAB executable analyzeThis.exe, create a launcher script, mllaunch.sh, that contains the following lines:
|
|
The chmod line sets the executable permission on the executable file; by default, the GenePattern server does not set this permission for uploaded files.
On the GenePattern module definition form, write a command line calls the launcher script, passing the <libdir> parameter as the first argument (so that it can be added to the path).
On Windows, the following command line calls the launcher script, mllaunch.bat:
|
|
On Linux or Mac OS X, the following command line calls the launcher script, mllaunch.sh:
|
|
In both command lines, the first <libdir> sets the path to the mllaunch script. The second <libdir> is passed as the first argument to the script so that the script can add this directory to the appropriate environment variables. The <param1> and <param2> variables are parameters to the MATLAB application, which you define in the module definition form and specify in the command line as usual.
For the compiled approach, you must specify at least two support files for the MATLAB application: the executable file and .ctf file. If your application requires additional files for its execution, also add those files as support files.
Should you choose to distribute your MATLAB based module to others, you must ensure you are in compliance with the MATLAB licensing agreement:
http://www.mathworks.com/company/aboutus/policies_statements/agreement.pdf
Following are a few key points for GenePattern developers:
Please refer to the MATLAB licensing agreement for exact details. You are responsible for reviewing and complying with the MATLAB software license. The above summary does not exempt you from this responsibility.
This section provides a step-by-step example of deploying a simple M-file application as a GenePattern module on a GenePattern server. Where the instructions are platform specific, the example shows instructions for Windows, Linux, and Mac OS X.
The first step is writing the MATLAB M-file that you want to share. For this example, write a simple application that takes a filename and a String and writes the String out to a file with the given name. This application consists of the following lines:
% write the variable whatToWrite to a file called filename in the current directory |
To call the M-file from the command line and pass it parameters, you must turn this script into a no-return function. To do this, add a function definition line at the start of the M-file and save the file using the name of the function (for example, writeToFile.m).
|
|
Within the MATLAB environment, call the MATLAB Compiler to convert this function into an application:
|
|
Within the current working directory, this creates a number of files, including the following:
Note: To use the MATLAB compiler on Mac OS X, you must have Xcode 2.2 installed; minimally, the Developer Tools, gcc 4.0, gcc 3.3, Mac OS X SDK, and BSD SDK. These instructions were tested using Xcode 2.2.1.
Install the MATLAB Component Runtime (MCR) on the GenePattern server, if you have not done so already. If the GenePattern server has MATLAB installed, it also has the MCR installed.
To install the MCR:
<matlabroot>\toolbox\compiler\deploy\win32\MRCInstaller.exe to the GenePattern server machine.|
|
To install the MCR:
>> buildmcr mcrdir |
mcrdir, beneath the current working directory and creates a file within that directory called MCRInstaller.zip.matlab, under the GenePattern server directory and install the library files in MCRInstaller.zip into that directory:
cd GenePatternServer |
Create the launcher script that sets the environment variables and then calls the MATLAB application.
Create the launcher script as a batch file that sets the PATH variable for the environment and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.bat file:
set LIBDIR=%1 |
Create the launcher script as an .sh file that sets the PATH and LD_LIBRARY_PATH variables for the environment, ensures that the application is executable, and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.sh file:
#!/bin/csh |
Note that the MCR_ROOT variable is set to the v70 directory, which you created by unzipping MCRInstaller.zip.
Create the launcher script as an .sh file that sets the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH variables for the environment, ensures that the application is executable, and then calls the MATLAB application. To do so, in a text editor, create the following mllaunch.sh file:
#!/bin/sh |
Use GenePattern to create a module that executes the launcher script.
sh <libdir>mllaunch.bat <libdir> <fname> <txt>
sh <libdir>mllaunch.sh <libdir> <fname> <txt>
Save the module and execute it. The module should create two files:
If the following error appears in the stdout file, you have not correctly set the path to the libraries that you installed from MCRInstaller.zip:
error while loading shared libraries: libmwmclmcrrt.so.7.0: cannot open shared object file: No such file or directory
Double check the path. If it is correct, you may be using a different Unix shell than the one used in this example. Check that the mllaunch.sh file uses the correct command (export in this example) to set PATH and LD_LIBRARY_PATH.
Using Java as a GenePattern client allows you to run GenePattern modules and visualizers from within a Java application. This section describes how you can use the GenePattern Java library to run GenePattern analyses as easily as calling a routine. It contains the following topics:
If you are not familiar with Java, see the http://java.sun.com website, which provides downloadable programs, samples, tutorials, and book suggestions.
The GenePattern Java library allows you to invoke a GenePattern module as if it were a local Java method running on your client and to get back from the module a list of result files. A zip file containing the Java library (and Javadoc that describes the API for accessing the server and running modules) is available on your GenePattern server.
To download the GenePattern Java library to your computer:
This section explores a simple Java application that preprocesses a dataset and displays it using the HeatMapViewer. The included code can be copied and pasted into your Java program so that you can try it out, modify it, and create your own solutions. The full source code of the sample application is available here.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL and GenePattern user name (typically, your e-mail address) with values appropriate for your GenePattern server.
import org.genepattern.matrix.Dataset;import org.genepattern.client.GPClient;import org.genepattern.webservice.JobResult; import org.genepattern.webservice.Parameter;import org.genepattern.io.IOUtil; import java.io.File;
public class MyProgram {
public static void main(String[] args)
throws Exception {
GPClient gpClient=new GPClient("http://localhost:8080",
"your email address"); |
After initializing the required settings, the application runs the PreprocessDataset module to preprocess a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. When you invoke the runAnalysis method, the GenePattern library invokes the appropriate module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, invoke the runAnalysisNoWait method or use the runAnalysis method in a separate thread.)
String inputDataset= "ftp://ftp.broadinstitute.org/pub/genepattern/all_aml/all_aml_train.res";
JobResult preprocess=gpClient.runAnalysis("PreprocessDataset",
new Parameter[] {
new Parameter("input.filename", inputDataset)
}); |
When the module completes, you can query the JobResult object for an array of filenames that are the output from the module. You can download the result files or leave them on the server and refer to them by URL. Referring to result files by URL is especially useful for intermediate results. In this example, the JobResult object named preprocess contains a list of filenames (of length 1, in this case), which the application displays in a heat map:
// view results in a HeatMapViewer visualizer
gpClient.runVisualizer("HeatMapViewer",
new Parameter[] {
new Parameter("dataset", preprocess.getURL().toString())
}); |
The last statements in the application download the preprocessed data and load it into a matrix for further analysis:
String downloadDirName=String.valueOf(preprocess.getJobNumber()); // download result files File[] outputFiles = preprocess.downloadFiles(downloadDirName); // load data into matrix for further manipulation Dataset dataset= IOUtil.readDataset(outputFiles[].getPath()); } } |
You can combine GenePattern analyses with any capabilities that the Java environment has to offer. Use Java's 2-D and 3-D graphics libraries to create graphic output, or summarize and report on the data using your own code. The basic idea to remember is that GenePattern modules create result files and those files are available to the Java application for processing.
For more information:
Life Science Identifiers (LSIDs) can be used instead of module names to identify modules for GenePattern to run. An LSID may be submitted in place of the module name in the methods runAnalysis and runVisualizer. When an LSID is provided that does not include a version, the latest available version of the module identified by the LSID will be used. If a module name is supplied, the latest version of the module with the nearest authority is selected. The nearest authority is the first match in the sequence: local authority, Broad authority, other authority.
If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
Using MATLAB as a GenePattern client allows you to run GenePattern modules and to manipulate and visualize the results in a powerful, commercial technical computing application that works on most major platforms. Using GenePattern allows you to invoke methods written in many other languages without having to worry about how to launch them. This section describes how you can use the GenePattern MATLAB library to run GenePattern analyses:
Resources and documentation for MATLAB are available at http://www.mathworks.com/.
The GenePattern MATLAB library allows you to invoke a GenePattern module as if it were a local MATLAB function running on your client and to get back from the module a list of result files. A zip file containing the MATLAB library is available on your GenePattern server.
To download the GenePattern MATLAB library to your computer:
MATLAB7/toolboxes directory. If you do not have permission to put files in that directory, unzip into any other directory.
>>pathtool |
GenePatternServer and GenePatternFileSupport directories, with subfolders, to the MATLAB search path.Note: MATLAB 7.0.4 (R14SP2) and later use Java Virtual Machine (JVM) 1.5. If you are using an earlier version of MATLAB, you must change the JVM that MATLAB is using to JVM 1.5. For instructions, see http://www.mathworks.com/support/solutions/data/1-1812J.html?solution=1-1812J.
This section explores a simple MATLAB program that runs a module, displays the resulting output, and loads it into a MATLAB matrix for further analysis. The included code can be copied and pasted into your MATLAB client so that you can try it out, modify it, and create your own solutions.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL, GenePattern user name (typically, your e-mail address) and password (if required) with values appropriate for your GenePattern server.
% Create a GenePattern server proxy instance |
After initializing the required settings, the application runs the TransposeDataset module to transpose a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. As shown below, you can call the GenePattern methods directly or by calling the runAnalysis method. When you call a GenePattern method, such as TransposeDataset, the GenePattern library invokes the module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, invoke the method in a separate thread.)
% input dataset for transpose operation |
When the module completes, it returns a MATLAB structure that contains a list of filenames that are the output from the module. In this example, transposeResult is a structure with a list of filenames (of length 1, in this case). The application displays the results in a file viewer window and also loads them into a matrix so that further manipulation can be performed:
% display the transposed results |
You can combine GenePattern analyses with all of the rich functionality of MATLAB. For example, you can use MATLAB's plotting methods to create graphic output, save modified matrices to files using save, or summarize and report on the data using your own code. The basic idea to remember is that GenePattern modules create result files and those files are available to the MATLAB client for processing.
For a list of the GenePattern modules available on your server, run the listMethods function on your GenePatternServer object. To view the names of the input parameters for a module, use the describeMethod function on your GenePatternServer object, passing it the module name.
% display the available GenePattern modules |
Alternatively, to get the parameters with their default values filled in, use the getMethodParameters function of the GenePatternServer object. This returns a MATLAB structure with named elements for each parameter, filled in with the default value if one exists. After filling in the missing parameters and overriding defaults if desired, this structure can then be passed on to the runAnalysis method.
% display the available GenePattern modules |
The GenePattern MATLAB library also has convenience methods to read and write GenePattern files (such as res, gct, and odf files). Even if you choose not to look in the library, you can extend the techniques shown above to implement your own analyses.
For more information:
You can use Life Science Identifiers (LSIDs) to identify a module when executing GenePattern code in MATLAB. An LSID may be submitted in place of the module name to getMethodParameters or runAnalysis. When providing an LSID to a method in addition to a module name, the LSID alone is used to determine what module to run. When an LSID is provided that does not include a version, the latest available version of the module identified by the LSID will be used. If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.
% Example using LSIDs from MATLAB |
Using R as a GenePattern client allows you to run GenePattern modules and to manipulate and visualize the results in a powerful, free statistical desktop package that works on most major platforms. Using GenePattern allows you to invoke methods written in many other languages without having to worry about how to launch them or whether you are passing incorrect parameters. This section describes how you can use the GenePattern R library to run GenePattern analyses:
If you are not familiar with R, see the following resources on the www.r-project.org website:
The GenePattern R package allows you to invoke a GenePattern module as if it were a local R method running on your client and to get back from the module a list of result files. The package requires R version 2.4.1 or greater and the rJava package. The package can be downloaded from your GenePattern server in Windows(.zip), source (.tar.gz), and Mac OS X (.tgz) formats.
To download the GenePattern R package to your computer:
|
|
Note: If you are using a version of R which you cannot modify (because it is a publicly-shared version and you do not have appropriate privilege), you can have it load the GenePattern library by setting the environment variable R_LIBS=<GenePattern install directory>/R/library in your autoexec.bat, .cshrc, .bashrc or other shell startup file. R will then load from its usual location, but will also search for and find the GenePattern library from your installation.
This section explores a simple R program that runs a module, displays the resulting output, and loads it into an R matrix for further analysis. The included code can be copied and pasted into your R environment so that you can try it out, modify it, and create your own solutions.
The first statements in the application initialize various settings, which you must do once in every application that accesses GenePattern. You will need to customize the italicized GenePattern server URL, GenePattern user name (typically, your e-mail address), and password with values appropriate for your GenePattern server. The gp.login method returns a GPClient object that contains the information required for running modules on a GenePattern server.
# Load GenePattern package |
After initializing the required settings, the application runs the PreprocessDataset module to preprocess a dataset. This example references the dataset using a publicly-accessible URL, but a filename would be equally valid. When you call an R method, such as run.analysis, the GenePattern package invokes the appropriate module on the server, passing all of the input parameters and input files. Control returns to your application when the module completes. (To run a module asynchronously, use the method runAnalysisNoWait.)
|
|
When the module completes, it returns a JobResult object with which you can execute various methods. For example, you can call a method using a JobResult object to get an R list of the filenames that are the output of the module. Afterwards, you can download the files or leave them on the server and refer to them by URL. In this example, we view the results in a heat map:
|
|
In this example, the application downloads the result file and displays the results in a file viewer window, then also loads the data into a matrix so that further manipulation can be performed in R:
|
|
You can combine GenePattern analyses with all of the rich statistical functionality of R. For example, you can use R's plot and legend methods to create graphic output, output JPEGs of your visualized data using savePlot, save modified matrices to files using save, or summarize and report on the data using your own code. Just remember: GenePattern modules create JobResult objects and those objects are available to the R client for processing.
The GenePattern R package also has methods to read and write GenePattern files (such as res, gct, and cls files), to enable running of multiple modules in parallel, to run modules with input from files that were output from previous modules without moving them from the server, and other utilities. Even if you choose not to look in the library, you can extend the techniques shown above to implement your own analyses.
For more information:
You can use Life Science Identifiers (LSIDs) instead of module names to identify modules for GenePattern to run. For R, this is primarily useful when you want to specify a particular version of a module for GenePattern to run. The easiest way to specify a particular version of a module is to specify the LSID as an argument to an R method such as run.analysis in place of the GenePattern module name. For example, the following statement invokes version 1 rather than the latest version of the PreprocesDataset module:
| preprocess.jobresult <- run.analysis(gp.server, "urn:lsid:broadinstitute.org:cancer.software.genepattern.module.analysis:00020:1", input.filename=input.ds) |
If you are unfamiliar with LSIDs and GenePattern versioning, see Concepts.