Using a Queuing System  Print-icon

Queuing systems such as the Load Sharing Facility (LSF) and the Sun Grid Engine (SGE) allow computational resources to be used effectively. If you have installed a queuing system, you can configure the GenePattern server to use it. On a heavily used server, using a queuing system to execute analysis jobs generally improves performance overall, especially for compute-intensive and long-running jobs; however, short jobs might take slightly longer because they must be dispatched to the queuing system.

The GenePattern server includes support for Sun Grid Engine (SGE) and LSF. To configure your server, you need to edit the configuration file and restart the server. Detailed documentation is in the 'config_example.yaml' file which is in the resources directory of your local GenePattern installation.

There are three additional ways to configure GenePattern's interaction with your queuing system; either programmatically or with a command line prefix.

JobRunner API

To integrate your queuing system with GenePattern:

  1. Implement the JobRunner API
  2. Deploy the jar file to the GenePattern server.
  3. Configure your server.
  4. Reload the new configuration.

1. Implement the JobRunner API

A source code snippet from the API is included here. Contact us for the full source code.

interface JobRunner {

    /**
     * The GenePattern Server calls this when it is ready to submit the job to the queue.
     * Submit the job to the queue and return immediately.
     * The drm jobId returned by this method is used as the key into a 
     * lookup table mapping the gp jobId to the drm jobId.
     * 
     * @return the drm jobId resulting from adding the job to the queue.
     */
    String startJob(DrmJobSubmission drmJobSubmission) throws CommandExecutorException;


    /**
     * Get the status of the job.
     * @param drmJobId
     * @return
     */
    DrmJobStatus getStatus(DrmJobRecord drmJobRecord);


    /**
     * This method is called when the GP server wants to cancel a job before it 
     * has completed on the queuing system. 
     * For example when a user terminates a job from the web ui.
     * 
     * @param drmJobRecord, contains a record of the job
     * @return true if the job was successfully cancelled, false otherwise.
     * @throws Exception
     */
    boolean cancelJob(DrmJobRecord drmJobRecord) throws Exception;
}
 
The required Java libraries come with your local install of GenePattern and can be found in the <GenePatternServer>/Tomcat/webapps/gp/WEB-INF/lib directory.
Follow steps 2-4 in the CommandExecutor Interface section below to configure your server.

CommandExecutor Interface

To use a queuing system with GenePattern:

  1. Implement the CommandExecutor Java API.
  2. Deploy the resulting jar file to the GenePattern server.
  3. Configure your server.
  4. Reload the new configuration.

Each step is described in detail below.

1. Implement the Command Executor Interface

The full source for the Command Executor API is included here:

/**
 * Interface for managing job execution via runtime exec or an external queuing system. This interface is responsible for both initialization and shutdown of external services,
 * as well as the management of job submission, getting job status, and killing, pausing, and resuming jobs.
 *
 * @author pcarr
 */
public interface CommandExecutor {
   //configuration support
   /**
    * [optionally] set a path to a configuration file.
    */
   void setConfigurationFilename(String filename);
 
   /**
    * [optionally] provide properties.
    * @param properties
    */
   void setConfigurationProperties(CommandProperties properties);
 
   /**
    * Start the service, typically called at application startup.
    */
   public void start();
 
   /**
    * Stop the service, typically called just before application shutdown.
    */
   public void stop();
 
   /**
    * Request the service to run a GenePattern job. It is up to the service to monitor for job completion and callback to GenePattern when the job is completed.
    *
    * @see GenePatternAnalysisTask#handleJobCompletion(int, String, String, int)
    *
    * @param commandLine
    * @param environmentVariables
    * @param runDir
    * @param stdoutFile
    * @param stderrFile
    * @param jobInfo
    * @param stdin
    *
    * @throws CommandExecutorException when errors occur attempting to submit the job
    */
   void runCommand(
           String commandLine[],
           Map<String, String> environmentVariables,
           File runDir,
           File stdoutFile,
           File stderrFile,
           JobInfo jobInfo,
           File stdinFile)
   throws CommandExecutorException;
 
   /**
    * Request the service to terminate a GenePattern job which is running via this service.
    * @param jobInfo
    * @throws Exception indicating that the job was not properly terminated.
    */
   void terminateJob(JobInfo jobInfo) throws Exception;
 
   /**
    * This method is called on server startup for each RUNNING job for this queue.
    *
    * For RuntimeExec, tell the GP server to delete the job results directory and requeue the job.
    * For other executors, (such as LSF), you may want to ignore this message.
    * For PipelineExec, you may need to determine the last successfully completed step before resuming the pipeline.
    *
    * @return an optional int flag to update the JOB_STATUS_ID in the GP database, ignore if it is less than zero
    */
   int handleRunningJob(JobInfo jobInfo) throws Exception;
}

The required Java libraries come with your local install of GenePattern and can be found in the <GenePatternServer>/Tomcat/webapps/gp/WEB-INF/lib directory.

The interface accepts requests to start and terminate jobs from the server. You will need to invoke a callback to the GP server when your job has completed.

Example snippet:

try {
     GenePatternAnalysisTask.handleJobCompletion(jobInfo.getJobNumber(), exitCode, null, runDir, stdoutFile, stderrFile);
 }
 catch (Exception e) {
     log.error("Error handling job completion for job "+jobInfo.getJobNumber(), e);
 }

Once you have implemented this interface, create a jar file to deploy to the GP server.

2. Deploy the jar File to the GenePattern Server

The jar file and all of the dependent libraries must be installed to <GenePatternServer>/Tomcat/webapps/gp/WEB-INF/lib.

3. Configure Your Server

To configure your server to interact with your queuing system, you must edit the config.yaml file. In a fresh install of GenePattern, there will be two .yaml files found in <GenePatternServer>/resources: config_default.yaml and config_example.yaml. It is highly recommended that you make a copy of config_default.yaml and name it something like config.yaml. This will give you a working copy of your configuration file, preserving the default and example versions for your future reference. Additionally this will prevent your working copy from getting overwritten during server upgrade.

Edit the config.file property in the <GenePatternServer>/resources/genepattern.properties file to point to your new configuration file. By default, the property looks like this:

config.file=config_default.yaml

For this example, you would edit the property as follows:

config.file=config.yaml

Now, edit your working copy of the configuration file, config.yaml. (The following code snippets come from the config_example.yaml file.)

a) Define an executor in the "executors" section. To do so add an item to the list of 'executors' in the yaml document.

# a list of command executors
# The executor id, 'org.genepattern.server.executor.PipelineExecutor', is reserved for the default executor which runs all GP pipelines.
# Don't use this as an executor id in this file.
# a map of <id>:<obj>, where
#    obj := <classname> | <map>
#    classname := fully qualified classname of a class which implements the org.genepattern.server.executor.CommandExecutor interface
#    map := classname=<classname> [configuration.file: <path_to_config_file> | configuration.properties: <map>] [default.properties: <map>]
executors:
    # default executor for all jobs, it is included in GenePattern
    RuntimeExec:
        classname: org.genepattern.server.executor.RuntimeCommandExecutor
        configuration.properties:
            # the total number of jobs to run concurrently
            num.threads: 20
            # the total number of jobs to keep on the queue, not yet implemented
            #max.pending.jobs: 20000
 
    # nested declaration with configuration file, <id>: { classname: <classname>, configuration: <config_file> }
    Test:
        classname: org.genepattern.server.executor.TestCommandExecutor
        configuration.properties:
            num.threads: 20

b) Configure your server to use your executor.

# apply these properties to all jobs
default.properties:
    executor: Test
    java_flags: -Xmx512m

c) Optionally, you can use the configuration file to override the default executor on a per module, group or user basis. The following example comes from per module section, more examples can be found in config_example.yaml.

# override default.properties and executor->default.properties based on taskname or lsid
# Note: executor->configuration.properties are intended to be applied at startup and are not overwritten here
module.properties:
    CBS:
        executor: LSF
        lsf.max.memory: 16
        java_flags: -Xmx16g

About the .yaml configuration file: As of GenePattern 3.4.0, you use the .yaml configuration file only to configure GenePattern for use with a queuing system. As you work with the .yaml file, you may notice that it contains several properties that are also defined in the genepattern.properties file. To avoid confusion, leave them set to agree with the genepattern.properties file. GenePattern 3.4.0 reads these properties from the genepattern.properties file, not from the .yaml file. (In a future release, the genepattern.properties file may define the default server settings and the .yaml configuration file may define custom server settings.)

4. Reload the Configuration

At this point, you have deployed your command executor, modified the .yaml configuration file to control its use, and modified the <GenePatternServer>/resources/genepattern.properties file to point to the modified .yaml configuration file. Now, stop and restart the GenePattern server to reload the server configuration and begin to use the new command executor.

As you use GenePattern with the queuing system, you may find it useful to modify the configuration. The Administration>Server Settings>Job Configuration page provides several useful tools for controlling the internal GenePattern job queue and reloading the .yaml configuration file. Use this page to confirm which command  executors are currently installed and the exact .yaml configuration file currently in use. If you make minor adjustments to the configuration file, such as overriding the command executor used for a module, group or user, you can use the Job Configuration page to reload the configuration file without restarting the GenePattern server. On the other hand, for major changes, such as adding a new command executor, we recommend restarting the server rather than simply reloading the configuration.

Command Line Prefix

Pros and Cons

Before the 3.2.3 release of GenePattern (June 2010), the only way to connect to an external queuing system was to use the command line prefix. Although this option requires no Java programming and allows for configuration via a web page, it has significant drawbacks:

The drawbacks are a result of how the command line prefix works. Each new job requires a dedicated server process which waits for the job to complete. When a user terminates a job, the server process is terminated but the external process launched on the queuing system is not terminated. Similarly, when the GenePattern server shuts down, all server processes halt but the processes running on the external queuing system become orphaned. When the GenePattern server restarts, the jobs are not restarted; the user must restart any unfinished job from the beginning.

If you are using the CommandExecutor Interface, we recommend that you not use the command line prefix. The command line prefix is appended to the module command line before the job is executed by the CommandExecutor. To be more precise:

  1. The server gets the initial command line from the manifest.
  2. The command line prefix is appended to the initial command line.
  3. The server resolves all system and user variables and breaks the command line string into tokens.
  4. The server calls the CommandExecutor.runCommand.

Using the Command Line Prefix

Although this is not the preferred method, you can still use the Command Line Prefix to connect to an external queuing system.

To use the Command Line Prefix to configure the GenePattern server to execute jobs using LSF or SGE:

  1. Add the GenePatternURL property to the GenePattern configuration file, GenePatternServer/resources/genepattern.properties, specifying the URL of your server. For example:

    GenePatternURL=http://myserver.company.com:8080/gp/

    When you run a pipeline, the GenePattern server uses this URL to construct the links to the output files.

    By default, the GenePatternURL property is not set. When you run a pipeline, the GenePattern server derives the URL at run time based on the current IP address of the host server. This is ideal for a user running on a laptop, where the IP address may change at startup. However, if you are using a queuing system, the derived URL is incorrect: it is based on the IP address of the queuing system server rather than the GenePattern server.

  2. For Sun Grid Engine modify the R2.5 property in the GenePattern configuration file, GenePatternServer/resources/genepattern.properties, to quote the <r_flags> options. For example:

    R2.5=<java> -DR_suppress\=<R.suppress.messages.file> -DR_HOME\=<R2.5_HOME>
    -Dr_flags\=\"<r_flags>\" -cp <run_r_path> RunR

    Modify other similar properties (if any) that were added to support additional versions of R.

  3. Click Administration>Server Settings and use the Command Line Prefix page to have the GenePattern server add the required options to the command line each time it executes a module.

For example, if you are using LSF, modify the Command Line Prefix options as follows:

  1. Click Administration>Server Settings and select Command Line Prefix. GenePattern displays the Command Line Prefix page.
  2. Enter the following text in the Default Command Prefix field:
    bsub -K -o lsf_log.txt
    • The –K flag instructs the bsub command to wait for the job to complete before returning.
    • The –o flag specifies the file to which the job writes standard output and standard error messages.
  3. Optionally, set the environment variables BSUB_QUIET and BSUB_QUIET2 to prevent bsub from printing common job messages to standard out:
    • Setting BSUB_QUIET prevents bsub from printing the messages <<Job is submitted to default queue <normal>>> and <<Waiting for dispatch>>.
    • Setting BSUB_QUIET2 prevents bsub from printing the message <<Job is finished>>.

Another alternative is to create a script that sets the environment variables and then executes the job using LSF or SGE. The command prefix would then execute the script. For example:

  1. Create the shell script to set the variables and execute the job using LSF. The script executes in the jobResults directory for the job; for example, for job  3248, the script executes in the GenePattern /Tomcat/webapps/gp/jobResults/3248/ directory. The following script sets the environment variables, submits the job to the LSF queue, waits for the job to complete, saves stdout to a new  file, stdout.txt, and saves stderr to a new file, stderr.txt. By convention, GenePattern considers a job to fail if there is any output to stderr.

    #!/bin/bash
    #
    # Submit the job to LSF
    # Save lsf out and err files in the jobResults directory.
    # If there is stdout from the job, pipe to stdout of this script.
    # If there is stderr from the job, pipe to stderr of this script.
    lsf_err=.lsf.err;
    cmd_out=cmd.out;
    BSUB_QUIET=
    BSUB_QUIET2=
    export BSUB_QUIET
    export BSUB_QUIET2
    # submit the job and wait (-K) for the job to complete
    bsub -q genepattern -K -o .lsf_%J.out -e $lsf_err $"$@" \>$cmd_out
    # sleep to allow for NFS delay
    sleep 2;
    # If there is stdout from the job, pipe to stdout of this script, then delete the output file
    if [ -e $cmd_out ]
    then
       cat $cmd_out >&1;
       rm $cmd_out;
    fi
    # If there is stderr from the job, pipe to stderr of this script then delete stderr file
    if [ -e $lsf_err ]
    then
       cat $lsf_err >&2;
       rm $lsf_err;
    fi
  2. Click Administration>Server Settings and select Command Line Prefix. GenePattern displays the Command Line Prefix page.
  3. Enter the following text in the Default Command Prefix field:
    /fully/qualified/path/to/lsf_default.sh
  4. The script shown here saves the lsf log file into the job results directory. In GenePattern, the log files are displayed with the other job result files. If you do not want the log files displayed in GenePattern, edit the /resources/genepattern.properties file and set the following property:
    jobs.FilenameFilter=.lsf*

<< Increasing Memory Allocation Up Securing the Server >>

Updated on June 26, 2014 14:21