Tagged with #faqs
7 documentation articles | 0 announcements | 0 forum discussions



Created 2016-03-25 18:34:09 | Updated 2016-05-25 15:57:45 | Tags: faq faqs firecloud cromwell docker wdl

Comments (0)

Tool Developers

What resources are available for tool developers?

Please post your tools questions to the FireCloud Forum.

What is the procedure for porting externally developed tools into FireCloud?

  1. Put a tool in a Docker image and push it to Docker Hub.
  2. Create Workflow Description Language (WDL) workflow that references the tool’s Docker image in Docker Hub.
  3. Use Cromwell to test the WDL and dockerized tool locally.
  4. Use the FireCloud Command Line Interface (CLI) to push the tool to FireCloud and run a dockerized tool on FireCloud.

For more information, go to the Tool Developers topic in the User Guide.

Will there be an SDK (or other appropriate mechanism) for others to contribute to FireCloud? FireCloud may support an open source SDK package in the future to simplify tool development. Currently, the individual components needed to develop tools are all open source and can be downloaded free of charge (e.g., Docker, FireCloud-CLI, Cromwell).

The Tool Developers topic in the User Guide includes the steps to download these components.


Created 2016-03-10 15:10:10 | Updated 2016-03-11 14:21:10 | Tags: faq faqs firecloud

Comments (0)

General

What is “Firecloud”? FireCloud is one of three National Cancer Institute (NCI) Cancer Genomics Cloud Pilots for analyzing genomic data on the cloud along with co-located TCGA data.

It is maintained and developed by the Broad Institute in collaboration with the University of California Berkeley and University of California Santa Cruz.

FireCloud aims to be both a data repository and computational As a data repository, FireCloud will hold all TCGA data, both controlled and open access, including raw data files and derived data. As a computational platform, FireCloud will contain and support the Broad Institute’s best practice tools and pipelines, as well as tools uploaded by users.

FireCloud’s design is modeled in part after Firehose, the Broad Institute’s platform that supports large-scale analysis and production. Like Firehose, FireCloud supports computational sandboxes containing data models that help analysts organize information, along with analysis methods for operating on those models. Key differences include (1) FireCloud’s capacity for scalable, ‘elastic’ cloud computing on the Google Cloud Platform and (2) the packaging of all tools into docker images.

How would we describe FireCloud in one sentence to a new user? FireCloud is a cancer genome analysis platform built on a cloud computing environment, fully loaded with tools, workflows and co-located TCGA data.

Who can access Firecloud today? Currently, FireCloud is open to the Broad Institute. Soon, FireCloud will be open to the public.

All FireCloud users will be able to access free tutorial workspaces to view and run tutorial analyses using pre-loaded data and tools.

In order to create your own workspace, you must be granted access to at least one FireCloud Google Project. FireCloud Google Projects must be created by FireCloud administrators for FireCloud users. Google Projects created outside of FireCloud can not be used within the system.

FireCloud administrators can create a new FireCloud Google Project upon request. FireCloud administrators can also grant you access to an existing FireCloud Google Project that was previously created for another user. Please see FireCloud Google Projects and Google Billing Accounts for more information.

What is the target user base for FireCloud?

  • Analysts
  • Tool Developers
  • Production Managers

Will the Broad Institute phase out Firehose (the internal precursor to FireCloud) over time? Yes. We have not set a specific date to close Firehose, although our goal is to transition the Firehose user base to FireCloud by Spring of 2016.

What features are available in FireCloud? For information on FireCloud’s features, you can browse through FireCloud Basics in the User Guide.

What is FireCloud’s workspace data model? To help you organize your data and results, FireCloud’s workspace data model provides predefined entity types, entity relationships, and entity attributes.

FireCloud Entity Types:

  • participant
  • participant set
  • sample
  • sample set
  • pair
  • pair set

FireCloud’s workspace data model also provides structure for analysis runs by requiring an entity type (e.g., pair set) on which to run an analysis.

Results from data runs can be populated directly to the workspace data model for use in downstream analyses.

For more information, refer to the Data Model section of FireCloud Basics in the User Guide.

Can Firecloud be used for non-cancer projects? Yes, FireCloud’s current data model entities, such as participant, sample, and pair, help to categorize cancer data, but are generally adaptable for non-cancer projects.

FireCloud is also built to be extensible to multiple data models, and may support data model customization in the near future.

FireCloud attributes attached to entities in the data model are extensible. Workflows reference these attributes as input parameters. Attribute values may include demographic information, clinical lab values, sample and tissue classifications, as well as references to data files residing in cloud storage.

For more information, refer to the Data Model section of FireCloud Basics in the User Guide.

Can I request platform enhancements? How are enhancements prioritized? You can request enhancements directly through the FireCloud Forum.

We will periodically review the FireCloud Forum to gain insight about desired enhancements, and prioritize based on the frequency of requests for a particular feature.

We will also prioritize enhancements that improve system stability, ease-of-use, speed, scalability, cost-efficiency, and security.

Does FireCloud cost money? No, if you use Tutorial and shared workspaces. Tutorial workspaces will be provided at no costs and be available after you register for FireCloud. With the tutorial workspaces, you can view and run tutorial analyses using pre-loaded data and tools.

If another FireCloud user shares a workspace with you, granting you WRITER access, you can run analyses within that workspace but will not be charged for it. The compute and storage costs will be charged to the FireCloud Google Project under which the workspace was created.

Yes, if you create or clone new workspaces. In order to create your own workspace, you must be granted access to at least one FireCloud Google Project. FireCloud Google Projects must be created by FireCloud administrators for FireCloud users. Google Projects created outside of FireCloud cannot be used within the system.

FireCloud administrators can create new FireCloud Google Projects upon request or grant you access to an existing FireCloud Google Project that they previously set up for another user. Please see FireCloud Google Projects and Google Billing Accounts for more information.


Created 2016-03-10 15:07:57 | Updated | Tags: faq faqs firecloud

Comments (0)

Access and Registration

How do I register for FireCloud? You must first have a Google-managed identity such as a Gmail account or institutional Google Apps account.

To register, go to Firecloud.org, then click on Use FireCloud. You will be asked to provide basic information during registration. Refer to the Registration topic in the User Guide for more information.

Why hasn’t my FireCloud account been activated? Although anyone is welcome to register, FireCloud has only been approved for use within the Broad Institute. If you are a user from outside the Broad community, we will activate you account as soon as FireCloud is approved for the general public. You will receive an email when your account has been activated.

Why does FireCloud require me to have a Gmail or Institutional Google Apps account? Currently, FireCloud is built on top of the Google Cloud Platform and utilizes some Google features, including its authentication services.

Do I need to create a Google Billing Account or Google Project to use FireCloud? No. Tutorial workspaces will be created for you when your account is activated. These do not require a FireCloud Google Project or Google Billing Account. You can compute on data within these workspaces free of charge, and view and run tutorial data using pre-loaded tools.

In addition, if another FireCloud user shares a workspace with you, granting you WRITER access, you can compute within that workspace.

If you want to create and clone workspaces, you will need access to a FireCloud Google Project. Google Billing Accounts are required if you want FireCloud administrators to create your own FireCloud Google Project that transmits charges to a Google Billing Account.

Steps to set up a Google Billing Account and information about FireCloud Google Projects can be found here.

Why is FireCloud requesting offline access? You may see a screen requesting offline access during the registration process. FireCloud would like to acquire new Google access tokens on your behalf. This is useful for large jobs that may take longer to process than the lifetime of a Google access token. For more information, please see this answer in the FireCloud Forum.


Created 2016-03-10 15:07:05 | Updated 2016-03-11 14:19:30 | Tags: faq faqs firecloud

Comments (0)

Functionality

What are Google buckets and how do they connect with FireCloud? Google buckets store data in the Google Cloud Platform. Every workspace is attached to a bucket where data is stored and results are written.

Buckets are automatically generated and attached to a workspace whenever a new workspace is created (either through cloning or de novo).

Workspace data model attributes may also reference files in buckets created outside of FireCloud. It is the responsibility of the bucket owner to manage the ACLs on that bucket. Analysis results will always be written back to the bucket attached to the workspace.

If you would like support with setting up a bucket or workspace, please post your question to the FireCloud Forum.

How do I access the bucket that is associated with my workspace? On the workspace Summary tab, you can click on a link to your bucket. The link will begin with “fc” e.g., fc-098345xx-25aa-47c6-8d52-51c9ea889986d. For more information on buckets, refer to the Workspaces topic in the User Guide.

How can I give FireCloud users access to my data? You can share your workspaces with any FireCloud user, provided the user has registered, has an activated account, and you know the Gmail address or Google-managed identity that the user provided during registration.

What file types are supported on Firecloud? FireCloud is agnostic to file types. The required file types are determined by the methods, tools and workflows. Please see FireCloud Tutorials for tool- and workflow-specific inputs and outputs. If you have questions about a specific file type, you can post to the FireCloud Forum.

Does it matter what browser I use with FireCloud? Although FireCloud may run on any browsers, it will be most stable and functional on Google Chrome.

*How are compute and storage costs charged to FireCloud users?** Before you can clone or create workspaces, FireCloud administrators must create a FireCloud Google Project for you or grant you access to an existing FireCloud Google Project that they created for another user.

All compute and cloud storage costs in FireCloud are tracked through FireCloud Google Projects, which transmit charges to a Project’s Google Billing Account. When you create or clone a workspace in FireCloud, it is linked to a single FireCloud Google Project that tracks all storage and compute costs for that workspace.

FireCloud administrators can create a FireCloud Google Project if you own a Google Billing Account. Steps to set up a Google Billing Account and more information about FireCloud Google Projects can be found here.

Alternatively, FireCloud administrators can grant you access to an existing FireCloud Google Project that they created for another user. For example, if you are a member of a lab or program, your principal investigator (PI) or program manager may have requested the creation of a FireCloud Google Project.

In that case, they would email help@firecloud.org requesting to add you to their FireCloud Google Project. They would need to provide the name of the FireCloud Google Project and the Gmail address or Google-managed identity you used to register.

Please note that all FireCloud Google Projects are created by FireCloud administrators and only FireCloud administrators can grant users access to an existing FireCloud Google Project.

Also, note that if a FireCloud user (collaborator or PI) shares a workspace with you, granting you WRITER access, you can compute within that workspace. All charges incurred for that workspace will be tracked through the workspace’s FireCloud Google Project and charged to that Project’s Google Billing Account.

Can FireCloud workflow results be visualized? FireCloud currently supports Nozzle Report integration as in Firebrowse and will soon support the Integrative Genomics Viewer (IGV).

How can I see jobs in progress? You can view jobs through the workspace Monitor tab. The workspace display screen will also highlight running workspaces in blue.

How can I see if a job has failed and at what stage? The workspace display screen highlights workspaces in red to indicate failure. You can also view the workspace Monitor tab to view details of the failure.

Does FireCloud send a notification if a job fails? No. Currently, there are no automatically-generated notifications. You would need to log-in and check the status on the workspace display screen or workspace Monitor tab. If you click on the Monitor tab, you can view details of the job failure.

What is FireCloud’s workspace data model? To help you organize your data and results, FireCloud’s workspace data model provides predefined entity types, entity relationships, and entity attributes.

FireCloud Entity Types:

  • participant
  • participant set
  • sample
  • sample set
  • pair
  • pair set

FireCloud’s workspace data model also provides structure for analysis runs by requiring an entity type (e.g., pair set) on which to run an analysis.

Results from data runs can be populated directly to the workspace data model for use in downstream analyses.

For more information, refer to the Data Model section of FireCloud Basics in the User Guide.

Does my data have to live in a Google bucket to be accessible to FireCloud? Technically yes, although it does not need to live in the Google bucket owned by the workspace. Data copies (both explicit and through workspace cloning) are shallow: references are copied rather than creating new instances of the data objects. This leads to workspaces containing references to data in buckets external to the workspace. Note, however, that the results of any analysis run within a workspace will be written to the bucket attached to the workspace.


Created 2016-03-08 22:38:14 | Updated | Tags: faq faqs firecloud

Comments (0)

TCGA Data and dbGaP Authorization

Can I access TCGA data in FireCloud? TCGA open access data is available to all FireCloud users. Open access data will be found in pre-loaded workspaces. TCGA controlled access data is accessible to users who have dbGaP authorization to use controlled access data.

How do I access controlled access TCGA data in FireCloud? To access workspaces in FireCloud containing controlled access data, you must have an eRA Commons or NIH account with dbGaP authorization; and link your FireCloud account to that eRA Commons or NIH account.

How do I gain dbGaP authorization to access controlled access data? Information about applying for dbGaP authorization can be found on the dbGaP website. If you are not a principal investigator (PI), your PI may need to apply on your behalf.

What data can I put in FireCloud from a regulatory point-of-view? If the Data Use Agreement (DUA) for your data set explicitly states that the data may be used on a public cloud computing environment, you may use it on FireCloud.

FireCloud requires users to abide by all DUAs. It is the responsibility of the users to ensure that all data is used in compliance with the associated DUA.

Is FireCloud secure? Yes. FireCloud has been developed in accordance with security guidelines for a Federal Information Security Management Act (FISMA) Moderate System (http://csrc.nist.gov/groups/SMA/fisma/). Secure Sockets Layer (SSL) connections are employed for web browsers and system APIs. Data are encrypted at rest by Google Cloud Storage. A separate system security plan (SSP) will govern Google Cloud Infrastructure development in adherence to Federal Risk and Authorization Management Program (FedRAMP) guidelines (https://www.fedramp.gov/).


Created 2016-03-08 22:36:55 | Updated | Tags: faq faqs firecloud

Comments (0)

Compute Costs and Speed

What does it cost to store data on Google Cloud Platform? Uploading data does not incur any charge. Storing data costs $26/tb/month (see https://cloud.google.com/storage/pricing).

Downloading data will incur egress charges, which are charged to the associated Google Billing Account. Downloading large data can be expensive! Eventually, we will provide tools for predicting storage and egress costs.

What does it cost to compute with Google Cloud Platform? Computing costs about $0.05 per core-hour if you use less than 6.5GB RAM per core. Using more RAM will require running additional cores and result in higher charges.

Go to https://cloud.google.com/storage/pricing for more information. Eventually, we will provide tools for predicting compute costs in a workspace.


Created 2016-03-08 22:35:18 | Updated | Tags: faq faqs firecloud

Comments (0)

Troubleshooting

What is the best way to get support for FireCloud? Please post your questions to the FireCloud Forum. If you have a billing or registration question, you can email help@firecloud.org.

Why can’t I create or clone a workspace? First, you must have access to a FireCloud Google Project. All compute and storage costs in FireCloud are tracked through FireCloud Google Projects, which transmit charges to Google Billing Accounts. When you create or clone a workspace in FireCloud, it is linked to a single FireCloud Google Project that tracks all storage and compute costs for that workspace.

FireCloud administrators must create for you a new FireCloud Google Project or grant you access to an existing FireCloud Google Project. See FireCloud Google Projects and Google Billing Accounts for more information.

Why am I seeing Internal Server Error or Gateway Timeout? FireCloud may be temporarily down. Please wait five minutes and refresh your screen in FireCloud. If you still see this message, please let us know through the FireCloud Forum

No articles to display.

No articles to display.