I've been running Queue using the old DataProcessingPipeline.scala script (unmodified) for over a day now, and I'm starting to think there's an infinite loop. The output keeps adding Qnodes. Right now it's up to almost 1700000 QNodes. I've run the same script before with 1000 smaller bam files (about 1/2Gb each) and that seemed to work fine. Now I'm trying to run it on 1000 exomes, each on average 8Gb. I'm using the following options:
Here's the output:
INFO 11:09:17,548 HelpFormatter - Queue v2.7-4-g6f46d11, Compiled 2013/10/10 17 :29:52 INFO 11:09:17,548 HelpFormatter - Copyright (c) 2012 The Broad Institute INFO 11:09:17,549 HelpFormatter - For support and documentation go to http://ww w.broadinstitute.org/gatk DEBUG 11:09:17,549 HelpFormatter - Current directory: /cluster/ifs/projects/Exom es/Biesecker_CS_WE/mito_express/ES/tmp INFO 11:09:17,549 HelpFormatter - Program Args: -S /home/singhln/Projects/ES/sr c/DataProcessingPipeline.scala -bwa /home/singhln/bin/bwa -bt 10 -i /home/singhl n/Projects/ES/Data/allalignedbams.list -R /home/singhln/Projects/ES/Data/human_g 1k_v37.fasta -D /home/singhln/Projects/ES/Data/dbsnp_137.b37.vcf -p CS -bwape -l og cses.log -gv -qsub -startFromScratch -jobReport queuereport.txt -memLimit 4 - tempDir /home/singhln/Projects/ES/tmp -runDir /home/singhln/Projects/ES/Dedup -l DEBUG -run INFO 11:09:17,550 HelpFormatter - Date/Time: 2013/12/03 11:09:17
INFO 11:09:17,561 QCommandLine - Scripting DataProcessingPipeline DEBUG 11:10:46,667 QGraph - adding QNode: 0 DEBUG 11:10:46,859 QGraph - adding QNode: 100 DEBUG 11:10:47,024 QGraph - adding QNode: 200 DEBUG 11:10:47,136 QGraph - adding QNode: 300 DEBUG 11:10:47,241 QGraph - adding QNode: 400 ... ... DEBUG 10:54:46,948 QGraph - adding QNode: 1647200 DEBUG 10:55:11,294 QGraph - adding QNode: 1647300 DEBUG 10:55:23,623 QGraph - adding QNode: 1647400 DEBUG 10:55:40,081 QGraph - adding QNode: 1647500 DEBUG 10:55:52,678 QGraph - adding QNode: 1647600 DEBUG 10:56:02,486 QGraph - adding QNode: 1647700
I'm not even sure how I'd go about debugging this or if this is normal, but it does seem very strange to me. No output seems to have been created during the last 24 hours either, other than the log file.
Thanks for any help, -Larry.