Hello,
sorry if i missed the same problem in other threads in the forum... but we are having trouble running BaseRecalibrator in a sample and i couldn't find the solution.
I tried many steps and here is what i've found until now:
1 - Other samples work fine
2 - Running picard ValidateSamFile in realigned.bam (after IndelRealigner) gives many erros :
2a - Mate negative strand flag does not match read negative strand flag of mate
2b - Mate alignment does not match alignment start of mate
3c - Value was put into PairInfoMap more than once. (fatal)
3 - Running BaseRecalibrator with option -L 1:428-249250621 works fine!
After the fact that -L works fine i discarded the problem in vcf files and reference file. I don't know how to go further in this investigation since GATK 1 realined.bam also gives me the errors in (2) and those error are peanuts comparing the total number of reads.
The big difference here is that we're are using bwa7.
Any ideas? Thanks!
(i'm filtering out "secondary hits" given by bwa7 and will update this thread, if it works it may be helpful in the future)
GATK output:
INFO 14:11:47,441 HelpFormatter - --------------------------------------------------------------------------------
INFO 14:11:47,443 HelpFormatter - The Genome Analysis Toolkit (GATK) v2.4-9-g532efad, Compiled 2013/03/19 07:35:36
INFO 14:11:47,443 HelpFormatter - Copyright (c) 2010 The Broad Institute
INFO 14:11:47,443 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk
INFO 14:11:47,447 HelpFormatter - Program Args: -nct 8 -T BaseRecalibrator -I /mnt/work/rlb/pac661825//OUT_661825.realigned.bam -R ../data/databases//1KGP/GRCh37_female_exome_mt1kg.fasta --knownSites ../data/databases//dbSNP/dbSNP_137/00-All.vcf -o /mnt/work/rlb/pac661825//OUT_661825.grp
INFO 14:11:47,447 HelpFormatter - Date/Time: 2013/03/26 14:11:47
INFO 14:11:47,447 HelpFormatter - --------------------------------------------------------------------------------
INFO 14:11:47,447 HelpFormatter - --------------------------------------------------------------------------------
INFO 14:11:47,458 ArgumentTypeDescriptor - Dynamically determined type of ../data/databases/dbSNP/dbSNP_137/00-All.vcf to be VCF
INFO 14:11:47,500 GenomeAnalysisEngine - Strictness is SILENT
INFO 14:11:47,558 GenomeAnalysisEngine - Downsampling Settings: No downsampling
INFO 14:11:47,565 SAMDataSource$SAMReaders - Initializing SAMRecords in serial
INFO 14:11:47,577 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.01
INFO 14:11:47,587 RMDTrackBuilder - Loading Tribble index from disk for file ../data/databases/dbSNP/dbSNP_137/00-All.vcf
INFO 14:11:47,704 MicroScheduler - Running the GATK in parallel mode with 8 total threads, 8 CPU thread(s) for each of 1 data thread(s), of 8 processors available on this machine
INFO 14:11:47,745 GenomeAnalysisEngine - Creating shard strategy for 1 BAM files
INFO 14:11:47,750 GenomeAnalysisEngine - Done creating shard strategy
INFO 14:11:47,750 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 14:11:47,750 ProgressMeter - Location processed.reads runtime per.1M.reads completed total.runtime remaining
INFO 14:11:47,773 BaseRecalibrator - The covariates being used here:
INFO 14:11:47,773 BaseRecalibrator - ReadGroupCovariate
INFO 14:11:47,773 BaseRecalibrator - QualityScoreCovariate
INFO 14:11:47,773 BaseRecalibrator - ContextCovariate
INFO 14:11:47,774 ContextCovariate - Context sizes: base substitution model 2, indel substitution model 3
INFO 14:11:47,774 BaseRecalibrator - CycleCovariate
INFO 14:11:47,776 ReadShardBalancer$1 - Loading BAM index data for next contig
INFO 14:11:47,777 ReadShardBalancer$1 - Done loading BAM index data for next contig
INFO 14:12:18,626 ProgressMeter - 1:15956928 1.10e+06 30.0 s 28.0 s 0.5% 95.1 m 94.6 m
INFO 14:12:48,655 ProgressMeter - 1:34102053 2.70e+06 60.0 s 22.0 s 1.1% 89.0 m 88.0 m
INFO 14:13:18,685 ProgressMeter - 1:59096606 4.50e+06 90.0 s 20.0 s 1.9% 77.1 m 75.6 m
INFO 14:13:48,714 ProgressMeter - 1:103467532 5.90e+06 120.0 s 20.0 s 3.4% 58.7 m 56.7 m
INFO 14:14:18,745 ProgressMeter - 1:153234111 7.50e+06 2.5 m 20.0 s 5.0% 49.5 m 47.0 m
INFO 14:14:48,774 ProgressMeter - 1:172414433 9.30e+06 3.0 m 19.0 s 5.7% 53.1 m 50.1 m
INFO 14:15:19,054 ProgressMeter - 1:208266349 1.10e+07 3.5 m 19.0 s 6.9% 51.3 m 47.8 m
INFO 14:15:49,095 ProgressMeter - 1:247611815 1.27e+07 4.0 m 19.0 s 8.2% 49.3 m 45.2 m
INFO 14:15:56,507 GATKRunReport - Uploaded run statistics report to AWS S3
org.broadinstitute.sting.utils.exceptions.ReviewedStingException: START (0) > (-1) STOP -- this should never happen -- call Mauricio! at org.broadinstitute.sting.utils.clipping.ReadClipper.hardClipByReferenceCoordinates(ReadClipper.java:537) at org.broadinstitute.sting.utils.clipping.ReadClipper.hardClipByReferenceCoordinatesLeftTail(ReadClipper.java:176) at org.broadinstitute.sting.utils.clipping.ReadClipper.hardClipAdaptorSequence(ReadClipper.java:389) at org.broadinstitute.sting.utils.clipping.ReadClipper.hardClipAdaptorSequence(ReadClipper.java:392) at org.broadinstitute.sting.gatk.walkers.bqsr.BaseRecalibrator.map(BaseRecalibrator.java:244) at org.broadinstitute.sting.gatk.walkers.bqsr.BaseRecalibrator.map(BaseRecalibrator.java:131) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano$TraverseReadsMap.apply(TraverseReadsNano.java:230) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano$TraverseReadsMap.apply(TraverseReadsNano.java:218) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler$ReadMapReduceJob.run(NanoScheduler.java:471) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679)
INFO 17:40:38,765 ProgressMeter - chrX:147886444 3.21e+07 5.5 h 10.3 m 97.9% 5.6 h 7.1 m INFO 17:41:08,775 ProgressMeter - chrX:151653849 3.21e+07 5.5 h 10.3 m 98.0% 5.6 h 6.7 m INFO 17:41:38,785 ProgressMeter - chrX:153812787 3.22e+07 5.5 h 10.3 m 98.1% 5.6 h 6.5 m
java.lang.NoClassDefFoundError: net/sf/samtools/util/CloserUtil
at net.sf.picard.util.PeekableIterator.close(PeekableIterator.java:46)
at net.sf.picard.sam.MergingSamRecordIterator.addIfNotEmpty(MergingSamRecordIterator.java:169)
at net.sf.picard.sam.MergingSamRecordIterator.next(MergingSamRecordIterator.java:125)
at net.sf.picard.sam.MergingSamRecordIterator.next(MergingSamRecordIterator.java:39)
at org.broadinstitute.sting.gatk.iterators.PrivateStringSAMCloseableIterator.next(StingSAMIteratorAdapter.java:100)
at org.broadinstitute.sting.gatk.iterators.PrivateStringSAMCloseableIterator.next(StingSAMIteratorAdapter.java:84)
at org.broadinstitute.sting.gatk.datasources.reads.SAMDataSource$ReleasingIterator.next(SAMDataSource.java:1091)
at org.broadinstitute.sting.gatk.datasources.reads.SAMDataSource$ReleasingIterator.next(SAMDataSource.java:1057)
at org.broadinstitute.sting.gatk.filters.CountingFilteringIterator.getNextRecord(CountingFilteringIterator.java:105)
at org.broadinstitute.sting.gatk.filters.CountingFilteringIterator.next(CountingFilteringIterator.java:81)
at org.broadinstitute.sting.gatk.filters.CountingFilteringIterator.next(CountingFilteringIterator.java:41)
at org.broadinstitute.sting.gatk.iterators.PrivateStringSAMCloseableIterator.next(StingSAMIteratorAdapter.java:100)
at org.broadinstitute.sting.gatk.iterators.PrivateStringSAMCloseableIterator.next(StingSAMIteratorAdapter.java:84)
at org.broadinstitute.sting.gatk.iterators.VerifyingSamIterator.next(VerifyingSamIterator.java:24)
at org.broadinstitute.sting.gatk.iterators.VerifyingSamIterator.next(VerifyingSamIterator.java:12)
at org.broadinstitute.sting.utils.baq.ReadTransformingIterator.next(ReadTransformingIterator.java:36)
at org.broadinstitute.sting.utils.baq.ReadTransformingIterator.next(ReadTransformingIterator.java:15)
at net.sf.picard.util.PeekableIterator.advance(PeekableIterator.java:71)
at net.sf.picard.util.PeekableIterator.next(PeekableIterator.java:57)
at org.broadinstitute.sting.gatk.datasources.reads.ReadShard.fill(ReadShard.java:135)
at org.broadinstitute.sting.gatk.datasources.reads.ReadShardBalancer$1.advance(ReadShardBalancer.java:153)
at org.broadinstitute.sting.gatk.datasources.reads.ReadShardBalancer$1.next(ReadShardBalancer.java:116)
at org.broadinstitute.sting.gatk.datasources.reads.ReadShardBalancer$1.next(ReadShardBalancer.java:75)
at org.broadinstitute.sting.gatk.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:65)
at org.broadinstitute.sting.gatk.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:281)
at org.broadinstitute.sting.gatk.CommandLineExecutable.execute(CommandLineExecutable.java:113)
at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:237)
at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:147)
at org.broadinstitute.sting.gatk.CommandLineGATK.main(CommandLineGATK.java:91)
Caused by: java.lang.ClassNotFoundException: net.sf.samtools.util.CloserUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
... 29 more
Caused by: java.util.zip.ZipException: error reading zip file
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1200(ZipFile.java:29)
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447)
at java.util.zip.ZipFile$1.fill(ZipFile.java:230)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
at java.util.jar.Manifest$FastInputStream.fill(Manifest.java:422)
at java.util.jar.Manifest$FastInputStream.readLine(Manifest.java:358)
at java.util.jar.Manifest$FastInputStream.readLine(Manifest.java:390)
at java.util.jar.Attributes.read(Attributes.java:359)
at java.util.jar.Manifest.read(Manifest.java:182)
at java.util.jar.Manifest.
Any Idea what is the solution??
Hi,
I had a seemingly random RUNTIME ERROR previously (mentioned on this thread http://gatkforums.broadinstitute.org/discussion/1860/runtime-error-in-baserecalibrator-version-2-2-8-g99996f2), but I'm seeing this more and more for UnifiedGenotyper runs.
It seems that every 6-10 runs I get one or two that fail as below. Re-runinng seems to work correctly, but this is annoying to find that a 12hr job has died randomly. Please can this be investigated.
java.lang.NullPointerException at java.util.concurrent.locks.AbstractQueuedSynchronizer.hasQueuedPredecessors(AbstractQueuedSynchronizer.java:1453) at java.util.concurrent.locks.ReentrantLock$FairSync.tryAcquire(ReentrantLock.java:240) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1136) at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:229) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) at java.util.concurrent.PriorityBlockingQueue.peek(PriorityBlockingQueue.java:286) at org.broadinstitute.sting.utils.nanoScheduler.Reducer.reduceNextValueInQueue(Reducer.java:89) at org.broadinstitute.sting.utils.nanoScheduler.Reducer.reduceAsMuchAsPossible(Reducer.java:120) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler$MapReduceJob.run(NanoScheduler.java:510) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636)
My commandline is this: java -Xmx8g -jar GenomeAnalysisTKLite.jar -nct 4 -T UnifiedGenotyper --genotype_likelihoods_model BOTH --genotyping_mode DISCOVERY -R fastafile -I infile -o outfile --dbsnp dbsnp_137.b37.vcf -stand_call_conf 30 -stand_emit_conf 30
This usually coincides with errors is calling home code, although I don't know if it's cause or effect.
"INFO 16:41:33,533 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty INFO 16:41:33,534 HttpMethodDirector - Retrying request INFO 16:41:33,716 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty INFO 16:41:33,716 HttpMethodDirector - Retrying request INFO 16:41:33,897 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty INFO 16:41:33,897 HttpMethodDirector - Retrying request INFO 16:41:34,077 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty INFO 16:41:34,085 HttpMethodDirector - Retrying request INFO 16:41:34,267 HttpMethodDirector - I/O exception (javax.net.ssl.SSLException) caught when processing request: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty INFO 16:41:34,267 HttpMethodDirector - Retrying request "
Hi, I am trying to run the base recalibrator. For some of my sequences, it work perfectly, but for others, it's always crashing and giving the following error message :
INFO 16:49:21,718 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:49:21,720 HelpFormatter - The Genome Analysis Toolkit (GATK) v2.3-6-gebbba25, Compiled 2013/01/08 19:29:18
INFO 16:49:21,720 HelpFormatter - Copyright (c) 2010 The Broad Institute
INFO 16:49:21,720 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk
INFO 16:49:21,724 HelpFormatter - Program Args: -T BaseRecalibrator -I alnSortedNoDupRGRealigned1.bam -l INFO -R 1.198.A.contig22.fa -knownSites alnSortedNoDupRGRealigned1_t0.01.vcf -o test.rcl -fixMisencodedQuals --filter_mismatching_base_and_quals
INFO 16:49:21,725 HelpFormatter - Date/Time: 2013/02/11 16:49:21
INFO 16:49:21,725 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:49:21,725 HelpFormatter - --------------------------------------------------------------------------------
INFO 16:49:21,737 ArgumentTypeDescriptor - Dynamically determined type of alnSortedNoDupRGRealigned1_t0.01.vcf to be VCF
INFO 16:49:21,744 GenomeAnalysisEngine - Strictness is SILENT
INFO 16:49:21,970 GenomeAnalysisEngine - Downsampling Settings: No downsampling
INFO 16:49:21,977 SAMDataSource$SAMReaders - Initializing SAMRecords in serial
INFO 16:49:21,992 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.01
INFO 16:49:22,005 RMDTrackBuilder - Loading Tribble index from disk for file alnSortedNoDupRGRealigned1_t0.01.vcf
INFO 16:49:22,040 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 16:49:22,040 ProgressMeter - Location processed.reads runtime per.1M.reads completed total.runtime remaining
INFO 16:49:22,134 BaseRecalibrator - The covariates being used here:
INFO 16:49:22,134 BaseRecalibrator - ReadGroupCovariate
INFO 16:49:22,134 BaseRecalibrator - QualityScoreCovariate
INFO 16:49:22,134 BaseRecalibrator - ContextCovariate
INFO 16:49:22,135 ContextCovariate - Context sizes: base substitution model 2, indel substitution model 3
INFO 16:49:22,135 BaseRecalibrator - CycleCovariate
INFO 16:49:22,137 NestedIntegerArray - Creating NestedIntegerArray with dimensions [1, 3]
INFO 16:49:22,138 NestedIntegerArray - Pre-allocating first 2 dimensions
INFO 16:49:22,139 NestedIntegerArray - Done pre-allocating first 2 dimensions
INFO 16:49:22,139 NestedIntegerArray - Creating NestedIntegerArray with dimensions [1, 94, 3]
INFO 16:49:22,139 NestedIntegerArray - Pre-allocating first 2 dimensions
INFO 16:49:22,140 NestedIntegerArray - Done pre-allocating first 2 dimensions
INFO 16:49:22,140 NestedIntegerArray - Creating NestedIntegerArray with dimensions [1, 94, 1012, 3]
INFO 16:49:22,140 NestedIntegerArray - Pre-allocating first 2 dimensions
INFO 16:49:22,140 NestedIntegerArray - Done pre-allocating first 2 dimensions
INFO 16:49:22,140 NestedIntegerArray - Creating NestedIntegerArray with dimensions [1, 94, 1002, 3]
INFO 16:49:22,140 NestedIntegerArray - Pre-allocating first 2 dimensions
INFO 16:49:22,141 NestedIntegerArray - Done pre-allocating first 2 dimensions
INFO 16:49:22,145 ReadShardBalancer$1 - Loading BAM index data for next contig
INFO 16:49:22,147 ReadShardBalancer$1 - Done loading BAM index data for next contig
INFO 16:49:24,487 GATKRunReport - Uploaded run statistics report to AWS S3
java.lang.ArrayIndexOutOfBoundsException: -31 at org.broadinstitute.sting.utils.baq.BAQ.calcEpsilon(BAQ.java:158) at org.broadinstitute.sting.utils.baq.BAQ.hmm_glocal(BAQ.java:225) at org.broadinstitute.sting.utils.baq.BAQ.calcBAQFromHMM(BAQ.java:542) at org.broadinstitute.sting.utils.baq.BAQ.calcBAQFromHMM(BAQ.java:595) at org.broadinstitute.sting.utils.baq.BAQ.calcBAQFromHMM(BAQ.java:530) at org.broadinstitute.sting.utils.baq.BAQ.baqRead(BAQ.java:663) at org.broadinstitute.sting.gatk.walkers.bqsr.BaseRecalibrator.calculateBAQArray(BaseRecalibrator.java:428) at org.broadinstitute.sting.gatk.walkers.bqsr.BaseRecalibrator.map(BaseRecalibrator.java:243) at org.broadinstitute.sting.gatk.walkers.bqsr.BaseRecalibrator.map(BaseRecalibrator.java:112) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano$TraverseReadsMap.apply(TraverseReadsNano.java:203) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano$TraverseReadsMap.apply(TraverseReadsNano.java:191) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler.executeSingleThreaded(NanoScheduler.java:248) at org.broadinstitute.sting.utils.nanoScheduler.NanoScheduler.execute(NanoScheduler.java:219) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano.traverse(TraverseReadsNano.java:91) at org.broadinstitute.sting.gatk.traversals.TraverseReadsNano.traverse(TraverseReadsNano.java:55) at org.broadinstitute.sting.gatk.executive.LinearMicroScheduler.execute(LinearMicroScheduler.java:83) at org.broadinstitute.sting.gatk.GenomeAnalysisEngine.execute(GenomeAnalysisEngine.java:281) at org.broadinstitute.sting.gatk.CommandLineExecutable.execute(CommandLineExecutable.java:113) at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:237) at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:147) at org.broadinstitute.sting.gatk.CommandLineGATK.main(CommandLineGATK.java:91)
Here is the command line I am using: java -Xmx4g -jar /mit/sjlabrie/software/GenomeAnalysisTK-2.3-6-gebbba25/GenomeAnalysisTK.jar \ -T BaseRecalibrator \ -I alnSortedNoDupRGRealigned1.bam \ -l INFO \ -R 1.198.A.contig22.fa \ -knownSites alnSortedNoDupRGRealigned1_t0.01.vcf \ -o test.rcl \ -fixMisencodedQuals \ --filter_mismatching_base_and_quals \
Any Idea what's going on ?
Thank you,
Simon
