Hello,
During running of the depthOfCoverage tool, I get the error: /tmp/RsQHCt1W: No space left on device
I have tried changing the TMPDIR environment variable (and exporting) but eventually I get the same error. Is there a way to change the temporary directory that GATK uses?
I'm running GATK v2.1-8-g5efb575 on a Linux system.
Thanks, Rick
While I have been able to work around this issue, I am still curious as to what is causing it in the first place.
I am currently engaged in variant calling on a rather large set of exome sequenced samples (prepared according to GATK best practices 4), but I have run into some issues with UnifiedGenotyper.
Put simply, I am able to run any of our input intervals through UnfiedGenotyper with a single data thread, and either 1 or 24 CPU threads with no trouble. Upon looking at the actual memory usage of the process while running, I decided to try increasing the number of data threads as it appeared that I could easily run 4 threads in the heap I have available. However, regardless of the thread or memory settings, when I raise the number of data threads beyond 1, once the initialization phase has completed, I immediately get a Java core dump. The GATK invocation I use is as follows:
java -Djava.io.tmpdir=$TMP -Xmx48G -jar $GATK -T UnifiedGenotyper \
-R $REF \
--dbsnp $ROD \
${GATKArg} \
-o ${TMP}/${BASENAME}/${BASENAME}.${INTERVAL}.raw.variants.vcf \
-L ${INTERVAL} \
-nt 4 \
-nct 8 \
-glm BOTH \
-A DepthOfCoverage \
-A AlleleBalance \
-A HaplotypeScore \
-A HomopolymerRun \
-A MappingQualityZero \
-A QualByDepth \
-A RMSMappingQuality \
-A SpanningDeletions \
-A MappingQualityRankSumTest \
-A ReadPosRankSumTest \
-A FisherStrand \
-A InbreedingCoeff
For reference, this is GATK version 2.3-9, JRE 1.6.
The error I receive is shown below. Any insight into this issue would be greatly appreciated.
Sincerely, Jason Kost
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (exceptions.cpp:364), pid=14830, tid=1113614656
# Error: ExceptionMark destructor expects no pending exceptions
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode linux-amd64 )
# Can not save log file, dump to screen..
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (exceptions.cpp:364), pid=14830, tid=1113614656
# Error: ExceptionMark destructor expects no pending exceptions
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode linux-amd64 )
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00000000490c3800): JavaThread "S3Put-Thread" daemon [_thread_in_vm, id=14864, stack(0x0000000042506000,0x0000000042607000)]
Stack: [0x0000000042506000,0x0000000042607000], sp=0x0000000042602bd0, free space=1010k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x6bd1ef]
V [libjvm.so+0x2be556]
V [libjvm.so+0x3082dd]
V [libjvm.so+0x2592d0]
V [libjvm.so+0x258922]
V [libjvm.so+0x2589a6]
V [libjvm.so+0x25a37e]
V [libjvm.so+0x650eab]
V [libjvm.so+0x64f222]
V [libjvm.so+0x64e101]
V [libjvm.so+0x64dd63]
V [libjvm.so+0x42c8e9]
V [libjvm.so+0x43310e]
V [libjvm.so+0x411d68]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.lang.ClassLoader.findBootstrapClass(Ljava/lang/String;)Ljava/lang/Class;+0
j java.lang.ClassLoader.findBootstrapClass0(Ljava/lang/String;)Ljava/lang/Class;+23
j java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+32
j java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+23
j sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+41
j java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
j java.lang.ClassLoader.loadClassInternal(Ljava/lang/String;)Ljava/lang/Class;+2
v ~StubRoutines::call_stub
j org.broadinstitute.sting.gatk.phonehome.GATKRunReport$S3PutRunnable.run()V+6
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
=>0x00000000490c3800 JavaThread "S3Put-Thread" daemon [_thread_in_vm, id=14864, stack(0x0000000042506000,0x0000000042607000)]
0x0000000049207000 JavaThread "Thread-3" [_thread_in_native, id=14863, stack(0x0000000042405000,0x0000000042506000)]
0x0000000049384800 JavaThread "Thread-2" [_thread_blocked, id=14862, stack(0x0000000042304000,0x0000000042405000)]
0x00000000493fe000 JavaThread "ProgressMeterDaemon" daemon [_thread_blocked, id=14858, stack(0x0000000042203000,0x0000000042304000)]
0x00002ab6c0026000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=14856, stack(0x0000000042001000,0x0000000042102000)]
0x00002ab6c0023000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14855, stack(0x0000000041bd6000,0x0000000041cd7000)]
0x00002ab6c0020800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=14854, stack(0x0000000041ad5000,0x0000000041bd6000)]
0x00002ab6c001e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=14853, stack(0x00000000419d4000,0x0000000041ad5000)]
0x00000000484b3000 JavaThread "Finalizer" daemon [_thread_blocked, id=14852, stack(0x0000000041f00000,0x0000000042001000)]
0x00000000484ab800 JavaThread "Reference Handler" daemon [_thread_blocked, id=14851, stack(0x0000000041dff000,0x0000000041f00000)]
0x00000000481a8800 JavaThread "main" [_thread_blocked, id=14831, stack(0x0000000040450000,0x0000000040551000)]
Other Threads:
0x00000000484a6800 VMThread [stack: 0x0000000041cfe000,0x0000000041dff000] [id=14850]
0x00002ab6c0029000 WatcherThread [stack: 0x0000000042102000,0x0000000042203000] [id=14857]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 63360K, used 3359K [0x00002ab2b3440000, 0x00002ab2ba990000, 0x00002ab6b3440000)
eden space 16384K, 20% used [0x00002ab2b3440000,0x00002ab2b3787fb8,0x00002ab2b4440000)
from space 46976K, 0% used [0x00002ab2b7550000,0x00002ab2b7550000,0x00002ab2ba330000)
to space 50240K, 0% used [0x00002ab2b4440000,0x00002ab2b4440000,0x00002ab2b7550000)
PSOldGen total 353536K, used 271234K [0x00002aaab3440000, 0x00002aaac8d80000, 0x00002ab2b3440000)
object space 353536K, 76% used [0x00002aaab3440000,0x00002aaac3d20bd8,0x00002aaac8d80000)
PSPermGen total 21504K, used 16865K [0x00002aaaae040000, 0x00002aaaaf540000, 0x00002aaab3440000)
object space 21504K, 78% used [0x00002aaaae040000,0x00002aaaaf0b8538,0x00002aaaaf540000)
Dynamic libraries:
Can not get library information for pid = 14864
VM Arguments:
jvm_args: -Djava.io.tmpdir=/home/kostj/scratch/temp/ -Xmx48G
java_command: /home/kostj/nearline/gatk2/GenomeAnalysisTK.jar -T UnifiedGenotyper -R /home/kostj/nearline/hg19/hg19.fasta --dbsnp /home/kostj/nearline/hg19/gatk2/dbsnp_137.hg19.vcf -I /home/kostj/scratch/temp//SPr0059/SPr0059.reduced.bam -I /home/kostj/scratch/temp//SP3608/SP3608.reduced.bam -I /home/kostj/scratch/temp//SP3528/SP3528.reduced.bam -I /home/kostj/scratch/temp//SP3508/SP3508.reduced.bam -I /home/kostj/scratch/temp//SP3474/SP3474.reduced.bam -I /home/kostj/scratch/temp//SP3277/SP3277.reduced.bam -I /home/kostj/scratch/temp//SP3255/SP3255.reduced.bam -I /home/kostj/scratch/temp//SP3148/SP3148.reduced.bam -I /home/kostj/scratch/temp//sp3127/sp3127.reduced.bam -I /home/kostj/scratch/temp//SP3070/SP3070.reduced.bam -I /home/kostj/scratch/temp//sp3035/sp3035.reduced.bam -I /home/kostj/scratch/temp//SP0618/SP0618.reduced.bam -I /home/kostj/scratch/temp//SNc0063/SNc0063.reduced.bam -I /home/kostj/scratch/temp//SNc0036/SNc0036.reduced.bam -I /home/kostj/scratch/temp//SMa0078/SMa0078.reduced.bam -I /home/kostj/scratch/temp//SMa0020/SMa0020.reduced.bam -I /home/kostj/scratch/temp//SLA969/SLA969.reduced.bam -I /home/kostj/scratch/temp//SLA966/SLA966.reduced.bam -I /home/kostj/scratch/temp//SLA956/SLA956.reduced.bam -I /home/kostj/scratch/temp//SLA929/SLA929.reduced.bam -I /home/kostj/scratch/temp//SLA922/SLA922.reduced.bam -I /home/kostj/scratch/temp//SLA885/SLA885.reduced.bam -I /home/kostj/scratch/temp//SLA877/SLA877.reduced.bam -I /home/kostj/scratch/temp//SLA863/SLA863.reduced.bam -I /home/kostj/scratch/temp//SLA861/SLA861.reduced.bam -I /home/kostj/scratch/temp//SLA833/SLA833.reduced.bam -I /home/kostj/scratch/temp//SLA809/SLA809.reduced.bam -I /home/kostj/scratch/temp//SLA808/SLA808.reduced.bam -I /home/kostj/scratch/temp//SLA807/SLA807.reduced.bam -I /home/kostj/scratch/temp//SLA798/SLA798.reduced.bam -I /home/kostj/scratch/temp//SLA791/SLA791.reduced.bam -I /home/kostj/scratch/temp//SLA783/SLA783.reduced.bam -I /home/kostj/scratch/temp//SLA767/SLA767.redu
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/share/bin/java/bin:/home/kostj/nearline/R/bin/:/tmp/8474173.1.chassis12.q:/share/bin/gcc/bin:/share/bin:/opt/SUNWhpc/HPC8.1/gnu/bin/:/share/bin/intel/bin/intel64:/share/bin/gmp/include:/share/bin/mpfr/include:/sge/bin/lx24-amd64:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin
LD_LIBRARY_PATH=/share/bin/jre1.6.0_16/lib/amd64/server:/share/bin/jre1.6.0_16/lib/amd64:/share/bin/jre1.6.0_16/../lib/amd64:/lib:/usr/lib:/usr/local/lib:/opt/SUNWhpc/HPC8.1/gnu/lib/lib64:/share/bin/gcc/lib64:/share/bin/gcc/lib:/share/bin/intel/fc/10.1.015/lib:/share/bin/intel/fc/10.1.015/licenses:/share/bin/intel/cc/10.1.015/lib:/share/bin/mpfr/lib:/share/bin/gmp/lib:/share/lib:
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x6bddc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x594f90], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x597750], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x5974a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Linux
uname:Linux 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64
libc:glibc 2.5 NPTL 2.5
rlimit: STACK infinity, CORE infinity, NPROC 399360, NOFILE 1024, AS infinity
load average:0.00 0.00 0.00
CPU:total 24 (16 cores per cpu, 2 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, ht
Memory: 4k page, physical 49451764k(23657140k free), swap 16779884k(16697572k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (14.2-b01) for linux-amd64 JRE (1.6.0_16-b01), built on Jul 31 2009 05:52:33 by "java_re" with gcc 3.2.2 (SuSE Linux)
time: Mon Mar 11 16:00:00 2013
elapsed time: 923 seconds
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
/sge/default/spool/pem610-040/job_scripts/8474173: line 83: 14830 Aborted (core dumped) java -Djava.io.tmpdir=$TMP -Xmx48G -jar $GATK -T UnifiedGenotyper -R $REF --dbsnp $ROD ${GATKArg} -o ${TMP}/${BASENAME}/${BASENAME}.${INTERVAL}.raw.variants.vcf -L ${INTERVAL} -nt 2 -nct 12 -glm BOTH -A DepthOfCoverage -A AlleleBalance -A HaplotypeScore -A HomopolymerRun -A MappingQualityZero -A QualByDepth -A RMSMappingQuality -A SpanningDeletions -A MappingQualityRankSumTest -A ReadPosRankSumTest -A FisherStrand -A InbreedingCoeff
I am facing a "fatal error by java runtime enviormnet" after using GATK DataProcessingPipeline, my java version is
> java version "1.6.0_35"
> Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)
>
and I am using GATK 2.3-6-gebbba25 The pipeline spit out the following error,
> lelOldGC' '-XX:ParallelGCThreads=4' '-XX:GCTimeLimit=50' '-XX:GCHeapFreeLimit=10' '-Djava.io.tmpdir=/medpop/mpg-psrl/Parabase/tmp' '-cp' '/medpop/mpg-psrl/Parabase/Tools/Queue-2.3-5-g49ed93c/Queue.jar' 'org.broadinstitute.sting.gatk.CommandLineGATK' '-T' 'RealignerTargetCreator' '-I' '/medpop/mpg-psrl/Parabase/MyRuns/EX_c1004CARa_1ln_hg19._new.1.realigned.rg.bam' '-R' '/medpop/mpg-psrl/Parabase/hg19/ucsc.hg19.fasta' '-o' '/medpop/mpg-psrl/Parabase/MyRuns/EX_c1004CARa_1ln_hg19.EX_c1004CARa_1ln_hg19.intervals' '-known' '/medpop/mpg-psrl/Parabase/hg19/dbsnp_137.hg19.vcf' '-mismatch' '0.0'
> INFO 11:32:30,896 FunctionEdge - Output written to /medpop/mpg-psrl/Parabase/MyRuns/EX_c1004CARa_1ln_hg19.EX_c1004CARa_1ln_hg19.intervals.out
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00002ae7a5a50380, pid=562, tid=47174397446800
> #
> # JRE version: 6.0_35-b10
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.10-b01 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # V [libjvm.so+0x712380] SR_handler(int, siginfo*, ucontext*)+0x30
> #
> # An error report file with more information is saved as:
> # /medpop/mpg-psrl/Parabase/MyRuns/hs_err_pid562.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
the log file is lengthy; but If you like to have a look I can paste here it later on, thanks for your support,