HaplotypeCaller using more threads than requested
Answered
Hi
I'm running GATK v 4.2.0.0 on a SGE cluster, requesting max of 1 thread for java garbage collection and 1 for native-pair-hmm-threads when running HaplotypeCaller. However, GATK prefers to gobble as many threads as possible (>16), which is not making me very popular. I'm calling GATK as follows:
gatk --java-options "-XX:ConcGCThreads=$NSLOTS -XX:ParallelGCThreads=$NSLOTS -Xmx28g" HaplotypeCaller \
-R /ref.fasta -I $reads --ERC GVCF --native-pair-hmm-threads $NSLOTS --sample-ploidy $ploidy -A AlleleFraction -O ./VCALL/$sample/$sample.$ctg.$bpi.$bpe.raw.g.vcf.gz -L $itv
Anyone know how to limit the threads to the actual number requested?
Cheers
D
-
A few more details. I've also run this with serial garbage collection, requesting 1 thread for --native-pair-hmm:
gatk --java-options "-XX:+UseSerialGC -Xmx28g" HaplotypeCaller \
-R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I $reads --ERC GVCF \
--native-pair-hmm-threads 1 --sample-ploidy $ploidy -A AlleleFraction -O ./VCALL/$sample/$sample.$ctg.$bpi.$bpe.raw.g.vcf.gz -L $itvThe GATK ouput is below:
WARNING: We recommend that you use a minimum of 4 GB of virtual memory when running Java 1.8.0_74 on Eddie. Please see the following for details:
https://www.wiki.ed.ac.uk/display/ResearchServices/Java
Using GATK jar /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -XX:+UseSerialGC -Xmx28g -jar /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar Haplo
typeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz -L c
arnica.NW_020555920.1:1-9803
12:07:48.587 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Apr 13, 2021 12:07:48 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
12:07:48.767 INFO HaplotypeCaller - ------------------------------------------------------------
12:07:48.768 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.0.0
12:07:48.768 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
12:07:48.768 INFO HaplotypeCaller - Executing as dwragg@node2i15.ecdf.ed.ac.uk on Linux v3.10.0-862.14.4.el7.x86_64 amd64
12:07:48.768 INFO HaplotypeCaller - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_74-b02
12:07:48.768 INFO HaplotypeCaller - Start Date/Time: 13 April 2021 12:07:48 BST
12:07:48.769 INFO HaplotypeCaller - ------------------------------------------------------------
12:07:48.769 INFO HaplotypeCaller - ------------------------------------------------------------
12:07:48.769 INFO HaplotypeCaller - HTSJDK Version: 2.24.0
12:07:48.769 INFO HaplotypeCaller - Picard Version: 2.25.0
12:07:48.769 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
12:07:48.769 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
12:07:48.769 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
12:07:48.770 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
12:07:48.770 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
12:07:48.770 INFO HaplotypeCaller - Deflater: IntelDeflater
12:07:48.770 INFO HaplotypeCaller - Inflater: IntelInflater
12:07:48.770 INFO HaplotypeCaller - GCS max retries/reopens: 20
12:07:48.770 INFO HaplotypeCaller - Requester pays: disabled
12:07:48.770 INFO HaplotypeCaller - Initializing engine
12:07:49.258 INFO IntervalArgumentCollection - Processing 9803 bp from intervals
12:07:49.264 INFO HaplotypeCaller - Done initializing engine
12:07:49.266 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
12:07:49.276 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
12:07:49.276 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
12:07:49.291 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
12:07:49.292 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
12:07:49.348 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
12:07:49.348 INFO IntelPairHmm - Available threads: 16
12:07:49.348 INFO IntelPairHmm - Requested threads: 1
12:07:49.348 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
12:07:49.388 INFO ProgressMeter - Starting traversal
12:07:49.388 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
12:07:50.060 WARN InbreedingCoeff - InbreedingCoeff will not be calculated at position carnica.NW_020555920.1:388 and possibly subsequent; at least 10 samples must have called genotypes
12:08:06.699 INFO ProgressMeter - carnica.NW_020555920.1:2784 0.3 20 69.3
12:08:06.846 INFO HaplotypeCaller - 559 read(s) filtered by: MappingQualityReadFilter
0 read(s) filtered by: MappingQualityAvailableReadFilter
0 read(s) filtered by: MappedReadFilter
1 read(s) filtered by: NotSecondaryAlignmentReadFilter
1456 read(s) filtered by: NotDuplicateReadFilter
0 read(s) filtered by: PassesVendorQualityCheckReadFilter
0 read(s) filtered by: NonZeroReferenceLengthAlignmentReadFilter
0 read(s) filtered by: GoodCigarReadFilter
0 read(s) filtered by: WellformedReadFilter
2016 total reads filtered
12:08:06.847 INFO ProgressMeter - carnica.NW_020555920.1:9336 0.3 51 175.3
12:08:06.847 INFO ProgressMeter - Traversal complete. Processed 51 total regions in 0.3 minutes.
12:08:06.852 INFO VectorLoglessPairHMM - Time spent in setup for JNI call : 0.005628103000000001
12:08:06.852 INFO PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 13.988080472
12:08:06.852 INFO SmithWatermanAligner - Total compute time in java Smith-Waterman : 1.07 sec
12:08:06.852 INFO HaplotypeCaller - Shutting down engine
[13 April 2021 12:08:06 BST] org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller done. Elapsed time: 0.30 minutes.
Runtime.totalMemory()=207604940And I counted at least 18 processes for the script whilst it was running.
-
Hi Dave Wragg,
When you are seeing the 18 processes, how many are GATK and how many are Java?
Thanks,
Genevieve
-
Thanks for getting back quickly. I'll share the screenshots of using --pair-hmm-native-threads 1 and 4 so you can maybe determine what is happening.
-
I've tried with the below java options and they both spawn multiple processes as in the above examples. Those particular screenshot examples are based on +UseSerialGG which as I understand it should force 1 thread for GC.
-XX:ConcGCThreads=1 -XX:ParallelGCThreads=1
-XX:ConcGCThreads=1 -XX:+UseSerialGC
-
Could you clarify if you are seeing multiple processes or threads? Multiple processes would be, for example, multiple GATK commands, whereas multiple threads would be within a single process.
If your script is spawning multiple GATK processes, then there would have to be a different solution than the java options.
Let me know then we can trouble shoot from there.
Genevieve
-
Based on the above screenshots, there is a single GATK python process that spawns multiple java processes. I've re-run the tool to check thread usage with the following options:
gatk --java-options "-XX:ConcGCThreads=1 -XX:+UseSerialGC -Xmx28g" HaplotypeCaller --native-pair-hmm-threads 1
The number of threads increased by 21 (786 to 806), while the number of running threads increased from of 8-9 to upto 16.
-
Thanks for the info, Dave Wragg.
Can you try running it with just these four extra arguments?
- --use-jdk-deflater
- --use-jdk-inflater
- --pair-hmm-implementation LOGLESS_CACHING
- --java-options '-XX:ParallelGCThreads=1'
Let me know how that goes.
Genevieve
-
I've re-run with those options. The number of threads increased from 737 to 763, while the number of running threads increased from 1 to upto 10. I'd save on average, while the script was running, the number of running threads was 2-3, with periodic spikes. There are still lots of java processes spawned (see below).
-
Here is what one of the developers said about one GATK process and multiple Java processes:
The Python GATK frontend script (ie, the "gatk" command) launches a single GATK Java process. It's not possible for it to launch multiple GATK processes, unless it's being invoked multiple times.
Could you run the ./gatk HaplotypeCaller command directly from the terminal (without running the bash script HaplotypeCaller_pipeline.sh)?
You can examine the number of GATK processes with this command:
ps -ef | grep -i gatk
There should be only one process if you run the single gatk command. Then, you can get the number of threads with this command:
ps -o thcount <process_id>
Let me know what you find,
Genevieve
-
I've tried the following:
gatk --java-options "-XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g" HaplotypeCaller \
-R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta \
-I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam \
-O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz \
--ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803Which returned:
ps -ef | grep -i gatk
PID=$(ps -ef | grep -i gatk | awk '{print $2;}' -)
ps -o thcount ${PID}
dwragg 75000 58992 0 10:10 pts/22 00:00:00 python /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk --java-options -XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803
dwragg 75001 75000 99 10:10 pts/22 00:00:20 java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g -jar /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803
dwragg 75033 68544 0 10:10 pts/24 00:00:00 grep --color=auto -i gatk
[dwragg@node2i15(eddie) workflow]$ PID=$(ps -ef | grep -i gatk | awk '{print $2;}' -)
[dwragg@node2i15(eddie) workflow]$ ps -o thcount ${PID}
THCNT
1
27And I've tried:
gatk --java-options "-XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g" HaplotypeCaller \
-R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta \
-I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam \
-O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz \
--ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803 \
--use-jdk-inflater --use-jdk-deflater --pair-hmm-implementation LOGLESS_CACHINGWhich also returned:
ps -ef | grep -i gatk
PID=$(ps -ef | grep -i gatk | awk '{print $2;}' -)
ps -o thcount ${PID}
dwragg 75105 58992 0 10:11 pts/22 00:00:00 python /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk --java-options -XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803 --use-jdk-inflater --use-jdk-deflater --pair-hmm-implementation LOGLESS_CACHING
dwragg 75106 75105 99 10:11 pts/22 00:00:40 java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -XX:ConcGCThreads=1 -XX:ParallelGCThreads=1 -Xmx28g -jar /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803 --use-jdk-inflater --use-jdk-deflater --pair-hmm-implementation LOGLESS_CACHING
dwragg 75195 68544 0 10:11 pts/24 00:00:00 grep --color=auto -i gatk
[dwragg@node2i15(eddie) workflow]$ PID=$(ps -ef | grep -i gatk | awk '{print $2;}' -)
[dwragg@node2i15(eddie) workflow]$ ps -o thcount ${PID}
THCNT
1
27Which I interpret as GATK running 1 thread, which then spawns 27 Java threads?
-
Yes, it looks like there is 1 GATK process and 27 Java threads. Can you run your command with just these options to control the threads?
- --use-jdk-deflater
- --use-jdk-inflater
- --pair-hmm-implementation LOGLESS_CACHING
- --java-options '-XX:ParallelGCThreads=1'
Best,
Genevieve
-
[dwragg@node2i15(eddie) workflow]$ ps -ef | grep -i gatk
| awk '{print $2;}' -)
ps -o thcount ${PID}
dwragg 78940 78455 0 16:36 pts/13 00:00:00 python /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk --java-options -XX:ParallelGCThreads=1 -Xmx28g HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803 --use-jdk-inflater --use-jdk-deflater --pair-hmm-implementation LOGLESS_CACHING
dwragg 78941 78940 9 16:36 pts/13 00:00:02 java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -XX:ParallelGCThreads=1 -Xmx28g -jar /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/software/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar HaplotypeCaller -R /exports/cmvm/eddie/eb/groups/Wragg_Hong_MSc/xianrui/ref/HAv3.1/HAv3.1.fasta -I ./DEDUP/unicolor_MAD13/unicolor_MAD13.dedup.bam -O ./VCALL/unicolor_MAD13/unicolor_MAD13.carnica.NW_020555920.1.1.9803.raw.g.vcf.gz --ERC GVCF --native-pair-hmm-threads 1 --sample-ploidy 2 -A AlleleFraction -L carnica.NW_020555920.1:1-9803 --use-jdk-inflater --use-jdk-deflater --pair-hmm-implementation LOGLESS_CACHING
dwragg 79035 78641 0 16:37 pts/14 00:00:00 grep --color=auto -i gatk
[dwragg@node2i15(eddie) workflow]$ PID=$(ps -ef | grep -i gatk | awk '{print $2;}' -)
[dwragg@node2i15(eddie) workflow]$ ps -o thcount ${PID}
THCNT
1
27 -
Thank you for trying that. Could you share what you are seeing from the jstack of the GATK java process?
- Run gatk HaplotypeCaller directly from the terminal
- Get the PID of the GATK java process with ps -ef | grep -i gatk
- Send the full output of the command jstack <pid> (Where the <pid> is the process ID of the GATK java process)
Thank you!
-
Full thread dump Java HotSpot(TM) 64-Bit Server VM (10+46 mixed mode):
Threads class SMR info:
_java_thread_list=0x00007f7228002830, length=20, elements={
0x00007f72b8011000, 0x00007f72b841a800, 0x00007f72b841c800, 0x00007f72b842f000,
0x00007f72b8430800, 0x00007f72b8432800, 0x00007f72b8434800, 0x00007f72b8436800,
0x00007f72b8439000, 0x00007f72b843b000, 0x00007f72b8447000, 0x00007f72b8449000,
0x00007f72b844b000, 0x00007f72b844d800, 0x00007f72b844f800, 0x00007f72b8451800,
0x00007f72b8453800, 0x00007f72b8504800, 0x00007f72b850f000, 0x00007f7228001000
}
"main" #1 prio=5 os_prio=0 tid=0x00007f72b8011000 nid=0x1a09c runnable [0x00007f72c05c0000]
java.lang.Thread.State: RUNNABLE
at org.broadinstitute.hellbender.utils.pairhmm.LoglessPairHMM.subComputeReadLikelihoodGivenHaplotypeLog10(LoglessPairHMM.java:49)
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeReadLikelihoodGivenHaplotypeLog10(PairHMM.java:313)
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeLog10Likelihoods(PairHMM.java:245)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculationEngine.java:339)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculationEngine.java:188)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:654)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.apply(HaplotypeCaller.java:273)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:200)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:173)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1058)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f72b841a800 nid=0x1a0a3 waiting on condition [0x00007f729ca38000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@10/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@10/Reference.java:174)
at java.lang.ref.Reference.access$000(java.base@10/Reference.java:44)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@10/Reference.java:138)
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f72b841c800 nid=0x1a0a4 in Object.wait() [0x00007f729c937000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@10/Native Method)
- waiting on <0x00000000c0086a38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10/ReferenceQueue.java:151)
- waiting to re-lock in wait() <0x00000000c0086a38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10/ReferenceQueue.java:172)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@10/Finalizer.java:216)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f72b842f000 nid=0x1a0a5 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f72b8430800 nid=0x1a0a6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f72b8432800 nid=0x1a0a7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f72b8434800 nid=0x1a0a8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f72b8436800 nid=0x1a0a9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f72b8439000 nid=0x1a0aa waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f72b843b000 nid=0x1a0ab waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f72b8447000 nid=0x1a0ac waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f72b8449000 nid=0x1a0ad waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f72b844b000 nid=0x1a0ae waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f72b844d800 nid=0x1a0af waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f72b844f800 nid=0x1a0b0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f72b8451800 nid=0x1a0b1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #17 daemon prio=9 os_prio=0 tid=0x00007f72b8453800 nid=0x1a0b2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #18 daemon prio=9 os_prio=0 tid=0x00007f72b8504800 nid=0x1a0b3 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #19 daemon prio=8 os_prio=0 tid=0x00007f72b850f000 nid=0x1a0b4 in Object.wait() [0x00007f72550f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@10/Native Method)
- waiting on <0x00000000c0086b60> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@10/ReferenceQueue.java:151)
- waiting to re-lock in wait() <0x00000000c0086b60> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@10/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@10/Thread.java:844)
at jdk.internal.misc.InnocuousThread.run(java.base@10/InnocuousThread.java:134)
"Attach Listener" #22 daemon prio=9 os_prio=0 tid=0x00007f7228001000 nid=0x1a11c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" os_prio=0 tid=0x00007f72b8410800 nid=0x1a0a2 runnable
"GC Thread#0" os_prio=0 tid=0x00007f72b8029000 nid=0x1a09d runnable
"G1 Main Marker" os_prio=0 tid=0x00007f72b8075000 nid=0x1a09e runnable
"G1 Conc#0" os_prio=0 tid=0x00007f72b8077000 nid=0x1a09f runnable
"G1 Refine#0" os_prio=0 tid=0x00007f72b8381000 nid=0x1a0a0 runnable
"G1 Young RemSet Sampling" os_prio=0 tid=0x00007f72b8383000 nid=0x1a0a1 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f72b8513000 nid=0x1a0b5 waiting on condition
JNI global references: 14 -
Hi Dave Wragg,
Thanks for posting that! We took a look at the jstack output and here's a note from my colleague:
There is one GATK thread only (the one at the top called "main"). The other threads are all a normal built-in part of Java, not GATK: the "Reference Handler" and "Finalizer" threads, a bunch of JIT (just-in-time compiler) threads, a signal dispatch thread, etc. These Java threads are not actually taking up any CPU -- they are just waiting idly doing nothing.
Here is a description of those threads: https://stackoverflow.com/questions/55666827/what-are-threads-i-e-lightweight-processes-named-java-created-for/55669949#55669949
You shouldn't need to worry about these threads on your cluster, the actual CPU usage for this whole process is about 1 core, and can sometimes rise to 2 cores when the one garbage collector thread has something to do. These idle Java threads either cannot be disabled, or should not be disabled.
If you are wanting to know more about the CPU usage while GATK is running to make sure it stays between 1-2 cores, you can run "top" while GATK is running to see the core usage.
Hope this helps explain why you are seeing multiple threads, but unfortunately we can't do much to change this since GATK uses Java.
Have a good weekend,
Genevieve
-
Thanks Genevieve
That being the case, are all or any of the following options required to ensure the CPU usage stays within the requested threads, or were they simply required for debugging the issue? I ask because there is a performance difference between running with and without these flags.
- --use-jdk-deflater
- --use-jdk-inflater
- --pair-hmm-implementation LOGLESS_CACHING
Best regards
David
-
Hi David,
Our general recommendation to reduce the threads is with the argument -XX:ConcGCThreads=1. (More information in this article)
You can try using only that argument and check the thread usage with the jstack <pid>. If there are too many threads using just that argument, then you can use --use-jdk-deflater, --use-jdk-inflater, and --pair-hmm-implementation LOGLESS_CACHING.
Genevieve
-
Thank for you both. I'm now troubled with the threads, too. Currently for some special purpose, I embeded the GATK HaplotypeCaller and sambamba in a bash script which is called in a python script. The problem arose when I was conducting the multiprocessing: GATK HaplotypeCaller occupied so many threads that other software were not able to create new thread.
I've used both "-XX:ConcGCThreads=1" and "--use-jdk-deflater, --use-jdk-inflater, and --pair-hmm-implementation LOGLESS_CACHING" paramaters for limiting the threads to use. However, both of the tests showd "THCNT: 31".
Here is my command for gatk HaplotypeCaller:
os.system("gatk --java-options '-Xmx2G -XX:ParallelGCThreads=1 -XX:ConcGCThreads=1 ' HaplotypeCaller %s -R %s -O %s --bam-output %s --stand-call-conf 30 -L %s --tmp-dir %s --native-pair-hmm-threads 1 --linked-de-bruijn-graph --use-jdk-deflater --use-jdk-inflater --pair-hmm-implementation LOGLESS_CACHING" % (a, 'temp/20211226_part1_0.fasta', 'test.raw.vcf', 'test.HC.bam', 'MH_sc_0/20211226_part1_0_0.bed', 'MH_sc_0'))
-
Hi Lei Xia,
Can you do the same as I recommended above, where David ran only one HaplotypeCaller process at once on the command line, then checked the output with these commands? This will help to verify if there are any issues with the threads of HaplotypeCaller.
- Run gatk HaplotypeCaller directly from the terminal
- Get the PID of the GATK java process with ps -ef | grep -i gatk
- Send the full output of the command jstack <pid> (Where the <pid> is the process ID of the GATK java process)
Best,
Genevieve
-
2022-01-05 09:04:40
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode):"Attach Listener" #30 daemon prio=9 os_prio=0 tid=0x00007ff488001000 nid=0x64cdc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"SAMFileWriterThread-0" #29 daemon prio=5 os_prio=0 tid=0x00007ff4b859f800 nid=0x643b8 waiting on condition [0x00007ff49c1ce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fa319258> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at htsjdk.samtools.util.AbstractAsyncWriter$WriterRunnable.run(AbstractAsyncWriter.java:122)
at java.lang.Thread.run(Thread.java:745)"Service Thread" #26 daemon prio=9 os_prio=0 tid=0x00007ff4b815b000 nid=0x64125 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread20" #25 daemon prio=9 os_prio=0 tid=0x00007ff4b814e000 nid=0x64124 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread19" #24 daemon prio=9 os_prio=0 tid=0x00007ff4b814b800 nid=0x64123 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread18" #23 daemon prio=9 os_prio=0 tid=0x00007ff4b8149800 nid=0x64122 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread17" #22 daemon prio=9 os_prio=0 tid=0x00007ff4b8147800 nid=0x64121 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread16" #21 daemon prio=9 os_prio=0 tid=0x00007ff4b8145800 nid=0x64120 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread15" #20 daemon prio=9 os_prio=0 tid=0x00007ff4b8143800 nid=0x6411f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007ff4b8141000 nid=0x6411e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007ff4b813f000 nid=0x6411d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007ff4b813d000 nid=0x6411c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007ff4b813a800 nid=0x6411b waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007ff4b8138800 nid=0x6411a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007ff4b8136000 nid=0x64119 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007ff4b8134000 nid=0x64118 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007ff4b8131800 nid=0x64117 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007ff4b812f800 nid=0x64116 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007ff4b812d800 nid=0x64115 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007ff4b8123800 nid=0x64114 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007ff4b8121000 nid=0x64113 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007ff4b811f000 nid=0x64112 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ff4b811d000 nid=0x64111 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ff4b811a000 nid=0x64110 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ff4b8118800 nid=0x6410e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ff4b80e6800 nid=0x6410d in Object.wait() [0x00007ff49faf9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000801d88f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000801d88f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ff4b80e1800 nid=0x6410c in Object.wait() [0x00007ff49fbfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)- waiting on <0x00000000801d8938> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000801d8938> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)"main" #1 prio=5 os_prio=0 tid=0x00007ff4b800b000 nid=0x64109 runnable [0x00007ff4bfa86000]
java.lang.Thread.State: RUNNABLE
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeReadLikelihoodGivenHaplotypeLog10(PairHMM.java:314)
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeLog10Likelihoods(PairHMM.java:246)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculation
Engine.java:252)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculation
Engine.java:183)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:594)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.apply(HaplotypeCaller.java:240)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:299)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:272)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:984)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)
"VM Thread" os_prio=0 tid=0x00007ff4b80da000 nid=0x6410b runnable"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007ff4b8020000 nid=0x6410a runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007ff4b815e800 nid=0x64126 waiting on condition
JNI global references: 1072
-
The above output is generated when I ran gatk HaplotypeCaller from the python script. I reran gatk HaplotypeCaller directly from the terminal with the same command used in the python script, and the full jstack output is as follows:
2022-01-05 09:13:23
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode):"Attach Listener" #30 daemon prio=9 os_prio=0 tid=0x00007f5da0001000 nid=0x6b777 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"SAMFileWriterThread-0" #29 daemon prio=5 os_prio=0 tid=0x00007f5dc66b4000 nid=0x6883a waiting on condition [0x00007f5d413d4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000eed99258> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at htsjdk.samtools.util.AbstractAsyncWriter$WriterRunnable.run(AbstractAsyncWriter.java:122)
at java.lang.Thread.run(Thread.java:745)"Service Thread" #26 daemon prio=9 os_prio=0 tid=0x00007f5dc4153800 nid=0x682f4 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread20" #25 daemon prio=9 os_prio=0 tid=0x00007f5dc414e800 nid=0x682f3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread19" #24 daemon prio=9 os_prio=0 tid=0x00007f5dc414c000 nid=0x682f2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread18" #23 daemon prio=9 os_prio=0 tid=0x00007f5dc414a000 nid=0x682f1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread17" #22 daemon prio=9 os_prio=0 tid=0x00007f5dc4148000 nid=0x682f0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread16" #21 daemon prio=9 os_prio=0 tid=0x00007f5dc4145800 nid=0x682ef waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread15" #20 daemon prio=9 os_prio=0 tid=0x00007f5dc4144000 nid=0x682ee waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f5dc4141800 nid=0x682ed waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f5dc413f800 nid=0x682ec waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f5dc413d000 nid=0x682eb waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f5dc413b000 nid=0x682ea waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f5dc4138800 nid=0x682e9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f5dc4136800 nid=0x682e8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f5dc4134000 nid=0x682e7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f5dc4132000 nid=0x682e6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f5dc4130000 nid=0x682e5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f5dc412d800 nid=0x682e4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f5dc4123800 nid=0x682e3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f5dc4121000 nid=0x682e2 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f5dc411f000 nid=0x682e1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f5dc411d000 nid=0x682e0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f5dc411a000 nid=0x682df waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f5dc4118800 nid=0x682de runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f5dc40e6800 nid=0x682dd in Object.wait() [0x00007f5dc94ec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000801d88f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000801d88f8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f5dc40e1800 nid=0x682dc in Object.wait() [0x00007f5dc95ed000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)- waiting on <0x00000000801d8938> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000801d8938> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)"main" #1 prio=5 os_prio=0 tid=0x00007f5dc400b000 nid=0x682d9 runnable [0x00007f5dce285000]
java.lang.Thread.State: RUNNABLE
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeReadLikelihoodGivenHaplotypeLog10(PairHMM.java:314)
at org.broadinstitute.hellbender.utils.pairhmm.PairHMM.computeLog10Likelihoods(PairHMM.java:246)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculation
Engine.java:252)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.PairHMMLikelihoodCalculationEngine.computeReadLikelihoods(PairHMMLikelihoodCalculation
Engine.java:183)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:594)
at org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller.apply(HaplotypeCaller.java:240)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:299)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:272)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:984)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
at org.broadinstitute.hellbender.Main.main(Main.java:291)
"VM Thread" os_prio=0 tid=0x00007f5dc40da000 nid=0x682db runnable"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f5dc4020000 nid=0x682da runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f5dc4157000 nid=0x682f5 waiting on condition
JNI global references: 1063
-
Thanks for sharing those details. With both your runs, it looks like there is only one main GATK thread (SAMFileWriterThread-0). The rest are Java threads not taking up any CPUs and these cannot and should not be disabled.
Could you give more details about the error you found when other tools were not able to run?
-
core.thread.ThreadError@core/thread.d(3233): Error creating thread
When I was using the multiprocesses with sambamba, it raised the error like that. Furthermore, there are only 4096 threads available for me in the server, and the manager of the server said that gatk occupied 3900+ with 50 multiprocesses. So I guess the reason why I got the error above seemed to be attributed to the "empty" threads gatk occupied.
-
Yes, this may just be a limit of how much you can run at once within your server limits. I'll look into this closer to see if there is anything you can do.
-
Thank you very much! Waiting for your reply.
-
Lei Xia I brought this issue up with my colleagues to get their insights. They noticed that the number of compiler threads was higher than normal. We think you can control this number with this java option: XX:CICompilerCount=4 to limit the compiler threads to 4. If this doesn't work, you could also limit the active processors with XX:ActiveProcessorCount=2 but we weren't as sure about this option.
-
Thank you very much! I may have a try.
-
Ok! Let me know how it goes.
Please sign in to leave a comment.
28 comments