Mutect2 - No .stats file created with SLURM
Using GATK 4.4.0.0 Mutect2 the program does not generate .stats file when running via SBATCH.
I've utilized the exact same command running on the headnode vs. running via SBATCH or SRUN. The .stats file is only created on the headnode. I've also replicated this on 4.3.0.0
/tools/gatk-4.4.0.0/gatk Mutect2 -R /references/homo_sapiens/hg38/genome/GATK/Homo_sapiens_assembly38.fasta -I 1.recal.cram -I 2.recal.cram -I 3.recal.cram -I 4.recal.cram -I 5.recal.cram -I 6.recal.cram -normal mock_1 -normal mock_2 -normal mock_3 -O output.vcf
Error message:
Looks like there is an open issue and the original poster was running via sbatch
Error message:
09:35:19.035 INFO ProgressMeter - chrUn_JTFH01000206v1_decoy:901 3655.2 12948780 3542.6
09:36:49.135 INFO Mutect2 - Shutting down engine
[June 16, 2023 at 9:36:49 AM UTC] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 3,656.69 minutes.
Runtime.totalMemory()=4081057792
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
Using GATK jar /tools/gatk-4.3.0.0/gatk-package-4.3.0.0-local.jar
Running:
java-Dsamjdk.use_async_io_read_samtools=false-Dsamjdk.use_async_io_write_samtools=true-Dsamjdk.use_async_io_write_tribble=false-Dsamjdk.compression_level=2-jar/tools/gatk-4.3.0.0/gatk-package-4.3.0.0-local.jarFilterMutectCalls-Voutput.vcf-Ooutput_filtered.vcf-R/references/homo_sapiens/hg38/genome/GATK/Homo_sapiens_assembly38.fasta
09:37:01.708 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file://tools/gatk-4.3.0.0/gatk-package-4.3.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
09:37:02.731 INFO FilterMutectCalls - ------------------------------------------------------------
09:37:02.731 INFO FilterMutectCalls - The Genome Analysis Toolkit (GATK) v4.3.0.0
09:37:02.731 INFO FilterMutectCalls - For support and documentation go to https://software.broadinstitute.org/gatk/
09:37:02.731 INFO FilterMutectCalls - Executing as ryan@cpu-c6i-dy-c6i-2xlarge-2 on Linux v5.10.173-154.642.amzn2.x86_64 amd64
09:37:02.731 INFO FilterMutectCalls - Java runtime: Java HotSpot(TM) 64-Bit Server VM v17.0.2+8-LTS-86
09:37:02.732 INFO FilterMutectCalls - Start Date/Time: June 16, 2023 at 9:37:01 AM UTC
09:37:02.732 INFO FilterMutectCalls - ------------------------------------------------------------
09:37:02.732 INFO FilterMutectCalls - ------------------------------------------------------------
09:37:02.732 INFO FilterMutectCalls - HTSJDK Version: 3.0.1
09:37:02.732 INFO FilterMutectCalls - Picard Version: 2.27.5
09:37:02.732 INFO FilterMutectCalls - Built for Spark Version: 2.4.5
09:37:02.732 INFO FilterMutectCalls - HTSJDK Defaults.COMPRESSION_LEVEL : 2
09:37:02.732 INFO FilterMutectCalls - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
09:37:02.732 INFO FilterMutectCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
09:37:02.733 INFO FilterMutectCalls - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
09:37:02.733 INFO FilterMutectCalls - Deflater: IntelDeflater
09:37:02.733 INFO FilterMutectCalls - Inflater: IntelInflater
09:37:02.733 INFO FilterMutectCalls - GCS max retries/reopens: 20
09:37:02.733 INFO FilterMutectCalls - Requester pays: disabled
09:37:02.733 INFO FilterMutectCalls - Initializing engine
09:37:02.985 INFO FeatureManager - Using codec VCFCodec to read file file:///crams/output.vcf
09:37:03.116 INFO FilterMutectCalls - Done initializing engine
09:37:03.282 INFO FilterMutectCalls - Shutting down engine
[June 16, 2023 at 9:37:03 AM UTC] org.broadinstitute.hellbender.tools.walkers.mutect.filtering.FilterMutectCalls done. Elapsed time: 0.04 minutes.
Runtime.totalMemory()=104857600
***********************************************************************
A USER ERROR has occurred: Mutect stats table output.vcf.stats not found. When Mutect2 outputs a file calls.vcf it also creates a calls.vcf.stats file. Perhaps this file was not moved along with the vcf, or perhaps it was not delocalized from a virtual machine while running in the cloud.
***********************************************************************
Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--java-options '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.
Looks like there is an open issue and the original poster was running via sbatch
https://github.com/broadinstitute/gatk/issues/6271
Is there any way around this?
-
It may be related to the temporary folder used by the slurm process being wiped out before the java VM is closed. Can you try moving the temporary folder for Mutect2 to somewhere you have read write access such as ./tmp ?
-
Yes this worked! Added the --tmp-dir argument to achieve. Thanks.
Please sign in to leave a comment.
2 comments