"main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
AnsweredHi, GATK team! I'm working on GATK WGS somatic CNV calling pipeline.
When I tried gatk --java-options "-Xmx2800g" ModelSegments --denoised-copy-ratios ${tumor}.denoisedCR.tsv --allelic-counts ${tumor}.allelicCounts.tsv --normal-allelic-counts ${normal}.allelicCounts.tsv --output-prefix ${tumor} -O ${outdir}, I got this type of error:
10:00:18.408 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/lustre/home/acct-medliuyb/medliuyb-user1/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Jan 10, 2022 10:00:18 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
10:00:18.544 INFO ModelSegments - ------------------------------------------------------------
10:00:18.544 INFO ModelSegments - The Genome Analysis Toolkit (GATK) v4.2.0.0
10:00:18.544 INFO ModelSegments - For support and documentation go to https://software.broadinstitute.org/gatk/
10:00:18.544 INFO ModelSegments - Executing as medliuyb-user1@huge2.pi.sjtu.edu.cn on Linux v3.10.0-1062.el7.x86_64 amd64
10:00:18.545 INFO ModelSegments - Java runtime: OpenJDK 64-Bit Server VM v10.0.2+13
10:00:18.545 INFO ModelSegments - Start Date/Time: January 10, 2022 at 10:00:18 AM CST
10:00:18.545 INFO ModelSegments - ------------------------------------------------------------
10:00:18.545 INFO ModelSegments - ------------------------------------------------------------
10:00:18.545 INFO ModelSegments - HTSJDK Version: 2.24.0
10:00:18.545 INFO ModelSegments - Picard Version: 2.25.0
10:00:18.545 INFO ModelSegments - Built for Spark Version: 2.4.5
10:00:18.545 INFO ModelSegments - HTSJDK Defaults.COMPRESSION_LEVEL : 2
10:00:18.546 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
10:00:18.546 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
10:00:18.546 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
10:00:18.546 INFO ModelSegments - Deflater: IntelDeflater
10:00:18.546 INFO ModelSegments - Inflater: IntelInflater
10:00:18.546 INFO ModelSegments - GCS max retries/reopens: 20
10:00:18.546 INFO ModelSegments - Requester pays: disabled
10:00:18.546 INFO ModelSegments - Initializing engine
10:00:18.546 INFO ModelSegments - Done initializing engine
10:00:18.549 INFO ModelSegments - Reading file (/lustre/home/acct-medliuyb/medliuyb-user1/PP_JOB/HiC_PC/seq-data/GATK_practise/CNV/WGS_1_T.denoisedCR.tsv)...
10:00:22.289 INFO ModelSegments - Reading file (/lustre/home/acct-medliuyb/medliuyb-user1/PP_JOB/HiC_PC/seq-data/GATK_practise/CNV/CollectAllelicCounts/WGS_1_T.allelicCounts.tsv)...
10:19:45.750 INFO ModelSegments - Shutting down engine
[January 10, 2022 at 10:19:45 AM CST] org.broadinstitute.hellbender.tools.copynumber.ModelSegments done. Elapsed time: 19.46 minutes.
Runtime.totalMemory()=2628889083904
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.base/java.util.Arrays.copyOf(Arrays.java:3688)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList$Builder.getReadyToExpandTo(ImmutableList.java:768)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:787)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:748)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:456)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:847)
at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:275)
at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractRecordCollection.(AbstractRecordCollection.java:83)
at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractLocatableCollection.(AbstractLocatableCollection.java:58)
at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractSampleLocatableCollection.(AbstractSampleLocatableCollection.java:44)
at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AllelicCountCollection.(AllelicCountCollection.java:58)
at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$$Lambda$168/595759572.apply(Unknown Source)
at org.broadinstitute.hellbender.tools.copynumber.ModelSegments.readOptionalFileOrNull(ModelSegments.java:599)
at org.broadinstitute.hellbender.tools.copynumber.ModelSegments.doWork(ModelSegments.java:481)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
My allelicCounts.tsv is about 69 G. I know it's array size problem, java can not load allelicCounts file that big. I wonder if you know some how to solve this error.
my interval_list is generated by the command as follow:
gatk PreprocessIntervals
-R Homo_sapiens_assembly38.fasta
--bin-length 1000
--padding 0
-O preprocessed_intervals.interval_list
am I using correctly?
thanks!
-
Hello,
Do you have 2800G memory available on your machine? It looks like your error might be coming from GATK trying to use 2800G but it is not available.
Best,
Genevieve
-
Hi,
I figured out that it was caused by wrong interval_list input. it should be snp-only.interval_list, but I didn't find any online file.
I'm working on generating it by myself. Do you know where I can find it?
thanks!
PENG PU
-
Hi,@Genevieve Brandt
Problem solved!
First, I slected snp-only.vcf from gnomad.genomes.r2.1.sites.liftoverToHg38.INFO_ANNOTATIONS_FIXED.vcf.gz using SelectVariants, and got interval_list by IntervalListTools.
Just in case some friends do have same issue.
Best,
PENG PU
-
Thank you for posting your solution PENG PU! I'm glad you have solved your problem.
Please sign in to leave a comment.
4 comments