Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

Error while running ModelSegments

Answered
Follow
Avatar
onurerdoganmd

Hi !

When i run ModelSegments command, I have the following error. While reading input file containing allelic counts in tumor sample, command line is frozen and command is interrupted. I use a personal laptop with 8 gb RAM.

Thanks in advance


a) GATK version used: v4.2.2.0
b) Exact command used: :~/Desktop/BAM/somatic/P3/cnv2$ gatk --java-options "-Xmx8g" ModelSegments --denoised-copy-ratios 3tDenoisedCR.tsv --allelic-counts P3_T_clean.allelicCounts.tsv --normal-allelic-counts P3_N_clean.allelicCounts.tsv --output ~/Desktop/BAM/somatic/P3/cnv2/sandbox/ --output-prefix P3_T_clean --tmp-dir /home/onur/Desktop/BAM
c) Entire error log: Using GATK jar /home/onur/gatk/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx8g -jar /home/onur/gatk/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar ModelSegments --denoised-copy-ratios 3tDenoisedCR.tsv --allelic-counts P3_T_clean.allelicCounts.tsv --normal-allelic-counts P3_N_clean.allelicCounts.tsv --output /home/onur/Desktop/BAM/somatic/P3/cnv2/sandbox/ --output-prefix P3_T_clean --tmp-dir /home/onur/Desktop/BAM/
23:13:31.338 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/onur/gatk/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 09, 2021 11:13:31 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
23:13:31.703 INFO ModelSegments - ------------------------------------------------------------
23:13:31.704 INFO ModelSegments - The Genome Analysis Toolkit (GATK) v4.2.2.0
23:13:31.704 INFO ModelSegments - For support and documentation go to https://software.broadinstitute.org/gatk/
23:13:31.705 INFO ModelSegments - Executing as onur@onur on Linux v5.11.0-40-generic amd64
23:13:31.705 INFO ModelSegments - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_152-release-1056-b12
23:13:31.705 INFO ModelSegments - Start Date/Time: November 9, 2021 11:13:31 PM EET
23:13:31.706 INFO ModelSegments - ------------------------------------------------------------
23:13:31.706 INFO ModelSegments - ------------------------------------------------------------
23:13:31.707 INFO ModelSegments - HTSJDK Version: 2.24.1
23:13:31.707 INFO ModelSegments - Picard Version: 2.25.4
23:13:31.707 INFO ModelSegments - Built for Spark Version: 2.4.5
23:13:31.707 INFO ModelSegments - HTSJDK Defaults.COMPRESSION_LEVEL : 2
23:13:31.707 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
23:13:31.707 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
23:13:31.707 INFO ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
23:13:31.708 INFO ModelSegments - Deflater: IntelDeflater
23:13:31.708 INFO ModelSegments - Inflater: IntelInflater
23:13:31.708 INFO ModelSegments - GCS max retries/reopens: 20
23:13:31.708 INFO ModelSegments - Requester pays: disabled
23:13:31.708 INFO ModelSegments - Initializing engine
23:13:31.708 INFO ModelSegments - Done initializing engine
23:13:31.709 INFO ModelSegments - Used memory (MB) after initializing engine: 21
23:13:31.726 INFO ModelSegments - Reading file (3tDenoisedCR.tsv)...
23:13:32.783 INFO ModelSegments - Reading file (P3_T_clean.allelicCounts.tsv)...


 

0

15 comments

Sort by Date Votes
  • Avatar
    Genevieve Brandt (she/her)

    Hi onurerdoganmd,

    Try specifying less than 8GB of memory with the Java options. There needs to be a small amount of extra memory available other than what Java is using.

    Let me know if that works!

    Best,

    Genevieve

    0
    Comment actions Permalink
  • Avatar
    onurerdoganmd

    Hi Genevieve,

    Thank you for your help. It solved my problem.

    Best,
    Onur

    0
    Comment actions Permalink
  • Avatar
    Genevieve Brandt (she/her)

    Great to hear! Thanks for the update, Onur. 

    0
    Comment actions Permalink
  • Avatar
    jianrong zheng

    Hi !

    When i run ModelSegments command, I have the following error. While reading input file containing allelic counts in tumor sample, command line is frozen.

    Thanks in advance

    15:27:43.015 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/public/home/zhengjianrong/software/gatk-4.2.4.1/gatk-package-4.2.4.1-local.jar!/com/intel/gkl/native/libgkl_compression.so

    Sep 14, 2022 3:27:43 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine

    INFO: Failed to detect whether we are running on Google Compute Engine.

    15:27:43.175 INFO  ModelSegments - ------------------------------------------------------------

    15:27:43.175 INFO  ModelSegments - The Genome Analysis Toolkit (GATK) v4.2.4.1

    15:27:43.175 INFO  ModelSegments - For support and documentation go to https://software.broadinstitute.org/gatk/

    15:27:43.176 INFO  ModelSegments - Executing as zhengjianrong@cluster on Linux v3.10.0-1160.el7.x86_64 amd64

    15:27:43.176 INFO  ModelSegments - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_332-b09

    15:27:43.176 INFO  ModelSegments - Start Date/Time: September 14, 2022 3:27:42 PM CST

    15:27:43.176 INFO  ModelSegments - ------------------------------------------------------------

    15:27:43.176 INFO  ModelSegments - ------------------------------------------------------------

    15:27:43.176 INFO  ModelSegments - HTSJDK Version: 2.24.1

    15:27:43.176 INFO  ModelSegments - Picard Version: 2.25.4

    15:27:43.176 INFO  ModelSegments - Built for Spark Version: 2.4.5

    15:27:43.177 INFO  ModelSegments - HTSJDK Defaults.COMPRESSION_LEVEL : 2

    15:27:43.177 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

    15:27:43.177 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

    15:27:43.177 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

    15:27:43.177 INFO  ModelSegments - Deflater: IntelDeflater

    15:27:43.177 INFO  ModelSegments - Inflater: IntelInflater

    15:27:43.177 INFO  ModelSegments - GCS max retries/reopens: 20

    15:27:43.177 INFO  ModelSegments - Requester pays: disabled

    15:27:43.177 INFO  ModelSegments - Initializing engine

    15:27:43.177 INFO  ModelSegments - Done initializing engine

    15:27:43.177 INFO  ModelSegments - Used memory (MB) after initializing engine: 226

    15:27:43.184 INFO  ModelSegments - Reading file (/public/home/zhengjianrong/project/GWAS/9.cnv/counts/SSC/denoisedCR/1MT000381WL1.clean.denoisedCR.tsv)...

    15:27:43.667 INFO  ModelSegments - Reading file (/public/home/zhengjianrong/project/GWAS/9.cnv/counts/SSC/allelicCounts/1MT000381WL1.allelicCounts.tsv)...

     

    0
    Comment actions Permalink
  • Avatar
    jianrong zheng

    I try specifying less than 4GB of memory with the Java options, but the. command line is still frozen

    0
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    Thank you for writing to the GATK forum! I hope we can help you sort this out.

    Could you please let me know how much physical memory your machine has available? Once I know this, we will be able to help you further. 

    I look forward to hearing back from you!
    Anthony

    0
    Comment actions Permalink
  • Avatar
    jianrong zheng 
    Hi, Anthony Dias-Ciarla
    Thank you for your help. The picture shows my machine physical memory.
    Best wishes,
    Jianrong

    0
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    Thank you for providing this information!

    Could you please specify how many GBs your machine has? The screenshot that you included does not have any units. 

    Have you tried to run ModelSegments with 8GB or just 4GB? The tool requires about 3GB of overhead memory to run successfully. 

    We recommend using an 8GB machine and running ModelSegments with 5GB as the --java-options to leave 3GB of overhead.

    I hope this provides some helpful context! I look forward to your response. If you have any other questions in the meantime, please do not hesitate to reach out.

    Best,
    Anthony

    1
    Comment actions Permalink
  • Avatar
    jianrong zheng 
    Hi, Anthony Dias-Ciarla
    Thank you for providing me this suggestion! Actually, how much memory do I use,it will occour the same problem.
    I still have not run successfully.
    Best,
    jianrong
    0
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    I'm sorry to hear that you are still having trouble! 

    Please go ahead and try the specifications I mentioned in my previous response. Try running with a machine with 8 gigabytes (GB) of memory. Then try allocating 5GB to run ModelSegments using --java-options. Please find an example below.

    gatk --java-options "-Xmx5G" [program arguments]

    Please let me know if this works! Please include the entire command and complete program log in your response if not. I look forward to hearing back from you.

    Best,
    Anthony

    0
    Comment actions Permalink
  • Avatar
    jianrong zheng 
    Hi, Anthony Dias-Ciarla
    Thank you for providing me this suggestion! I try again ues my MACos which with 16GB of memory,and allocating 5GB to run ModelSegments using --java-options.

    Running:

        java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx5G -jar /Users/jianrongzheng/Downloads/software/gatk-4.2.5.0/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar ModelSegments --denoised-copy-ratios ./M03137AZW1L1.clean.denoisedCR.tsv --allelic-counts ./M03137AZW1L1.allelicCounts.tsv --normal-allelic-counts ./M03137AZW1B1.allelicCounts.tsv --output ./ --output-prefix M03137AZW1L1

    11:16:39.301 INFO  NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/Users/jianrongzheng/Downloads/software/gatk-4.2.5.0/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib

    Sep 29, 2022 11:16:39 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine

    INFO: Failed to detect whether we are running on Google Compute Engine.

    11:16:39.865 INFO  ModelSegments - ------------------------------------------------------------

    11:16:39.866 INFO  ModelSegments - The Genome Analysis Toolkit (GATK) v4.2.5.0

    11:16:39.866 INFO  ModelSegments - For support and documentation go to https://software.broadinstitute.org/gatk/

    11:16:39.866 INFO  ModelSegments - Executing as jianrongzheng@Jianrongs-iMac.local on Mac OS X v12.2.1 x86_64

    11:16:39.866 INFO  ModelSegments - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_321-b07

    11:16:39.866 INFO  ModelSegments - Start Date/Time: September 29, 2022 11:16:39 AM CST

    11:16:39.866 INFO  ModelSegments - ------------------------------------------------------------

    11:16:39.866 INFO  ModelSegments - ------------------------------------------------------------

    11:16:39.868 INFO  ModelSegments - HTSJDK Version: 2.24.1

    11:16:39.868 INFO  ModelSegments - Picard Version: 2.25.4

    11:16:39.868 INFO  ModelSegments - Built for Spark Version: 2.4.5

    11:16:39.868 INFO  ModelSegments - HTSJDK Defaults.COMPRESSION_LEVEL : 2

    11:16:39.868 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

    11:16:39.869 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

    11:16:39.869 INFO  ModelSegments - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

    11:16:39.869 INFO  ModelSegments - Deflater: IntelDeflater

    11:16:39.869 INFO  ModelSegments - Inflater: IntelInflater

    11:16:39.869 INFO  ModelSegments - GCS max retries/reopens: 20

    11:16:39.869 INFO  ModelSegments - Requester pays: disabled

    11:16:39.869 INFO  ModelSegments - Initializing engine

    11:16:39.869 INFO  ModelSegments - Done initializing engine

    11:16:39.869 INFO  ModelSegments - Used memory (MB) after initializing engine: 162

    11:16:39.879 INFO  ModelSegments - Reading file (./M03137AZW1L1.clean.denoisedCR.tsv)...

    11:16:40.317 INFO  ModelSegments - Reading file (./M03137AZW1L1.allelicCounts.tsv)...

    11:28:23.443 INFO  ModelSegments - Shutting down engine

    [September 29, 2022 11:28:23 AM CST] org.broadinstitute.hellbender.tools.copynumber.ModelSegments done. Elapsed time: 11.74 minutes.

    Runtime.totalMemory()=4772593664

    Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

    at java.lang.StringBuilder.toString(StringBuilder.java:413)

    at com.opencsv.CSVParser.parseLine(CSVParser.java:383)

    at com.opencsv.CSVParser.parseLineMulti(CSVParser.java:299)

    at com.opencsv.CSVReader.readNext(CSVReader.java:275)

    at org.broadinstitute.hellbender.utils.tsv.TableReader.fetchNextRecord(TableReader.java:357)

    at org.broadinstitute.hellbender.utils.tsv.TableReader.access$200(TableReader.java:99)

    at org.broadinstitute.hellbender.utils.tsv.TableReader$1.hasNext(TableReader.java:472)

    at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)

    at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)

    at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:455)

    at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:847)

    at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:275)

    at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractRecordCollection.<init>(AbstractRecordCollection.java:91)

    at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractLocatableCollection.<init>(AbstractLocatableCollection.java:58)

    at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AbstractSampleLocatableCollection.<init>(AbstractSampleLocatableCollection.java:44)

    at org.broadinstitute.hellbender.tools.copynumber.formats.collections.AllelicCountCollection.<init>(AllelicCountCollection.java:58)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$ModelSegmentsData$$Lambda$94/278254138.apply(Unknown Source)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments.readOptionalFileOrNull(ModelSegments.java:781)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments.access$500(ModelSegments.java:394)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$ModelSegmentsData.lambda$new$3(ModelSegments.java:673)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$ModelSegmentsData$$Lambda$93/497104907.apply(Unknown Source)

    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)

    at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)

    at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)

    at java.util.stream.StreamSpliterators$WrappingSpliterator$$Lambda$56/1193577032.getAsBoolean(Unknown Source)

    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)

    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)

    at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)

    at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)

    at org.broadinstitute.hellbender.relocated.com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:268)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$ModelSegmentsData.<init>(ModelSegments.java:672)

    at org.broadinstitute.hellbender.tools.copynumber.ModelSegments$ModelSegmentsData.<init>(ModelSegments.java:638)

    Best,

    jianrong

    0
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    Thank you again for providing this information!

    It appears that you are now encountering an OutOfMemoryError. Our developers said that the default configuration we have for running ModelSegments is 13GB of machine memory and 10GB of java memory.

    We now recommend that you try increasing the machine memory to 25GB. Please continue to leave 3GB for overhead by allocating 22GB to run ModelSegments with --java-options.

    I hope this helps get you across the finish line! Please let me know if this proves to be successful. If not, please let me know so we can figure out our next steps.

    Best,
    Anthony

    1
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    We haven't heard from you in a while so we're going to close out this ticket. If you still require assistance, simply respond to this email and we'll be happy to pick up where we left off!

    Kind regards,

    Anthony​

    1
    Comment actions Permalink
  • Avatar
    jianrong zheng 
    Hi, Anthony Dias-Ciarla，
    Thank you for your attention for my question. I Running ModelSegments is 45GB, increasing the machine free memory to 500GB,then I secceed.
    0
    Comment actions Permalink
  • Avatar
    Anthony DiCi

    Hi jianrong zheng,

    Thank you for following up with news of your success! I am happy to hear that we were able to sort this out together.

    We wish you continued success and luck. Thank you for being a valued contributor to the GATK community. Please do not hesitate to reach out with any questions/issues that may arise in the future.

    Best,
    Anthony

    0
    Comment actions Permalink

Please sign in to leave a comment.

Welcome

Forum Guidelines

GATK Announcements

Become a GATK Guru

Read Before You Post

Get Email Notifications

Didn't find what you were looking for?

New post

Quick Links

GATK Latest Releases

Known Issues & Troubleshooting

Submitting a Bug Report

How to Track a Github issue

GATK on the Cloud with Terra

Recent GATK News

GATK Blog Posts

GATK Twitter Page

About the GATK community

  • 133,119 Total Users
  • 0 Comments
  • 0 Posts
  • 0 Votes
Powered by Zendesk