VariantEval IndexOutOfBoundsException
Answereda) GATK version used: v4.1.9
b) Exact command used:
/gatk-4.1.9.0/gatk-package-4.1.9.0-local.jar \
VariantEval \
-R /$PATH_TO_REFERENCE/chm13/t2t-chm13.20200921.withGRCh38chrY.chrEBV.chrYKI270740v1r.fasta \
--eval /$PATH_TO_VCF/1kgp.chrX.recalibrated.snp_indel.pass.vcf.gz \
--pedigree /$PATH_TO_PED/1kgp_trios.ped \
-no-ev -no-st -ST Family \
-EV MendelianViolationEvaluator \
-O 1kgp.chrX.recalibrated.snp_indel.pass.MVs.byFamily.table
c) Entire error log:
19:35:29.408 INFO VariantEval - ------------------------------------------------------------ 19:35:29.408 INFO VariantEval - The Genome Analysis Toolkit (GATK) v4.1.9.0 19:35:29.409 INFO VariantEval - For support and documentation go to https://software.broadinstitute.org/gatk/ 19:35:29.409 INFO VariantEval - Executing as root@0b79b5044551 on Linux v5.4.104+ amd64 19:35:29.409 INFO VariantEval - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_152-release-1056-b12 19:35:29.409 INFO VariantEval - Start Date/Time: May 27, 2021 7:35:29 PM UTC 19:35:29.409 INFO VariantEval - ------------------------------------------------------------ 19:35:29.409 INFO VariantEval - ------------------------------------------------------------ 19:35:29.410 INFO VariantEval - HTSJDK Version: 2.23.0 19:35:29.410 INFO VariantEval - Picard Version: 2.23.3 19:35:29.410 INFO VariantEval - HTSJDK Defaults.COMPRESSION_LEVEL : 2 19:35:29.410 INFO VariantEval - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 19:35:29.410 INFO VariantEval - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 19:35:29.410 INFO VariantEval - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 19:35:29.410 INFO VariantEval - Deflater: IntelDeflater 19:35:29.410 INFO VariantEval - Inflater: IntelInflater 19:35:29.410 INFO VariantEval - GCS max retries/reopens: 20 19:35:29.410 INFO VariantEval - Requester pays: disabled 19:35:29.411 WARN VariantEval - [1m[31m !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Warning: VariantEval is a BETA tool and is not yet ready for use in production !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![0m 19:35:29.411 INFO VariantEval - Initializing engine 19:35:29.755 INFO FeatureManager - Using codec VCFCodec to read file file:///cromwell_root/fc-47de7dae-e8e6-429c-b760-b4ba49136eee/joint_vcfs/recalibrated/PASS/1kgp.chrX.recalibrated.snp_indel.pass.vcf.gz 19:35:29.835 INFO VariantEval - Done initializing engine 19:35:29.836 INFO PedReader - Reading PED file /cromwell_root/fc-47de7dae-e8e6-429c-b760-b4ba49136eee/resources/1kgp/1kgp_trios.ped with missing fields: [] 19:35:29.854 INFO PedReader - Phenotype is other? true 19:35:32.686 INFO VariantEval - Creating 1881 combinatorial stratification states 19:35:32.742 INFO ProgressMeter - Starting traversal 19:35:32.742 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute 19:36:01.819 INFO VariantEval - Shutting down engine [May 27, 2021 7:36:01 PM UTC] org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval done. Elapsed time: 0.54 minutes. Runtime.totalMemory()=4964483072 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.rangeCheck(ArrayList.java:653) at java.util.ArrayList.get(ArrayList.java:429) at org.broadinstitute.hellbender.utils.samples.MendelianViolation.isViolation(MendelianViolation.java:180) at org.broadinstitute.hellbender.utils.samples.MendelianViolation.updateViolations(MendelianViolation.java:122) at org.broadinstitute.hellbender.utils.samples.MendelianViolation.countFamilyViolations(MendelianViolation.java:148) at org.broadinstitute.hellbender.tools.walkers.varianteval.evaluators.MendelianViolationEvaluator.update1(MendelianViolationEvaluator.java:122) at org.broadinstitute.hellbender.tools.walkers.varianteval.util.EvaluationContext.apply(EvaluationContext.java:74) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval.processComp(VariantEval.java:596) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval.doApply(VariantEval.java:562) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval$PositionAggregator.callDoApply(VariantEval.java:497) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval$PositionAggregator.addVariant(VariantEval.java:478) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval$PositionAggregator.access$100(VariantEval.java:469) at org.broadinstitute.hellbender.tools.walkers.varianteval.VariantEval.apply(VariantEval.java:511) at org.broadinstitute.hellbender.engine.MultiVariantWalker.lambda$traverse$1(MultiVariantWalker.java:120) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.broadinstitute.hellbender.engine.MultiVariantWalker.traverse(MultiVariantWalker.java:118) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1049) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203) at org.broadinstitute.hellbender.Main.main(Main.java:289) 2021/05/27 19:36:03 Starting delocalization. 2021/05/27 19:36:04 Delocalization script execution started... 2021/05/27 19:36:04 Delocalizing output /cromwell_root/memory_retry_rc -> gs://fc-51aefb1c-4e8e-4dcb-a59c-62e318ea351a/b4b25c64-84dc-4902-ae53-89bff2091e92/mendelian_analysis/fd25ef2c-8ed6-4b2a-9df9-b7287511aee8/call-mendelian_analysis/memory_retry_rc 2021/05/27 19:36:04 Delocalizing output /cromwell_root/rc -> gs://fc-51aefb1c-4e8e-4dcb-a59c-62e318ea351a/b4b25c64-84dc-4902-ae53-89bff2091e92/mendelian_analysis/fd25ef2c-8ed6-4b2a-9df9-b7287511aee8/call-mendelian_analysis/rc 2021/05/27 19:36:05 Delocalizing output /cromwell_root/stdout -> gs://fc-51aefb1c-4e8e-4dcb-a59c-62e318ea351a/b4b25c64-84dc-4902-ae53-89bff2091e92/mendelian_analysis/fd25ef2c-8ed6-4b2a-9df9-b7287511aee8/call-mendelian_analysis/stdout 2021/05/27 19:36:06 Delocalizing output /cromwell_root/stderr -> gs://fc-51aefb1c-4e8e-4dcb-a59c-62e318ea351a/b4b25c64-84dc-4902-ae53-89bff2091e92/mendelian_analysis/fd25ef2c-8ed6-4b2a-9df9-b7287511aee8/call-mendelian_analysis/stderr 2021/05/27 19:36:08 Delocalizing output /cromwell_root/1kgp.chrX.recalibrated.snp_indel.pass.MVs.byFamily.table -> gs://fc-51aefb1c-4e8e-4dcb-a59c-62e318ea351a/b4b25c64-84dc-4902-ae53-89bff2091e92/mendelian_analysis/fd25ef2c-8ed6-4b2a-9df9-b7287511aee8/call-mendelian_analysis/1kgp.chrX.recalibrated.snp_indel.pass.MVs.byFamily.table Required file output '/cromwell_root/1kgp.chrX.recalibrated.snp_indel.pass.MVs.byFamily.table' does not exist.
I am running VariantEval to detect Mendelian violations in large joint genotyped VCF files, so I'm running it on a per-chromosome basis. This error only occurs for the chromosome X file, and it only occurs with this FASTA file (GRCh38 on chrX does not cause this issue). IndexFeatureFile is run just before this error, which has also led to successful runs in other chromosomes, so that's not the issue. Any insight on this issue would be appreciated.
-
Hi Samantha Zarate, I just wanted to let you know we are looking into this issue. We don't have any updates as of now but we'll get back to you as soon as possible.
-
Hi Samantha Zarate,
We are thinking this is related to the data being on the X chromosome and the tool not being able to access the Y chromosome data. VariantEval has underlying assumptions about the data being diploid so the sex chromosomes cause issues with that.
Could you try running with the X and Y together and see if that helps?
Best,
Genevieve
-
Hi Genevieve Brandt (she/her), thanks for the insight! How would you recommend combining X and Y population-scale VCF files, given that the samples are not the same between the two (not every sample with an X has a Y)?
-
Hi Samantha Zarate, we are looking into solutions. Regarding this note from you:
"This error only occurs for the chromosome X file, and it only occurs with this FASTA file (GRCh38 on chrX does not cause this issue)."
It looks like you are using a chromosome Y fasta file, but your VCF is named chr X. Are you trying to evaluate variants on chromosme Y using a combined X and Y file with Y as the reference? If not, could you provide clarifications for what you are running?
Thank you,
Genevieve
-
Hi Genevieve Brandt (she/her), let me clarify what I mean. I'm comparing VCFs generated using two reference genomes, GRCh38 and CHM13. CHM13, as an organism, doesn't have a Y chromosome, so it's been taken from GRCh38 (hence the `withGRCh38chrY` part of the FASTA name). However, I have confirmed that this issue does in fact occur with the VCFs aligned to GRCh38, so I think your comment about ploidy is correct.
-
Thanks for the clarification. Unfortunately I don't think there is a workaround possible at this point to make VariantEval usable on the X chromosome. I have created a ticket on github so that the developers can add this functionality in the future: https://github.com/broadinstitute/gatk/issues/7304. You can follow along there as our team works on it!
Thanks for writing in and bringing up this issue! I'm sorry we were not able to solve this issue for you at this time.
Please sign in to leave a comment.
6 comments