Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

PathSeq error: java.lang.NumberFormatException

Follow
Avatar
Roshan Kumar

Packahe used: PathSeq

Gatk verison: 4.5.0

Command Used: 

gatk --java-options "-Xmx180G" PathSeqPipelineSpark --input sorted_unaligned_S5000_3D.bam --filter-bwa-image /hpc/refdata/gatk/Broad/BWAIndex/version_0.7.13/Homo_sapiens_assembly38.fa.img --kmer-file /hpc/refdata/gatk/Broad/BWAIndex/version_0.7.13/Homo_sapiens_assembly38.hss --min-clipped-read-length 70 --microbe-bwa-image /hpc/refdata/pathseq/microbe/pathseq_host.fa.img --microbe-dict /hpc/refdata/pathseq/microbe/pathseq_microbe.dict --taxonomy-file /hpc/refdata/pathseq/microbe/pathseq_microbe.db --output unaligned_S5000_3D.pathseq.bam --scores-output unaligned_S5000_3D.pathseq.txt

 

Error: 

11:58:54.165 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
11:58:54.165 INFO  TaskSetManager - Finished task 77.0 in stage 20.0 (TID 949) in 15 ms on cn02 (executor driver) (85/103)
11:58:54.165 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
11:58:54.165 INFO  Executor - Finished task 71.0 in stage 20.0 (TID 943). 1657 bytes result sent to driver
11:58:54.165 INFO  TaskSetManager - Finished task 71.0 in stage 20.0 (TID 943) in 16 ms on cn02 (executor driver) (86/103)
11:58:54.165 INFO  Executor - Finished task 72.0 in stage 20.0 (TID 944). 1657 bytes result sent to driver
11:58:54.165 INFO  TaskSetManager - Finished task 72.0 in stage 20.0 (TID 944) in 16 ms on cn02 (executor driver) (87/103)
11:58:54.165 INFO  Executor - Finished task 65.0 in stage 20.0 (TID 937). 1657 bytes result sent to driver
11:58:54.166 INFO  TaskSetManager - Finished task 65.0 in stage 20.0 (TID 937) in 19 ms on cn02 (executor driver) (88/103)
11:58:54.166 INFO  Executor - Finished task 84.0 in stage 20.0 (TID 956). 1657 bytes result sent to driver
11:58:54.166 INFO  Executor - Finished task 83.0 in stage 20.0 (TID 955). 1657 bytes result sent to driver
11:58:54.166 INFO  TaskSetManager - Finished task 84.0 in stage 20.0 (TID 956) in 14 ms on cn02 (executor driver) (89/103)
11:58:54.166 INFO  TaskSetManager - Finished task 83.0 in stage 20.0 (TID 955) in 15 ms on cn02 (executor driver) (90/103)
11:58:54.166 INFO  Executor - Finished task 68.0 in stage 20.0 (TID 940). 1657 bytes result sent to driver
11:58:54.166 INFO  TaskSetManager - Finished task 68.0 in stage 20.0 (TID 940) in 18 ms on cn02 (executor driver) (91/103)
11:58:54.166 INFO  Executor - Finished task 74.0 in stage 20.0 (TID 946). 1657 bytes result sent to driver
11:58:54.166 INFO  Executor - Finished task 94.0 in stage 20.0 (TID 966). 1657 bytes result sent to driver
11:58:54.166 INFO  TaskSetManager - Finished task 74.0 in stage 20.0 (TID 946) in 17 ms on cn02 (executor driver) (92/103)
11:58:54.167 INFO  Executor - Finished task 93.0 in stage 20.0 (TID 965). 1657 bytes result sent to driver
11:58:54.167 INFO  TaskSetManager - Finished task 94.0 in stage 20.0 (TID 966) in 5 ms on cn02 (executor driver) (93/103)
11:58:54.167 INFO  TaskSetManager - Finished task 93.0 in stage 20.0 (TID 965) in 6 ms on cn02 (executor driver) (94/103)
11:58:54.167 INFO  Executor - Finished task 96.0 in stage 20.0 (TID 968). 1657 bytes result sent to driver
11:58:54.167 INFO  Executor - Finished task 97.0 in stage 20.0 (TID 969). 1657 bytes result sent to driver
11:58:54.167 INFO  TaskSetManager - Finished task 96.0 in stage 20.0 (TID 968) in 5 ms on cn02 (executor driver) (95/103)
11:58:54.168 INFO  Executor - Finished task 99.0 in stage 20.0 (TID 971). 1657 bytes result sent to driver
11:58:54.168 INFO  TaskSetManager - Finished task 97.0 in stage 20.0 (TID 969) in 5 ms on cn02 (executor driver) (96/103)
11:58:54.168 INFO  Executor - Finished task 100.0 in stage 20.0 (TID 972). 1657 bytes result sent to driver
11:58:54.168 INFO  TaskSetManager - Finished task 99.0 in stage 20.0 (TID 971) in 5 ms on cn02 (executor driver) (97/103)
11:58:54.168 INFO  Executor - Finished task 101.0 in stage 20.0 (TID 973). 1657 bytes result sent to driver
11:58:54.168 INFO  TaskSetManager - Finished task 100.0 in stage 20.0 (TID 972) in 5 ms on cn02 (executor driver) (98/103)
11:58:54.168 INFO  TaskSetManager - Finished task 101.0 in stage 20.0 (TID 973) in 5 ms on cn02 (executor driver) (99/103)
11:58:54.168 INFO  Executor - Finished task 98.0 in stage 20.0 (TID 970). 1657 bytes result sent to driver
11:58:54.168 INFO  Executor - Finished task 95.0 in stage 20.0 (TID 967). 1657 bytes result sent to driver
11:58:54.168 INFO  TaskSetManager - Finished task 98.0 in stage 20.0 (TID 970) in 5 ms on cn02 (executor driver) (100/103)
11:58:54.169 INFO  TaskSetManager - Finished task 95.0 in stage 20.0 (TID 967) in 7 ms on cn02 (executor driver) (101/103)
11:58:54.169 INFO  Executor - Finished task 102.0 in stage 20.0 (TID 974). 1657 bytes result sent to driver
11:58:54.169 INFO  TaskSetManager - Finished task 102.0 in stage 20.0 (TID 974) in 5 ms on cn02 (executor driver) (102/103)
11:58:54.199 INFO  TaskSetManager - Finished task 91.0 in stage 20.0 (TID 963) in 41 ms on cn02 (executor driver) (103/103)
11:58:54.199 INFO  TaskSchedulerImpl - Removed TaskSet 20.0, whose tasks have all completed, from pool 
11:58:54.200 INFO  DAGScheduler - ShuffleMapStage 20 (repartition at PathSeqPipelineSpark.java:197) finished in 0.132 s
11:58:54.200 INFO  DAGScheduler - looking for newly runnable stages
11:58:54.200 INFO  DAGScheduler - running: Set()
11:58:54.200 INFO  DAGScheduler - waiting: Set(ShuffleMapStage 21, ResultStage 22)
11:58:54.200 INFO  DAGScheduler - failed: Set()
11:58:54.201 INFO  DAGScheduler - Submitting ShuffleMapStage 21 (MapPartitionsRDD[69] at mapPartitionsToPair at PSScorer.java:68), which has no missing parents
11:58:54.209 INFO  MemoryStore - Block broadcast_15 stored as values in memory (estimated size 14.5 KiB, free 105.1 GiB)
11:58:54.210 INFO  MemoryStore - Block broadcast_15_piece0 stored as bytes in memory (estimated size 7.4 KiB, free 105.1 GiB)
11:58:54.211 INFO  BlockManagerInfo - Added broadcast_15_piece0 in memory on cn02:33616 (size: 7.4 KiB, free: 107.7 GiB)
11:58:54.211 INFO  SparkContext - Created broadcast 15 from broadcast at DAGScheduler.scala:1580
11:58:54.211 INFO  DAGScheduler - Submitting 3 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[69] at mapPartitionsToPair at PSScorer.java:68) (first 15 tasks are for partitions Vector(0, 1, 2))
11:58:54.211 INFO  TaskSchedulerImpl - Adding task set 21.0 with 3 tasks resource profile 0
11:58:54.215 INFO  TaskSetManager - Starting task 0.0 in stage 21.0 (TID 975) (cn02, executor driver, partition 0, NODE_LOCAL, 7807 bytes) 
11:58:54.216 INFO  TaskSetManager - Starting task 1.0 in stage 21.0 (TID 976) (cn02, executor driver, partition 1, NODE_LOCAL, 7807 bytes) 
11:58:54.216 INFO  TaskSetManager - Starting task 2.0 in stage 21.0 (TID 977) (cn02, executor driver, partition 2, NODE_LOCAL, 7807 bytes) 
11:58:54.216 INFO  Executor - Running task 0.0 in stage 21.0 (TID 975)
11:58:54.216 INFO  Executor - Running task 1.0 in stage 21.0 (TID 976)
11:58:54.216 INFO  Executor - Running task 2.0 in stage 21.0 (TID 977)
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Getting 103 (344.4 KiB) non-empty blocks including 103 (344.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Getting 103 (345.3 KiB) non-empty blocks including 103 (345.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Getting 103 (453.8 KiB) non-empty blocks including 103 (453.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
11:58:54.450 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] skip orientation FR as there are not enough pairs
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] skip orientation FR as there are not enough pairs
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
11:59:03.797 INFO  MemoryStore - Block rdd_61_1 stored as values in memory (estimated size 976.6 KiB, free 105.1 GiB)
11:59:03.798 INFO  BlockManagerInfo - Added rdd_61_1 in memory on cn02:33616 (size: 976.6 KiB, free: 107.7 GiB)
11:59:03.818 INFO  MemoryStore - Block rdd_61_0 stored as values in memory (estimated size 1021.0 KiB, free 105.1 GiB)
11:59:03.813 ERROR Executor - Exception in task 1.0 in stage 21.0 (TID 976)
java.lang.NumberFormatException: For input string: "150M"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:668) ~[?:?]
at java.lang.Integer.valueOf(Integer.java:999) ~[?:?]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.Task.run(Task.scala:141) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) [gatk-package-4.5.0.0-local.jar:4.5.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
11:59:03.819 INFO  BlockManagerInfo - Added rdd_61_0 in memory on cn02:33616 (size: 1021.0 KiB, free: 107.7 GiB)
11:59:03.825 ERROR Executor - Exception in task 0.0 in stage 21.0 (TID 975)
java.lang.NumberFormatException: For input string: "150M"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:668) ~[?:?]
at java.lang.Integer.valueOf(Integer.java:999) ~[?:?]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.scheduler.Task.run(Task.scala:141) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) ~[gatk-package-4.5.0.0-local.jar:4.5.0.0]
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) [gatk-package-4.5.0.0-local.jar:4.5.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
11:59:03.841 WARN  TaskSetManager - Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
11:59:03.843 ERROR TaskSetManager - Task 0 in stage 21.0 failed 1 times; aborting job
11:59:03.845 INFO  TaskSetManager - Lost task 1.0 in stage 21.0 (TID 976) on cn02, executor driver: java.lang.NumberFormatException (For input string: "150M") [duplicate 1]
11:59:03.847 INFO  TaskSchedulerImpl - Cancelling stage 21
11:59:03.847 INFO  TaskSchedulerImpl - Killing all running tasks in stage 21: Stage cancelled: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
Driver stacktrace:
11:59:03.853 INFO  Executor - Executor is trying to kill task 2.0 in stage 21.0 (TID 977), reason: Stage cancelled: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
Driver stacktrace:
11:59:03.853 INFO  TaskSchedulerImpl - Stage 21 was cancelled
11:59:03.854 INFO  DAGScheduler - ShuffleMapStage 21 (mapPartitionsToPair at PSScorer.java:68) failed in 9.650 s due to Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
Driver stacktrace:
11:59:03.856 INFO  DAGScheduler - Job 4 failed: collectAsMap at PSScorer.java:71, took 9.820911 s
11:59:03.857 INFO  SparkContext - SparkContext is stopping with exitCode 0.
11:59:03.869 INFO  AbstractConnector - Stopped Spark@147be32c{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
11:59:03.875 INFO  SparkUI - Stopped Spark web UI at http://cn02:4040
11:59:03.892 INFO  MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
11:59:04.039 INFO  MemoryStore - Block rdd_62_0 stored as values in memory (estimated size 1705.0 KiB, free 105.1 GiB)
11:59:04.039 INFO  BlockManagerInfo - Added rdd_62_0 in memory on cn02:33616 (size: 1705.0 KiB, free: 107.7 GiB)
11:59:04.043 INFO  Executor - Executor killed task 2.0 in stage 21.0 (TID 977), reason: Stage cancelled: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
Driver stacktrace:
11:59:04.151 INFO  MemoryStore - MemoryStore cleared
11:59:04.152 INFO  BlockManager - BlockManager stopped
11:59:04.153 INFO  BlockManagerMaster - BlockManagerMaster stopped
11:59:04.156 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
11:59:04.182 INFO  SparkContext - Successfully stopped SparkContext
11:59:04.182 INFO  PathSeqPipelineSpark - Shutting down engine
[April 2, 2024 at 11:59:04 AM CDT] org.broadinstitute.hellbender.tools.spark.pathseq.PathSeqPipelineSpark done. Elapsed time: 2.92 minutes.
Runtime.totalMemory()=55331258368
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 21.0 failed 1 times, most recent failure: Lost task 0.0 in stage 21.0 (TID 975) (cn02 executor driver): java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
 
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2844)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2780)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2779)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2779)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1242)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1242)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1242)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3048)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2971)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:984)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2398)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2419)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2438)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2463)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1046)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:407)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1045)
at org.apache.spark.rdd.PairRDDFunctions.$anonfun$collectAsMap$1(PairRDDFunctions.scala:738)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:407)
at org.apache.spark.rdd.PairRDDFunctions.collectAsMap(PairRDDFunctions.scala:737)
at org.apache.spark.api.java.JavaPairRDD.collectAsMap(JavaPairRDD.scala:663)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.scoreReads(PSScorer.java:71)
at org.broadinstitute.hellbender.tools.spark.pathseq.PathSeqPipelineSpark.runTool(PathSeqPipelineSpark.java:271)
at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.runPipeline(GATKSparkTool.java:535)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:166)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:209)
at org.broadinstitute.hellbender.Main.main(Main.java:306)
Caused by: java.lang.NumberFormatException: For input string: "150M"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.valueOf(Integer.java:999)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidAlternateHits(PSScorer.java:268)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.getValidHits(PSScorer.java:234)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$1(PSScorer.java:178)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$mapGroupedReadsToTax$2f669b29$1(PSScorer.java:186)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.computeTaxScores(PSScorer.java:290)
at org.broadinstitute.hellbender.tools.spark.pathseq.PSScorer.lambda$scoreReads$3d020b2d$1(PSScorer.java:68)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsToPair$1(JavaRDDLike.scala:186)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
11:59:04.186 INFO  ShutdownHookManager - Shutdown hook called
11:59:04.187 INFO  ShutdownHookManager - Deleting directory /tmp/spark-0ba8bd5a-7c74-46e0-94f0-d504b405eada

 

0

8 comments

Sort by Date Votes
  • Avatar
    Gökalp Çelik

    Hi Roshan Kumar

    What are your locale settings within your compute environment? Are you using anything other than en_US locale?

    0
    Comment actions Permalink
  • Avatar
    Roshan Kumar

    We have the following settings on compute nodes: 

    $llocale

    LANG=en_US.UTF-8

    0
    Comment actions Permalink
  • Avatar
    Gökalp Çelik

    Hi again. 

    Looks like you might be hitting an edge case where aligner is emitting a CIGAR string instead of a number. Can you provide us a sample data that we can reproduce it as well?

    There is an article for how to upload sample files for us to check. 

    https://gatk.broadinstitute.org/hc/en-us/articles/360035889671-How-do-I-submit-a-detailed-bug-report 

    If you are able to just let us know from this topic and we will look into this issue in deep.

    Regards. 

    0
    Comment actions Permalink
  • Avatar
    Roshan Kumar

    I have uploaded a zip file with data and log file. Like the original version of PathSeq I was expecting, input as fastq files. But, in gatk-pathseq teh input file is bam. Therefore I generated this unaligned (want to keep both host and microbe reads) bam file. Any direct approach to generate the bam file?

    0
    Comment actions Permalink
  • Avatar
    Gökalp Çelik

    Thank you Roshan Kumar we will certainly take a look at shortly. 

    Can you tell us the name of the file you uploaded? 

    0
    Comment actions Permalink
  • Avatar
    Roshan Kumar

    Pathseq-error.zip

    0
    Comment actions Permalink
  • Avatar
    Roshan Kumar

    I am eagerly waiting for your response.

    0
    Comment actions Permalink
  • Avatar
    Gökalp Çelik
    • Edited

    Hi

    I noticed that your bwa img file and microbe dictionary files are differently named. Are you sure that the dictionary file belongs to the pathseq_host.fa.img file?

    Secondly, when I checked the code throwing error it looks like one of your reads is containing a malformed XA tag therefore instead of the mismatch bases it gets the CIGAR string in the tag. Can you try running the below command?

    gatk PathSeqBwaSpark  \
   --paired-input sorted_unaligned_S5000_3D.bam \
   --paired-output output_reads_paired.bam \
   --microbe-bwa-image /hpc/refdata/pathseq/microbe/pathseq_host.fa.img \
   --microbe-dict /hpc/refdata/pathseq/microbe/pathseq_microbe.dict

    This command will produce the bam file that PathSeqPipelineSpark uses to analyze. 

    Then the main quest will be checking XA tags of your reads. 

    We want you to check your bam output using 

    gatk FilterSamReads

    tool with the script code below

    function accept(rec) {
        if(rec.hasAttribute("XA"))
        {
            var xastring = rec.getStringAttribute("XA");
            var xaarray = xastring.split(";");
            for(var i=0; i<xaarray.length; i++)
            {
                var subxa = xaarray[i];
                var subxaarray = subxa.split(",");
                if(subxaarray.length>4)
                    print(rec.getSAMString());
            }
        }
        return false;
    }

    accept(record);

    Save this script as filter.js and run the below command.

    gatk FilterSamReads -JS filter.js -I output_reads_paired.bam -O filtered.bam --FILTER includeJavascript

    This code will start checking all XA tags within reads and will throw the SAM string of reads with violating XA tags. Then we want you to post some of those reads here so that we can find the actual cause of it. Possibly a fasta name error somewhere in the reference or dictionary. 

    Can you help us with this one so that we can solve this issue faster?

    Regards. 

    0
    Comment actions Permalink

Please sign in to leave a comment.

Welcome

Forum Guidelines

GATK Announcements

Become a GATK Guru

Read Before You Post

Get Email Notifications

Didn't find what you were looking for?

New post

Quick Links

GATK Latest Releases

Known Issues & Troubleshooting

Submitting a Bug Report

How to Track a Github issue

GATK on the Cloud with Terra

Recent GATK News

GATK Blog Posts

GATK Twitter Page

About the GATK community

  • 133,119 Total Users
  • 0 Comments
  • 0 Posts
  • 0 Votes
Powered by Zendesk