Error using FindBreakpointEvidenceSpark
AnsweredIf you are seeing an error, please provide(REQUIRED) :
a) GATK version used: v.4.2.2
b) Exact command used:
/app/Genome/gatk-4.2.2.0/gatk FindBreakpointEvidenceSpark \
-I Data/GM24385_2_S15.bam \ --aligner-index-image reference.fasta.img \ --kmers-to-ignore kmers_to_ignore.txt \
-L ../../recalset/nextera_dna_exome_targeted_regions_manifest_v1_2.bed \
-O assemblies2.sam
c) Entire error log:
Using GATK jar /app/Genome/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /app/Genome/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar FindBreakpointEvidenceSpark -I Data/GM24385_2_S15.bam --aligner-index-image reference.fasta.img --kmers-to-ignore kmers_to_ignore.txt -L ../../recalset/nextera_dna_exome_targeted_regions_manifest_v1_2.bed -O assemblies2.sam
10:07:00.597 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/app/Genome/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
10:07:00.626 WARN NativeLibraryLoader - Unable to load libgkl_compression.so from native/libgkl_compression.so (/tmp/libgkl_compression5151701974239540886.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/libgkl_compression5151701974239540886.so))
10:07:00.633 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/app/Genome/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
10:07:00.639 WARN NativeLibraryLoader - Unable to load libgkl_compression.so from native/libgkl_compression.so (/tmp/libgkl_compression1705808440983612689.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/libgkl_compression1705808440983612689.so))
Dec 06, 2021 10:07:00 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
10:07:00.789 INFO FindBreakpointEvidenceSpark - ------------------------------------------------------------
10:07:00.789 INFO FindBreakpointEvidenceSpark - The Genome Analysis Toolkit (GATK) v4.2.2.0
10:07:00.789 INFO FindBreakpointEvidenceSpark - For support and documentation go to https://software.broadinstitute.org/gatk/
10:07:00.789 INFO FindBreakpointEvidenceSpark - Executing as habusamra@klogin3 on Linux v2.6.32-358.23.2.el6.x86_64 amd64
10:07:00.789 INFO FindBreakpointEvidenceSpark - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_66-b17
10:07:00.790 INFO FindBreakpointEvidenceSpark - Start Date/Time: December 6, 2021 10:07:00 AM AST
10:07:00.790 INFO FindBreakpointEvidenceSpark - ------------------------------------------------------------
10:07:00.790 INFO FindBreakpointEvidenceSpark - ------------------------------------------------------------
10:07:00.790 INFO FindBreakpointEvidenceSpark - HTSJDK Version: 2.24.1
10:07:00.790 INFO FindBreakpointEvidenceSpark - Picard Version: 2.25.4
10:07:00.791 INFO FindBreakpointEvidenceSpark - Built for Spark Version: 2.4.5
10:07:00.791 INFO FindBreakpointEvidenceSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2
10:07:00.791 INFO FindBreakpointEvidenceSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
10:07:00.791 INFO FindBreakpointEvidenceSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
10:07:00.791 INFO FindBreakpointEvidenceSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
10:07:00.791 INFO FindBreakpointEvidenceSpark - Deflater: JdkDeflater
10:07:00.791 INFO FindBreakpointEvidenceSpark - Inflater: JdkInflater
10:07:00.791 INFO FindBreakpointEvidenceSpark - GCS max retries/reopens: 20
10:07:00.791 INFO FindBreakpointEvidenceSpark - Requester pays: disabled
10:07:00.791 WARN FindBreakpointEvidenceSpark -
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Warning: FindBreakpointEvidenceSpark is a BETA tool and is not yet ready for use in production
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10:07:00.792 INFO FindBreakpointEvidenceSpark - Initializing engine
10:07:00.792 INFO FindBreakpointEvidenceSpark - Done initializing engine
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/12/06 10:07:01 INFO SparkContext: Running Spark version 2.4.5
21/12/06 10:07:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/12/06 10:07:01 INFO SparkContext: Submitted application: FindBreakpointEvidenceSpark
21/12/06 10:07:01 INFO SecurityManager: Changing view acls to: habusamra
21/12/06 10:07:01 INFO SecurityManager: Changing modify acls to: habusamra
21/12/06 10:07:01 INFO SecurityManager: Changing view acls groups to:
21/12/06 10:07:01 INFO SecurityManager: Changing modify acls groups to:
21/12/06 10:07:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(habusamra); groups with view permissions: Set(); users with modify permissions: Set(habusamra); groups with modify permissions: Set()
21/12/06 10:07:01 INFO Utils: Successfully started service 'sparkDriver' on port 47790.
21/12/06 10:07:01 INFO SparkEnv: Registering MapOutputTracker
21/12/06 10:07:01 INFO SparkEnv: Registering BlockManagerMaster
21/12/06 10:07:01 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/12/06 10:07:01 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/12/06 10:07:01 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-ea817732-fab4-4ef5-a0cc-79afc54b232b
21/12/06 10:07:01 INFO MemoryStore: MemoryStore started with capacity 12.4 GB
21/12/06 10:07:01 INFO SparkEnv: Registering OutputCommitCoordinator
21/12/06 10:07:02 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/12/06 10:07:02 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://klogin3.local:4040
21/12/06 10:07:02 INFO Executor: Starting executor ID driver on host localhost
21/12/06 10:07:02 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33845.
21/12/06 10:07:02 INFO NettyBlockTransferService: Server created on klogin3.local:33845
21/12/06 10:07:02 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/12/06 10:07:02 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:07:02 INFO BlockManagerMasterEndpoint: Registering block manager klogin3.local:33845 with 12.4 GB RAM, BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:07:02 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:07:02 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, klogin3.local, 33845, None)
10:07:02.471 INFO FindBreakpointEvidenceSpark - Spark verbosity set to INFO (see --spark-verbosity argument)
21/12/06 10:07:02 INFO GoogleHadoopFileSystemBase: GHFS version: 1.9.4-hadoop3
10:07:02.639 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
21/12/06 10:07:02 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 307.9 KB, free 12.4 GB)
21/12/06 10:07:03 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 35.4 KB, free 12.4 GB)
21/12/06 10:07:03 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on klogin3.local:33845 (size: 35.4 KB, free: 12.4 GB)
21/12/06 10:07:03 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at PathSplitSource.java:96
10:07:04.156 INFO FeatureManager - Using codec BEDCodec to read file file:///fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/../../recalset/nextera_dna_exome_targeted_regions_manifest_v1_2.bed
21/12/06 10:07:04 INFO BlockManagerInfo: Removed broadcast_0_piece0 on klogin3.local:33845 in memory (size: 35.4 KB, free: 12.4 GB)
10:07:05.031 INFO IntervalArgumentCollection - Processing 45326818 bp from intervals
10:07:05.091 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 5.8 MB, free 12.4 GB)
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.1 MB, free 12.4 GB)
21/12/06 10:07:05 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on klogin3.local:33845 (size: 2.1 MB, free: 12.4 GB)
21/12/06 10:07:05 INFO SparkContext: Created broadcast 1 from broadcast at AbstractBinarySamSource.java:82
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 307.9 KB, free 12.4 GB)
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 35.4 KB, free 12.4 GB)
21/12/06 10:07:05 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on klogin3.local:33845 (size: 35.4 KB, free: 12.4 GB)
21/12/06 10:07:05 INFO SparkContext: Created broadcast 2 from newAPIHadoopFile at PathSplitSource.java:96
21/12/06 10:07:05 INFO FileInputFormat: Total input files to process : 1
21/12/06 10:07:05 INFO SparkContext: Starting job: collect at ReadMetadata.java:81
21/12/06 10:07:05 INFO DAGScheduler: Got job 0 (collect at ReadMetadata.java:81) with 484 output partitions
21/12/06 10:07:05 INFO DAGScheduler: Final stage: ResultStage 0 (collect at ReadMetadata.java:81)
21/12/06 10:07:05 INFO DAGScheduler: Parents of final stage: List()
21/12/06 10:07:05 INFO DAGScheduler: Missing parents: List()
21/12/06 10:07:05 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[12] at mapPartitions at ReadMetadata.java:78), which has no missing parents
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 249.7 KB, free 12.4 GB)
21/12/06 10:07:05 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 110.2 KB, free 12.4 GB)
21/12/06 10:07:05 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on klogin3.local:33845 (size: 110.2 KB, free: 12.4 GB)
21/12/06 10:07:05 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1163
21/12/06 10:07:05 INFO DAGScheduler: Submitting 484 missing tasks from ResultStage 0 (MapPartitionsRDD[12] at mapPartitions at ReadMetadata.java:78) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
21/12/06 10:07:05 INFO TaskSchedulerImpl: Adding task set 0.0 with 484 tasks
21/12/06 10:07:05 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, localhost, executor driver, partition 4, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, localhost, executor driver, partition 5, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, localhost, executor driver, partition 6, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, localhost, executor driver, partition 7, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, localhost, executor driver, partition 8, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, localhost, executor driver, partition 9, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, localhost, executor driver, partition 10, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, localhost, executor driver, partition 11, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, localhost, executor driver, partition 12, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, localhost, executor driver, partition 13, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, localhost, executor driver, partition 14, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, localhost, executor driver, partition 15, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, localhost, executor driver, partition 16, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, localhost, executor driver, partition 17, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, localhost, executor driver, partition 18, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, localhost, executor driver, partition 19, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, localhost, executor driver, partition 20, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, localhost, executor driver, partition 21, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, localhost, executor driver, partition 22, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, localhost, executor driver, partition 23, PROCESS_LOCAL, 7976 bytes)
21/12/06 10:07:05 INFO Executor: Running task 10.0 in stage 0.0 (TID 10)
21/12/06 10:07:05 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
21/12/06 10:07:05 INFO Executor: Running task 14.0 in stage 0.0 (TID 14)
21/12/06 10:07:05 INFO Executor: Running task 11.0 in stage 0.0 (TID 11)
21/12/06 10:07:05 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
21/12/06 10:07:05 INFO Executor: Running task 13.0 in stage 0.0 (TID 13)
21/12/06 10:07:05 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
21/12/06 10:07:05 INFO Executor: Running task 12.0 in stage 0.0 (TID 12)
21/12/06 10:07:05 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
21/12/06 10:07:05 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/12/06 10:07:05 INFO Executor: Running task 15.0 in stage 0.0 (TID 15)
21/12/06 10:07:05 INFO Executor: Running task 20.0 in stage 0.0 (TID 20)
21/12/06 10:07:05 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
21/12/06 10:07:05 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
21/12/06 10:07:05 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
21/12/06 10:07:05 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
21/12/06 10:07:05 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
21/12/06 10:07:05 INFO Executor: Running task 23.0 in stage 0.0 (TID 23)
21/12/06 10:07:05 INFO Executor: Running task 22.0 in stage 0.0 (TID 22)
21/12/06 10:07:05 INFO Executor: Running task 21.0 in stage 0.0 (TID 21)
21/12/06 10:07:05 INFO Executor: Running task 19.0 in stage 0.0 (TID 19)
21/12/06 10:07:05 INFO Executor: Running task 18.0 in stage 0.0 (TID 18)
21/12/06 10:07:05 INFO Executor: Running task 16.0 in stage 0.0 (TID 16)
21/12/06 10:07:05 INFO Executor: Running task 17.0 in stage 0.0 (TID 17)
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:536870912+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:167772160+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:704643072+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:301989888+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:402653184+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:671088640+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:33554432+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:469762048+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:234881024+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:67108864+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:771751936+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:637534208+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:0+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:134217728+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:100663296+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:335544320+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:268435456+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:570425344+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:603979776+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:369098752+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:503316480+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:436207616+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:738197504+33554432
21/12/06 10:07:06 INFO NewHadoopRDD: Input split: file:/fefs1/BDC/Analysis/NGS_Analysis/habusamra/CNV/Data/GM24385_2_S15.bam:201326592+33554432
10:07:06.154 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.155 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.159 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.160 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.161 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.162 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.163 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.164 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.170 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.171 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.171 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.173 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.176 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.178 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.178 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.178 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.179 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.180 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.180 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.185 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.185 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.186 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.188 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.189 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.189 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.189 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.189 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.189 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.190 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.191 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.191 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.191 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.191 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.192 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.193 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.193 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.193 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.193 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.193 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.194 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.194 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.195 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.195 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.196 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.196 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.198 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.199 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.199 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.199 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.200 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.201 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.201 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.201 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.201 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.202 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.204 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.204 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.205 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.206 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.207 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.207 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.207 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.208 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.209 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.210 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.210 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.213 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.215 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.216 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.226 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.245 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
10:07:06.245 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
21/12/06 10:12:37 WARN Executor: Issue communicating with driver in heartbeater
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10000 milliseconds]. This timeout is controlled by spark.executor.heartbeatInterval
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:92)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:846)
at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply$mcV$sp(Executor.scala:875)
at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:875)
at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:875)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
at org.apache.spark.executor.Executor$$anon$2.run(Executor.scala:875)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
... 14 more
21/12/06 10:12:37 WARN HeartbeatReceiver: Removing executor driver with no recent heartbeats: 277788 ms exceeds timeout 120000 ms
21/12/06 10:12:37 WARN NettyRpcEnv: Ignored failure: java.util.concurrent.TimeoutException: Cannot receive any reply from klogin3.local:47790 in 10000 milliseconds
21/12/06 10:12:37 ERROR TaskSchedulerImpl: Lost executor driver on localhost: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 23.0 in stage 0.0 (TID 23, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 ERROR TaskSetManager: Task 23 in stage 0.0 failed 1 times; aborting job
21/12/06 10:12:37 WARN TaskSetManager: Lost task 17.0 in stage 0.0 (TID 17, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 8.0 in stage 0.0 (TID 8, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 11.0 in stage 0.0 (TID 11, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 20.0 in stage 0.0 (TID 20, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 5.0 in stage 0.0 (TID 5, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 14.0 in stage 0.0 (TID 14, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 13.0 in stage 0.0 (TID 13, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 4.0 in stage 0.0 (TID 4, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 22.0 in stage 0.0 (TID 22, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 16.0 in stage 0.0 (TID 16, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 7.0 in stage 0.0 (TID 7, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 10.0 in stage 0.0 (TID 10, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 19.0 in stage 0.0 (TID 19, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 18.0 in stage 0.0 (TID 18, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 9.0 in stage 0.0 (TID 9, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 21.0 in stage 0.0 (TID 21, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 3.0 in stage 0.0 (TID 3, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 12.0 in stage 0.0 (TID 12, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 15.0 in stage 0.0 (TID 15, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
21/12/06 10:12:37 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
21/12/06 10:12:37 INFO TaskSchedulerImpl: Cancelling stage 0
21/12/06 10:12:37 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
21/12/06 10:12:37 INFO DAGScheduler: ResultStage 0 (collect at ReadMetadata.java:81) failed in 332.181 s due to Job aborted due to stage failure: Task 23 in stage 0.0 failed 1 times, most recent failure: Lost task 23.0 in stage 0.0 (TID 23, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
Driver stacktrace:
21/12/06 10:12:37 INFO Executor: Told to re-register on heartbeat
21/12/06 10:12:37 WARN SparkContext: Killing executors is not supported by current scheduler.
21/12/06 10:12:37 INFO BlockManager: BlockManager BlockManagerId(driver, klogin3.local, 33845, None) re-registering with master
21/12/06 10:12:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManager: Reporting 6 blocks to the master.
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_1_piece0 in memory on klogin3.local:33845 (current size: 2.1 MB, original size: 2.1 MB, free: 12.4 GB)
21/12/06 10:12:37 INFO DAGScheduler: Executor lost: driver (epoch 0)
21/12/06 10:12:37 INFO BlockManagerMasterEndpoint: Trying to remove executor driver from BlockManagerMaster.
21/12/06 10:12:37 WARN BlockManagerMasterEndpoint: No more replicas available for broadcast_2_piece0 !
21/12/06 10:12:37 WARN BlockManagerMasterEndpoint: No more replicas available for broadcast_1_piece0 !
21/12/06 10:12:37 WARN BlockManagerMasterEndpoint: No more replicas available for broadcast_3_piece0 !
21/12/06 10:12:37 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO DAGScheduler: Job 0 failed: collect at ReadMetadata.java:81, took 332.291412 s
21/12/06 10:12:37 INFO BlockManagerMaster: Removed driver successfully in removeExecutor
21/12/06 10:12:37 INFO DAGScheduler: Shuffle files lost for executor: driver (epoch 0)
21/12/06 10:12:37 INFO DAGScheduler: Host added was in lost list earlier: localhost
21/12/06 10:12:37 ERROR BlockManager: Failed to report broadcast_3_piece0 to master; giving up.
21/12/06 10:12:37 INFO Executor: Told to re-register on heartbeat
21/12/06 10:12:37 INFO BlockManager: BlockManager BlockManagerId(driver, klogin3.local, 33845, None) re-registering with master
21/12/06 10:12:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManagerMasterEndpoint: Registering block manager klogin3.local:33845 with 12.4 GB RAM, BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManager: Reporting 6 blocks to the master.
21/12/06 10:12:37 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on klogin3.local:33845 (size: 2.1 MB, free: 12.4 GB)
21/12/06 10:12:37 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on klogin3.local:33845 (size: 110.2 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on klogin3.local:33845 (size: 35.4 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO Executor: Told to re-register on heartbeat
21/12/06 10:12:37 INFO BlockManager: BlockManager BlockManagerId(driver, klogin3.local, 33845, None) re-registering with master
21/12/06 10:12:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManager: Reporting 6 blocks to the master.
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_1_piece0 in memory on klogin3.local:33845 (current size: 2.1 MB, original size: 2.1 MB, free: 12.4 GB)
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_3_piece0 in memory on klogin3.local:33845 (current size: 110.2 KB, original size: 110.2 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO SparkUI: Stopped Spark web UI at http://klogin3.local:4040
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_2_piece0 in memory on klogin3.local:33845 (current size: 35.4 KB, original size: 35.4 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO Executor: Told to re-register on heartbeat
21/12/06 10:12:37 INFO BlockManager: BlockManager BlockManagerId(driver, klogin3.local, 33845, None) re-registering with master
21/12/06 10:12:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, klogin3.local, 33845, None)
21/12/06 10:12:37 INFO BlockManager: Reporting 6 blocks to the master.
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_1_piece0 in memory on klogin3.local:33845 (current size: 2.1 MB, original size: 2.1 MB, free: 12.4 GB)
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_3_piece0 in memory on klogin3.local:33845 (current size: 110.2 KB, original size: 110.2 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO BlockManagerInfo: Updated broadcast_2_piece0 in memory on klogin3.local:33845 (current size: 35.4 KB, original size: 35.4 KB, free: 12.4 GB)
21/12/06 10:12:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/12/06 10:12:37 INFO MemoryStore: MemoryStore cleared
21/12/06 10:12:37 INFO BlockManager: BlockManager stopped
21/12/06 10:12:37 INFO BlockManagerMaster: BlockManagerMaster stopped
21/12/06 10:12:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/12/06 10:12:37 INFO SparkContext: Successfully stopped SparkContext
10:12:37.946 INFO FindBreakpointEvidenceSpark - Shutting down engine
[December 6, 2021 10:12:37 AM AST] org.broadinstitute.hellbender.tools.spark.sv.evidence.FindBreakpointEvidenceSpark done. Elapsed time: 5.62 minutes.
Runtime.totalMemory()=11475091456
org.apache.spark.SparkException: Job aborted due to stage failure: Task 23 in stage 0.0 failed 1 times, most recent failure: Lost task 23.0 in stage 0.0 (TID 23, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 277788 ms
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.collect(RDD.scala:989)
at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.ReadMetadata.<init>(ReadMetadata.java:81)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.FindBreakpointEvidenceSpark.buildMetadata(FindBreakpointEvidenceSpark.java:259)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.FindBreakpointEvidenceSpark.gatherEvidenceAndWriteContigSamFile(FindBreakpointEvidenceSpark.java:158)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.FindBreakpointEvidenceSpark.runTool(FindBreakpointEvidenceSpark.java:138)
at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.runPipeline(GATKSparkTool.java:546)
at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
21/12/06 10:12:37 INFO ShutdownHookManager: Shutdown hook called
21/12/06 10:12:37 INFO ShutdownHookManager: Deleting directory /tmp/spark-5dbad62b-edbf-4876-affc-a28e1cace00b
Why I'm keeping got this error?
-
Hi Heba B abusamra,
This looks like it may be a memory issue. What kind of machine are you running this on? You may want to specifiy --java-options -xmx to specify how much memory you would like to allocate to the tool as well as specify a number of executor cores to use. You can find some information about running spark tools here: https://github.com/broadinstitute/gatk#sparklocal. This may also work if you try running it with the GATK docker image. I'll also add that this tool is part of a pretty outdated pipeline. Here is a link to a more updated structural variant pipeline: https://github.com/broadinstitute/gatk-sv.
Kind regards,
Pamela
-
Thank you for your reply
I'm using HPC ~200G memory , and tried what you recommends regarding specifying executor cores to use but still have same problem
-
Hi Heba B abusamra,
Thank you for trying my suggestion. Could you please post the full new command that you ran?
Kind regards,
Pamela
-
Here it is
/app/Genome/gatk-4.2.2.0/gatk FindBreakpointEvidenceSpark \
-I Data/GM24385_2_S15.bam \
--aligner-index-image reference.fasta.img \
--kmers-to-ignore kmers_to_ignore.txt \
--spark-runner LOCAL \
--spark-master 'local[*]'\
-L ../../recalset/nextera_dna_exome_targeted_regions_manifest_v1_2.bed \
-O assemblies2.sam -
Thank you Heba B abusamra. Did you specify --java-options -Xmx in your command? I don't see it in your post and I believe specifying how much memory to give this job may help with your problem. I would recommend allocating about 70-80% of the available memory on your machine to the tool.
Kind regards,
Pamela
-
It gave me an error when using --java-options -Xmx with gatk. However I used it with gatk-package-4.2.2.0-local.jar. Unfortunately I still get the same problem
java -Xms200m -jar /app/Genome/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar FindBreakpointEvidenceSpark \
-I Data/GM24385_2_S15.bam \
--aligner-index-image reference.fasta.img \
--kmers-to-ignore kmers_to_ignore.txt \
-L ../../recalset/nextera_dna_exome_targeted_regions_manifest_v1_2.bed \
-O assemblies2.sam -
Hi Heba B abusamra,
Okay, thank you. I still think this is likely an issue with memory as spark tools tend to use up a lot of memory and require the right amount of memory/cores to be allocated. Can you confirm that you have adequate disk space available on your machine? I would recommend playing around with the memory allocation and the number of executor cores that you specify, particularly trying a very low number of cores.
Example:
--spark-runner LOCAL \ --num-executors 5 --executor-cores 2 --executor-memory 4g \ --conf spark.executor.memoryOverhead=600
Kind regards,
Pamela
Please sign in to leave a comment.
7 comments