Hi, GATK team and community!

I am trying to process a large number of BAM files with MarkDuplicatesSpark run locally (without a spark cluster). These BAM files come from a very deeply sequenced WGS sample with ~1.8 billion R1 reads. To speed up alignment, I sliced the R1 and R2 FASTQ files into 25 million reads, aligned in parallel, and queryname sorted. I've gotten the following error multiple times.

22/06/09 17:43:56 INFO DAGScheduler: ShuffleMapStage 69 (mapToPair at MarkDuplicatesSpark.java:215) failed in 7272.579 s due to Job aborted due to stage failure: Task 1005 in stage 69.0 failed 1 times, most recent failure: Lost task 1005.0 in stage 69.0 (TID 19573, localhost, executor driver): java.lang.OutOfMemoryError: GC overhead limit exceeded

Looks like the java call is running out of memory available in the heap. I allocated "-Xmx124000m". I'm honestly unsure if that is in decimal or binary, but regardless it's ~124GB as the max heap. The virtual machine it's running on has 186GB of RAM, 48 cores, and ~ 1500GB of free space (almost all of which should be available to the --tmp-dir). I can try allocating more RAM to the heap but it's hard to imagine 124GB not being enough, and I've grown weary of starting analysis, waiting 6 hours for it to fail, and iteratively upping the -Xmx. I was hoping for some help figuring out what the underlying problem is. I'm unsure if this is a bug report or not so I'm putting it in the "Other" section. Feel free to move if you think it's a good idea!

REQUIRED for all errors and issues:
a) GATK version used:
Running in a Docker pulled from broadinstitute/gatk:4.2.4.0

b) Exact command used:
My command got truncated in the logs (because there are 68 input BAM files). But here is my WDL command.

~{gatkPath} --java-options "-Dsamjdk.compression_level=5 -Xms4000m -Xmx~{heapMb}m" \
MarkDuplicatesSpark \
--input ~{sep=' --input ' bams} \
--output ~{outBam} \
--read-validation-stringency SILENT \
--optical-duplicate-pixel-distance ~{optDupPixDist} \
--tmp-dir /home/dnanexus/tmp \
--metrics-file ~{metrics}

c) Entire program log:
I am running this on the DNAnexus platform so even though I cannot paste the entire log in this post (I tried several times and it became non-responsive. Too much text to paste I think.), I am actually not able to view the entire log because DNAnexus only reports the first ~15000 lines and the last 100. MarkDuplicatesSpark is certainly verbose. DNAnexus reports rejecting 87100 lines. Below are the GATK relevant  lines at the top of the log and the last 100 lines, which include the error.

MrkDupsSparkAndMergeRg:body (MrkDupsSpark:body) (failed) job-GBQy9pQ085g4XBqzFXQpbb42
2022-06-09 07:42:27 MrkDupsSparkAndMergeRg:body STDOUT number of machine cores: 48
2022-06-09 07:42:27 MrkDupsSparkAndMergeRg:body STDOUT memory size: 186 GiB
2022-06-09 07:53:22 MrkDupsSparkAndMergeRg:body STDERR WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
2022-06-09 07:53:23 MrkDupsSparkAndMergeRg:body STDERR + /gatk/gatk --java-options '-Dsamjdk.compression_level=5 -Xms4000m -Xmx124000m' MarkDuplicatesSpark --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqp80y53ggQGG8Zg9pPV2/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq380vXJB2xjQ1597gpGQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqkj07X6x15BgFX97Xkbz/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq0j0FPJYkfz19JZ21ZZj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqfQ0V1JvVZ3V8b49f7PV/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqXj0BzqV38XjBKFv3ZY7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq000BYJ761k82yYkKQ25/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv000jj7z2Qyb988jfkJ3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqx00y4XP03PX97yybpFY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv1807j0bKg0y8b32Kz3p/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqyQ0xbfZXBqzFXQp96B9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqvj0kQ1qzP15BB5XfFV7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqvj0Qk1P2gkb983YQ42f/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv2808ZVfPqjZ8Zfv2fPf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv280PjFV38XjBKFv3Zbp/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv68080kX3yV55gG3F3zY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqzQ0268p2xjQ1597gpXf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv280B8zK9fjf5qZPJFJ1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ0Xv523yV55gG3BQBX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ0XFPvzB285f740y8x/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ04BFbgp4j5q7bvPG3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqp801Py5176XBB8PxGKk/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6Q0P6Kgy5y14ZxB1P4X/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJ80bJP761k82yYkKVbP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv4Q06B0xvYqz5fvyz5F4/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv9000Qpk5z9V97kVY95k/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvQ803xJV9q2p2yxQZJg1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv7j02JFGVZ3V8b49f7Z5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv900bgy3369Z2yPyxj02/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJ00PPx9KGQq5q4Q2kgv/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGj0KGvyjzFf9Ffyv9Xj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvZ00k74YyXGGP0QVzyp1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVpV80vGvB2XKk2yFb32Zf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGj09qgG4x9p8j0GVqb8/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6Q07qX5vxJPFVK2GJv5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvKj03jZg3BVX97bPky6q/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv980036JvXxb5g34QjzP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv7Q0Z2Jx4X1QB9pyxy9P/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv9802VPV9q2p2yxQZJXQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6j026pF1zKk15bXq59V/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvP00VF9Z5PpP2y7yqfjX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGQ0VqbvZ20kKyFgPKGf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJQ04b6f2p1G5yb2xbXj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvG00X7f23ZQQ5fVJ0pQP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJj0j8v5gp4j5q7bvPX1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvK80Pj6B2xjQ1597gpk9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvB002gv5ZV904gzBBFj5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvK805F4yfVp32yx08QpX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvqj0673v323ZKx802V0x/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqgj0yj3bKg0y8b32Kz0z/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvk80fBvyBVJ9FY9FvP26/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvz806q86Y373BGfQvg22/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvpj0zjGGx9bf8Zx82g5V/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvyQ0vXbP03PX97yybq23/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvzj04x7YKkj45fyP3vyY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx5j0kfzxp44j14xzKVf9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVxGQ0JjxFzP15BB5XfG61/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx3j0gbXP2gkb983YQ4Zg/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvkQ0Xqkbq4b55fz13Bf3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvy003Fff2p1G5yb2xbxQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvx8098pqBPFpFY4p1GBB/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvpQ0Pj8y9fjf5qZPJGXP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvf80qfJ5q4b55fz13BZ7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVxJj0VgVZXBqzFXQp96v3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx2j07YFvx9bf8Zx82gG8/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQX2v80QGVB2xjQ1597gyjZ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761... [truncated]
2022-06-09 07:53:23 MrkDupsSparkAndMergeRg:body STDERR Using GATK jar /gatk/gatk-package-4.2.4.0-local.jar
2022-06-09 07:53:23 MrkDupsSparkAndMergeRg:body STDERR Running:
2022-06-09 07:53:23 MrkDupsSparkAndMergeRg:body STDERR     java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Dsamjdk.compression_level=5 -Xms4000m -Xmx124000m -jar /gatk/gatk-package-4.2.4.0-local.jar MarkDuplicatesSpark --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqp80y53ggQGG8Zg9pPV2/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq380vXJB2xjQ1597gpGQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqkj07X6x15BgFX97Xkbz/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq0j0FPJYkfz19JZ21ZZj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqfQ0V1JvVZ3V8b49f7PV/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqXj0BzqV38XjBKFv3ZY7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVq000BYJ761k82yYkKQ25/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv000jj7z2Qyb988jfkJ3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqx00y4XP03PX97yybpFY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv1807j0bKg0y8b32Kz3p/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqyQ0xbfZXBqzFXQp96B9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqvj0kQ1qzP15BB5XfFV7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqvj0Qk1P2gkb983YQ42f/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv2808ZVfPqjZ8Zfv2fPf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv280PjFV38XjBKFv3Zbp/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv68080kX3yV55gG3F3zY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqzQ0268p2xjQ1597gpXf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv280B8zK9fjf5qZPJFJ1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ0Xv523yV55gG3BQBX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ0XFPvzB285f740y8x/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqpQ04BFbgp4j5q7bvPG3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqp801Py5176XBB8PxGKk/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6Q0P6Kgy5y14ZxB1P4X/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJ80bJP761k82yYkKVbP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv4Q06B0xvYqz5fvyz5F4/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv9000Qpk5z9V97kVY95k/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvQ803xJV9q2p2yxQZJg1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv7j02JFGVZ3V8b49f7Z5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv900bgy3369Z2yPyxj02/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJ00PPx9KGQq5q4Q2kgv/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGj0KGvyjzFf9Ffyv9Xj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvZ00k74YyXGGP0QVzyp1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVpV80vGvB2XKk2yFb32Zf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGj09qgG4x9p8j0GVqb8/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6Q07qX5vxJPFVK2GJv5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvKj03jZg3BVX97bPky6q/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv980036JvXxb5g34QjzP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv7Q0Z2Jx4X1QB9pyxy9P/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv9802VPV9q2p2yxQZJXQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVv6j026pF1zKk15bXq59V/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvP00VF9Z5PpP2y7yqfjX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvGQ0VqbvZ20kKyFgPKGf/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJQ04b6f2p1G5yb2xbXj/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvG00X7f23ZQQ5fVJ0pQP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvJj0j8v5gp4j5q7bvPX1/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvK80Pj6B2xjQ1597gpk9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvB002gv5ZV904gzBBFj5/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvK805F4yfVp32yx08QpX/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvqj0673v323ZKx802V0x/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVqgj0yj3bKg0y8b32Kz0z/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvk80fBvyBVJ9FY9FvP26/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvz806q86Y373BGfQvg22/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvpj0zjGGx9bf8Zx82g5V/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvyQ0vXbP03PX97yybq23/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvzj04x7YKkj45fyP3vyY/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx5j0kfzxp44j14xzKVf9/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVxGQ0JjxFzP15BB5XfG61/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx3j0gbXP2gkb983YQ4Zg/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvkQ0Xqkbq4b55fz13Bf3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvy003Fff2p1G5yb2xbxQ/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvx8098pqBPFpFY4p1GBB/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvpQ0Pj8y9fjf5qZPJGXP/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVvf80qfJ5q4b55fz13BZ7/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVxJj0VgVZXBqzFXQp96v3/DS-360233.hg19.aligned.sorted.bam --input /home/dnanexus/inputs/input761450227589884713/file-GBQVx2j07YFvx9bf8Zx82gG8/DS-360233.hg19... [truncated]
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.300 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR Jun 09, 2022 12:53:25 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR INFO: Failed to detect whether we are running on Google Compute Engine.
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.420 INFO  MarkDuplicatesSpark - ------------------------------------------------------------
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.420 INFO  MarkDuplicatesSpark - The Genome Analysis Toolkit (GATK) v4.2.4.0
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.420 INFO  MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.420 INFO  MarkDuplicatesSpark - Executing as root@job-GBQy9pQ085g4XBqzFXQpbb42 on Linux v5.4.0-1072-aws amd64
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.420 INFO  MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - Start Date/Time: June 9, 2022 12:53:25 PM GMT
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - ------------------------------------------------------------
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - ------------------------------------------------------------
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - HTSJDK Version: 2.24.1
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - Picard Version: 2.25.4
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - Built for Spark Version: 2.4.5
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 5
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - Deflater: IntelDeflater
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.421 INFO  MarkDuplicatesSpark - Inflater: IntelInflater
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.422 INFO  MarkDuplicatesSpark - GCS max retries/reopens: 20
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.422 INFO  MarkDuplicatesSpark - Requester pays: disabled
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.422 INFO  MarkDuplicatesSpark - Initializing engine
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 12:53:25.422 INFO  MarkDuplicatesSpark - Done initializing engine
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SparkContext: Running Spark version 2.4.5
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SparkContext: Submitted application: MarkDuplicatesSpark
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SecurityManager: Changing view acls to: root
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SecurityManager: Changing modify acls to: root
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SecurityManager: Changing view acls groups to: 
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SecurityManager: Changing modify acls groups to: 
2022-06-09 07:53:25 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2022-06-09 07:53:26 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 12:53:26 INFO Utils: Successfully started service 'sparkDriver' on port 40251.

########
LAST 100
########

2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1002.0 in stage 69.0 (TID 19570), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1003.0 in stage 69.0 (TID 19571), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 995.0 in stage 69.0 (TID 19563), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1025.0 in stage 69.0 (TID 19593), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 996.0 in stage 69.0 (TID 19564), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1026.0 in stage 69.0 (TID 19594), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1018.0 in stage 69.0 (TID 19586), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 997.0 in stage 69.0 (TID 19565), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1019.0 in stage 69.0 (TID 19587), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 998.0 in stage 69.0 (TID 19566), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 990.0 in stage 69.0 (TID 19558), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1020.0 in stage 69.0 (TID 19588), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 991.0 in stage 69.0 (TID 19559), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1021.0 in stage 69.0 (TID 19589), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1013.0 in stage 69.0 (TID 19581), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 992.0 in stage 69.0 (TID 19560), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1014.0 in stage 69.0 (TID 19582), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 993.0 in stage 69.0 (TID 19561), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1015.0 in stage 69.0 (TID 19583), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 994.0 in stage 69.0 (TID 19562), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 986.0 in stage 69.0 (TID 19554), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1016.0 in stage 69.0 (TID 19584), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 987.0 in stage 69.0 (TID 19555), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1017.0 in stage 69.0 (TID 19585), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1009.0 in stage 69.0 (TID 19577), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 988.0 in stage 69.0 (TID 19556), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1010.0 in stage 69.0 (TID 19578), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 989.0 in stage 69.0 (TID 19557), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1032.0 in stage 69.0 (TID 19600), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor is trying to kill task 1011.0 in stage 69.0 (TID 19579), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 994.0 in stage 69.0 (TID 19562), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO TaskSchedulerImpl: Stage 69 was cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 999.0 in stage 69.0 (TID 19567), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO DAGScheduler: ShuffleMapStage 69 (mapToPair at MarkDuplicatesSpark.java:215) failed in 7272.579 s due to Job aborted due to stage failure: Task 1005 in stage 69.0 failed 1 times, most recent failure: Lost task 1005.0 in stage 69.0 (TID 19573, localhost, executor driver): java.lang.OutOfMemoryError: GC overhead limit exceeded
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:188)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:185)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:156)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:90)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.scheduler.Task.run(Task.scala:123)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR  at java.lang.Thread.run(Thread.java:748)
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR Driver stacktrace:
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 996.0 in stage 69.0 (TID 19564), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 1006.0 in stage 69.0 (TID 19574), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 997.0 in stage 69.0 (TID 19565), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 995.0 in stage 69.0 (TID 19563), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 992.0 in stage 69.0 (TID 19560), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 1007.0 in stage 69.0 (TID 19575), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 988.0 in stage 69.0 (TID 19556), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 991.0 in stage 69.0 (TID 19559), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 998.0 in stage 69.0 (TID 19566), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 990.0 in stage 69.0 (TID 19558), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 989.0 in stage 69.0 (TID 19557), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 986.0 in stage 69.0 (TID 19554), reason: Stage cancelled
2022-06-09 12:43:56 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:43:56 INFO Executor: Executor killed task 993.0 in stage 69.0 (TID 19561), reason: Stage cancelled
2022-06-09 12:44:53 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:44:53 ERROR ShutdownHookManager: ShutdownHookManger shutdown forcefully after 30 seconds.
2022-06-09 12:44:53 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:44:53 INFO Executor: Executor killed task 1008.0 in stage 69.0 (TID 19576), reason: Stage cancelled
2022-06-09 12:44:53 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:44:53 INFO Executor: Executor killed task 1009.0 in stage 69.0 (TID 19577), reason: Stage cancelled
2022-06-09 12:44:53 MrkDupsSparkAndMergeRg:body STDERR 22/06/09 17:44:53 INFO DAGScheduler: Job 68 failed: collectAsMap at MarkDuplicatesSparkUtils.java:447, took 17428.083902 s
2022-06-09 12:44:58 MrkDupsSparkAndMergeRg:body STDERR +++ cat /home/dnanexus/meta/containerId
2022-06-09 12:44:58 MrkDupsSparkAndMergeRg:body STDERR ++ docker wait 021cecd594e9867cde2583d9c7e5d7adf00dcdb867900f114e01c603aae05d5f
2022-06-09 12:44:58 MrkDupsSparkAndMergeRg:body STDERR + rc=0
2022-06-09 12:44:58 MrkDupsSparkAndMergeRg:body STDERR ++ cat /home/dnanexus/meta/containerId
2022-06-09 12:44:58 MrkDupsSparkAndMergeRg:body STDERR + docker rm 021cecd594e9867cde2583d9c7e5d7adf00dcdb867900f114e01c603aae05d5f
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDOUT exit $rc021cecd594e9867cde2583d9c7e5d7adf00dcdb867900f114e01c603aae05d5f
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + exit 0
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + rc=1
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + [[ -f /home/dnanexus/meta/returnCode ]]
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR ++ cat /home/dnanexus/meta/returnCode
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + file_rc=52
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + [[ -z 52 ]]
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + rc=52
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + [[ 52 != 0 ]]
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + [[ -f '' ]]
2022-06-09 12:45:01 MrkDupsSparkAndMergeRg:body STDERR + exit 52
MrkDupsSparkAndMergeRg:body (MrkDupsSpark:body) (failed) job-GBQy9pQ085g4XBqzFXQpbb42
msnyder5 2022-06-09 07:17:46 (runtime 5:04:18)
AppInternalError: Please consult the job log; the job's reported error could not be parsed from
the file job_error.json