JointGenotyping.ImportGvcfs terminates without an active exception
JointGenotyping fails in ImportGvcfs with the c++ error "terminate called without an active exception", which occurs when a thread goes out of scope without calling join() or detach(). This occurs when running JointGenotyping on 345 gvcfs created by GATK4 ExomeGermlineSingleSample; the workflow is running on an HPC cluster in Singularity (single node, 32 cores/node, 1002GB node memory) NOTE that I am able to successfully run JointGenotyping on a set of 80 gvcfs, also produced by ExomeGermlineSingleSample, in this HPC/Singularity environment with 248GB memory, 24 cores/node - this doesn't seem to be a resource issue. The only difference appears to be the number of input gvcfs, which is still quite small (345 vs 80). The number of reader threads for GenomicsDBImport has been hard-coded to 1 because these are exome sequences; scatter count = 10, batch size = 50, gather_vcfs = false. GenomicsDBImport appears to succeed on all 10 shards but workflow execution fails with exactly the same c++ error, see below.
REQUIRED for all errors and issues:
a) GATK version used: v4.2.6.1
b) Exact command used:
java -Dconfig.file=/scratch.global/lee04110/config/sing-cache.conf -jar /home/pankrat2/public/bin/gatk4/cromwell-81.jar run -i /scratch.global/lee04110/config/jg.ca_defects.json /home/pankrat2/public/bin/gatk4/warp/pipelines/broad/dna_seq/germline/joint_genotyping/JointGenotyping.wdl -o <(echo '{"final_workflow_outputs_dir" : "/scratch.global/lee04110/tmp_jg", "use_relative_output_paths" : true, "workflow-log-temporary" : true}')
c) Entire program log: (too big to include the whole thing)
(From main process stderr, picking from SplitInterval setting status to Done)
[2022-10-18 15:38:20,88] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.SplitIntervalList:NA:1]: Status change from WaitingForReturnCode to Done
[2022-10-18 15:38:25,47] [info] WorkflowExecutionActor-9743b28a-3819-49a7-8598-b0c5267647ee [9743b28a]: Starting JointGenotyping.ImportGVCFs (10 shards)
[2022-10-18 15:38:33,03] [info] Assigned new job execution tokens to the following groups: 9743b28a: 10
[2022-10-18 15:38:33,14] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:3:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,14] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:2:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,15] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:6:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,15] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:9:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,16] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:0:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,17] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:4:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,17] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:5:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,18] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:8:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,18] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:7:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:33,19] [warn] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:1:1]: Unrecognized runtime attribute keys: preemptible, bootDiskSizeGb, disks, cpu, memory
[2022-10-18 15:38:43,39] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:4:1]: set -euo pipefail
rm -rf genomicsdb
.... (and so on)
singularity exec --containall docker://us.gcr.io/broad-gatk/gatk@sha256:21c3cb43b7d11891ed4b63cc7274f20187f00387cfaa0433b3e7991b5be34dbe \
echo "successfully pulled us.gcr.io/broad-gatk/gatk@sha256:21c3cb43b7d11891ed4b63cc7274f20187f00387cfaa0433b3e7991b5be34dbe!"
singularity exec --containall --bind /scratch.global/lee04110/cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-0:/cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-0 docker://us.gcr.io/broad-gatk/gatk@sha256:21c3cb43b7d11891ed4b63cc7274f20187f00387cfaa0433b3e7991b5be34dbe /bin/bash /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-0/execution/script
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:5:1]: job id: 698507
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:2:1]: job id: 698509
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:4:1]: job id: 698505
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:9:1]: job id: 698508
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:6:1]: job id: 698500
[2022-10-18 15:38:47,76] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:8:1]: job id: 698502
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:9:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:2:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:4:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:0:1]: job id: 698514
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:3:1]: job id: 698498
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:7:1]: job id: 698506
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:3:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,77] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:1:1]: job id: 698504
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:7:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:1:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:0:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:8:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:5:1]: Status change from - to WaitingForReturnCode
[2022-10-18 15:38:47,78] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:6:1]: Status change from - to WaitingForReturnCode
[2022-10-19 00:09:45,72] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:9:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 00:15:57,43] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:7:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 00:26:11,39] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:8:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 00:43:41,21] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:5:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 00:53:53,08] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:4:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 00:59:03,47] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:6:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 01:01:13,08] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:1:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 01:02:20,65] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:3:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 01:11:44,74] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:0:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 01:26:43,24] [info] BackgroundConfigAsyncJobExecutionActor [9743b28aJointGenotyping.ImportGVCFs:2:1]: Status change from WaitingForReturnCode to Done
[2022-10-19 01:26:43,97] [info] WorkflowManagerActor: Workflow 9743b28a-3819-49a7-8598-b0c5267647ee failed (during ExecutingWorkflowState): Job JointGenotyping.ImportGVCFs:5:1 exited with return code 250 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /scratch.global/lee04110/cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/execution/stderr.
[First 3000 bytes]:Using GATK jar /gatk/gatk-package-4.2.6.1-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms8000m -Xmx25000m -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -jar /gatk/gatk-package-4.2.6.1-local.jar GenomicsDBImport --genomicsdb-workspace-path genomicsdb --batch-size 50 -L /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/-1806236336/0005-scattered.interval_list -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0478.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0496.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0559.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0568_B.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0577_B.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0586.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0595.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0676.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0685.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0739.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0748.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0757.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0775.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0784.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ0793.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/000006KQ1479.rb.g.vcf.gz -V /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-5/inputs/1422537242/00000
...(all the shards fail in the same way)
(this is stderr.background for one shard; all 10 shards log the same error)
lee04110@ln0005 [/scratch.global/lee04110/batch] % cat /scratch.global/lee04110/cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/execution/stderr.background
INFO: Using cached SIF image
INFO: Using cached SIF image
Using GATK jar /gatk/gatk-package-4.2.6.1-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms8000m -Xmx25000m -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -jar /gatk/gatk-package-4.2.6.1-local.jar GenomicsDBImport --genomicsdb-workspace-path genomicsdb --batch-size 50 -L /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/inputs/-1806236336/0009-scattered.interval_list [...list of input gvcs] --reader-threads 1 --merge-input-intervals true --consolidate false
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/tmp.bd1b0bc7
20:38:55.819 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.6.1-local.jar!/com/intel/gkl/native/libgkl_compression.so
20:38:56.233 INFO GenomicsDBImport - ------------------------------------------------------------
20:38:56.233 INFO GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.2.6.1
20:38:56.234 INFO GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
20:38:56.237 INFO GenomicsDBImport - Executing as lee04110@cn2006.mesabi.msi.umn.edu on Linux v3.10.0-1160.76.1.el7.x86_64 amd64
20:38:56.237 INFO GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
20:38:56.238 INFO GenomicsDBImport - Start Date/Time: October 18, 2022 8:38:55 PM GMT
20:38:56.238 INFO GenomicsDBImport - ------------------------------------------------------------
20:38:56.238 INFO GenomicsDBImport - ------------------------------------------------------------
20:38:56.239 INFO GenomicsDBImport - HTSJDK Version: 2.24.1
20:38:56.239 INFO GenomicsDBImport - Picard Version: 2.27.1
20:38:56.239 INFO GenomicsDBImport - Built for Spark Version: 2.4.5
20:38:56.239 INFO GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 2
20:38:56.240 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
20:38:56.240 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
20:38:56.240 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
20:38:56.240 INFO GenomicsDBImport - Deflater: IntelDeflater
20:38:56.240 INFO GenomicsDBImport - Inflater: IntelInflater
20:38:56.240 INFO GenomicsDBImport - GCS max retries/reopens: 20
20:38:56.240 INFO GenomicsDBImport - Requester pays: disabled
20:38:56.241 INFO GenomicsDBImport - Initializing engine
20:39:48.002 INFO FeatureManager - Using codec IntervalListCodec to read file file:///cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/inputs/-1806236336/0009-scattered.interval_list
20:39:48.971 INFO IntervalArgumentCollection - Processing 7607171 bp from intervals
20:39:49.135 INFO GenomicsDBImport - Done initializing engine
20:39:51.479 INFO GenomicsDBLibLoader - GenomicsDB native library version : 1.4.3-6069e4a
20:39:51.491 INFO GenomicsDBImport - Vid Map JSON file will be written to /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/execution/genomicsdb/vidmap.json
20:39:51.491 INFO GenomicsDBImport - Callset Map JSON file will be written to /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/execution/genomicsdb/callset.json
20:39:51.491 INFO GenomicsDBImport - Complete VCF Header will be written to /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/execution/genomicsdb/vcfheader.vcf
20:39:51.491 INFO GenomicsDBImport - Importing to workspace - /cromwell-executions/JointGenotyping/9743b28a-3819-49a7-8598-b0c5267647ee/call-ImportGVCFs/shard-9/execution/genomicsdb
20:39:54.642 INFO GenomicsDBImport - Importing batch 1 with 50 samples
20:44:04.621 INFO GenomicsDBImport - Importing batch 1 with 50 samples
21:01:11.744 INFO GenomicsDBImport - Importing batch 1 with 50 samples
21:09:09.841 INFO GenomicsDBImport - Importing batch 1 with 50 samples
21:24:11.009 INFO GenomicsDBImport - Importing batch 1 with 50 samples
21:57:16.144 INFO GenomicsDBImport - Importing batch 1 with 50 samples
21:57:55.591 INFO GenomicsDBImport - Done importing batch 1/7
21:57:58.084 INFO GenomicsDBImport - Importing batch 2 with 50 samples
22:02:06.289 INFO GenomicsDBImport - Importing batch 2 with 50 samples
22:25:37.601 INFO GenomicsDBImport - Importing batch 2 with 50 samples
22:36:09.210 INFO GenomicsDBImport - Importing batch 2 with 50 samples
22:51:30.423 INFO GenomicsDBImport - Importing batch 2 with 50 samples
23:21:43.424 INFO GenomicsDBImport - Importing batch 2 with 50 samples
23:22:22.070 INFO GenomicsDBImport - Done importing batch 2/7
23:22:29.783 INFO GenomicsDBImport - Importing batch 3 with 50 samples
23:26:34.022 INFO GenomicsDBImport - Importing batch 3 with 50 samples
23:43:44.480 INFO GenomicsDBImport - Importing batch 3 with 50 samples
23:52:10.742 INFO GenomicsDBImport - Importing batch 3 with 50 samples
00:07:11.292 INFO GenomicsDBImport - Importing batch 3 with 50 samples
00:37:55.692 INFO GenomicsDBImport - Importing batch 3 with 50 samples
00:38:39.687 INFO GenomicsDBImport - Done importing batch 3/7
00:38:46.984 INFO GenomicsDBImport - Importing batch 4 with 50 samples
00:42:24.891 INFO GenomicsDBImport - Importing batch 4 with 50 samples
00:57:09.520 INFO GenomicsDBImport - Importing batch 4 with 50 samples
01:05:34.391 INFO GenomicsDBImport - Importing batch 4 with 50 samples
01:18:00.662 INFO GenomicsDBImport - Importing batch 4 with 50 samples
01:45:02.441 INFO GenomicsDBImport - Importing batch 4 with 50 samples
01:45:38.712 INFO GenomicsDBImport - Done importing batch 4/7
01:45:45.479 INFO GenomicsDBImport - Importing batch 5 with 50 samples
01:50:11.697 INFO GenomicsDBImport - Importing batch 5 with 50 samples
02:11:57.531 INFO GenomicsDBImport - Importing batch 5 with 50 samples
02:20:02.171 INFO GenomicsDBImport - Importing batch 5 with 50 samples
02:35:13.654 INFO GenomicsDBImport - Importing batch 5 with 50 samples
03:04:30.490 INFO GenomicsDBImport - Importing batch 5 with 50 samples
03:05:06.171 INFO GenomicsDBImport - Done importing batch 5/7
03:05:14.150 INFO GenomicsDBImport - Importing batch 6 with 50 samples
03:08:31.080 INFO GenomicsDBImport - Importing batch 6 with 50 samples
03:23:52.054 INFO GenomicsDBImport - Importing batch 6 with 50 samples
03:30:37.049 INFO GenomicsDBImport - Importing batch 6 with 50 samples
03:43:46.119 INFO GenomicsDBImport - Importing batch 6 with 50 samples
04:09:27.761 INFO GenomicsDBImport - Importing batch 6 with 50 samples
04:10:10.953 INFO GenomicsDBImport - Done importing batch 6/7
04:10:18.233 INFO GenomicsDBImport - Importing batch 7 with 45 samples
04:13:55.022 INFO GenomicsDBImport - Importing batch 7 with 45 samples
04:27:28.342 INFO GenomicsDBImport - Importing batch 7 with 45 samples
04:33:32.781 INFO GenomicsDBImport - Importing batch 7 with 45 samples
04:44:09.752 INFO GenomicsDBImport - Importing batch 7 with 45 samples
05:04:33.112 INFO GenomicsDBImport - Importing batch 7 with 45 samples
05:05:02.272 INFO GenomicsDBImport - Done importing batch 7/7
05:05:02.299 INFO GenomicsDBImport - Import of all batches to GenomicsDB completed!
05:05:02.300 INFO GenomicsDBImport - Shutting down engine
[October 19, 2022 5:05:02 AM GMT] org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport done. Elapsed time: 506.11 minutes.
Runtime.totalMemory()=8653373440
pure virtual method called
terminate called without an active exception
-
Hi Elizabeth Lee,
Thank you for writing to the GATK forum! I hope that we can help you sort this out.
I brought this to our developers and received some feedback to share. Firstly, they would like to know if you can see any output after the GenomicsDBImport run completes? The issue appears to be occurring after the tool finishes its run and tries to shut down.
Try running SelectVariants in the GenomicsDBImport. This appears to be a bug in the GenomicsDBImport code. I have created a GitHub ticket and tagged our GenomicsDBImport team members to help you going forward.
Please find the link to the GitHub ticket here. You can follow along with all progress there. I'm confident that our team will be able to help you solve this as soon as possible. If you have any other questions in the meantime, please do not hesitate to reach out.
Best,
Anthony
-
Hi Elizabeth Lee,
We haven't heard from you in a while so we're going to close out this ticket. If you still require assistance, simply respond to this email and we'll be happy to pick up where we left off!
Kind regards,
Anthony
-
Hi Elizabeth Lee,
As promised, I pinged our developers to see if they had any additional questions and feedback for you.
Upon further review, it appears that it is not a bug with GenomicsDBImport; the problem seems to originate from your environment. Can you try running GenomicsDBImport with Docker instead of Singularity? Do you have a workaround? Is this repeatable? Have you run this on separate days with the same commands and had it fail the same way?
Regrettably, we do not support or have much familiarity with Singularity, but we do support issues with Docker (containers - image - registry) and Google Cloud. So if you re-run with Docker and encounter the same problem, we may be better equipped and able to help you troubleshoot.
Thank you for being a valued contributor to the GATK community! We hope to hear back from you soon. In the meantime, please do not hesitate to reach out if any other questions arise.
Best,
Anthony
-
Hi Elizabeth Lee,
We haven't heard from you for a while, so we will close this GATK forum ticket internally. However, this forum thread will remain open, along with your GitHub ticket. If you still require assistance or have further updates to my last reply, you only need to respond to this email, and we'll re-open your ticket and would be more than happy to pick up where we left off!
Kind regards,
Anthony
Please sign in to leave a comment.
4 comments