Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

Jointgenotyping Genomicsdbimport failed for no obvious reasons

Follow
Avatar
Jason Ni
  • Edited

Hi! I've been trying to do the joint genotying with VSQR corrections using the pipeline on https://dockstore.org/workflows/github.com/broadinstitute/warp/JointGenotyping:JointGenotyping_v1.5.1?tab=info

I have 57 samples with gVCFs to merge. As you can see this is one of the shard of a total of 9 shards on terra. The import seems to be working just fine however I'm not sure why the GenomicsDB.tar is not present when the command finishes. Is there specific adaptations when using this workflow on terra? Thanks a ton! Technical reports posted below

REQUIRED for all errors and issues:
a) GATK version used:

v4.1.8.0


b) Exact command used:

Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms8g -jar /gatk/gatk-package-4.1.8.0-local.jar GenomicsDBImport --genomicsdb-workspace-path genomicsdb --batch-size 50 -L /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-SplitIntervalList/cacheCopy/glob-d928cd0f5fb17b6bd5e635f48c18ccfb/0008-scattered.interval_list --sample-name-map /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/1511c824-d248-4d69-86c3-469425952d44/GenerateSampleMap/08e26b9d-d47c-4408-a65c-51321dd20bc5/call-GenerateSampleMapFile/generate-sample-map_2022-10-12T04-05-19.sample_map --reader-threads 5 --merge-input-intervals --consolidate --gcs-project-for-requester-pays t2t-nhgri

 
c) Entire program log:

2022/10/12 05:09:16 Starting container setup.
2022/10/12 05:09:18 Done container setup.
2022/10/12 05:09:20 Starting localization.
2022/10/12 05:09:30 Localization script execution started...
2022/10/12 05:09:30 Localizing input gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/1511c824-d248-4d69-86c3-469425952d44/GenerateSampleMap/08e26b9d-d47c-4408-a65c-51321dd20bc5/call-GenerateSampleMapFile/generate-sample-map_2022-10-12T04-05-19.sample_map -> /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/1511c824-d248-4d69-86c3-469425952d44/GenerateSampleMap/08e26b9d-d47c-4408-a65c-51321dd20bc5/call-GenerateSampleMapFile/generate-sample-map_2022-10-12T04-05-19.sample_map
2022/10/12 05:09:33 Localizing input gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-SplitIntervalList/cacheCopy/glob-d928cd0f5fb17b6bd5e635f48c18ccfb/0008-scattered.interval_list -> /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-SplitIntervalList/cacheCopy/glob-d928cd0f5fb17b6bd5e635f48c18ccfb/0008-scattered.interval_list
2022/10/12 05:09:35 Localizing input gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/script -> /cromwell_root/script
2022/10/12 05:09:36 Localizing input gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.dict -> /cromwell_root/genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.dict
2022/10/12 05:09:38 Localizing input gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta -> /cromwell_root/genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta
2022/10/12 05:09:38 Localizing input gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta.fai -> /cromwell_root/genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta.fai
Copying gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta...
/ [0 files][    0.0 B/  3.0 GiB]                                                
-
- [0 files][ 68.6 MiB/  3.0 GiB]                                                
\
|
| [0 files][145.7 MiB/  3.0 GiB]                                                
/
/ [0 files][222.8 MiB/  3.0 GiB]                                                
-
\
\ [0 files][299.1 MiB/  3.0 GiB]                                                
|
/
/ [0 files][376.2 MiB/  3.0 GiB]                                                
-
- [0 files][453.0 MiB/  3.0 GiB]                                                
\
|
| [0 files][529.6 MiB/  3.0 GiB]                                                
/
-
- [0 files][605.1 MiB/  3.0 GiB]                                                
\
\ [0 files][681.7 MiB/  3.0 GiB]                                                
|
/
/ [0 files][758.0 MiB/  3.0 GiB]                                                
-
\
\ [0 files][823.8 MiB/  3.0 GiB]   74.1 MiB/s                                   
|
| [0 files][896.0 MiB/  3.0 GiB]   73.2 MiB/s                                   
/
-
- [0 files][965.1 MiB/  3.0 GiB]   71.3 MiB/s                                   
\
|
| [0 files][  1.0 GiB/  3.0 GiB]   69.3 MiB/s                                   
/
-
- [0 files][  1.1 GiB/  3.0 GiB]   66.7 MiB/s                                   
\
\ [0 files][  1.1 GiB/  3.0 GiB]   65.0 MiB/s                                   
|
/
/ [0 files][  1.2 GiB/  3.0 GiB]   64.3 MiB/s                                   
-
\
\ [0 files][  1.3 GiB/  3.0 GiB]   64.7 MiB/s                                   
|
| [0 files][  1.3 GiB/  3.0 GiB]   64.4 MiB/s                                   
/
-
- [0 files][  1.4 GiB/  3.0 GiB]   64.6 MiB/s                                   
\
\ [0 files][  1.4 GiB/  3.0 GiB]   64.2 MiB/s                                   
|
/
/ [0 files][  1.5 GiB/  3.0 GiB]   61.5 MiB/s                                   
-
\
\ [0 files][  1.5 GiB/  3.0 GiB]   58.4 MiB/s                                   
|
| [0 files][  1.6 GiB/  3.0 GiB]   58.5 MiB/s                                   
/
-
- [0 files][  1.7 GiB/  3.0 GiB]   58.4 MiB/s                                   
\
|
| [0 files][  1.7 GiB/  3.0 GiB]   60.2 MiB/s                                   
/
/ [0 files][  1.8 GiB/  3.0 GiB]   63.7 MiB/s                                   
-
\
\ [0 files][  1.9 GiB/  3.0 GiB]   67.8 MiB/s                                   
|
/
/ [0 files][  2.0 GiB/  3.0 GiB]   69.0 MiB/s                                   
-
- [0 files][  2.0 GiB/  3.0 GiB]   71.1 MiB/s                                   
\
|
| [0 files][  2.1 GiB/  3.0 GiB]   72.7 MiB/s                                   
/
-
- [0 files][  2.2 GiB/  3.0 GiB]   70.7 MiB/s                                   
\
\ [0 files][  2.2 GiB/  3.0 GiB]   69.0 MiB/s                                   
|
/
/ [0 files][  2.3 GiB/  3.0 GiB]   64.8 MiB/s                                   
-
\
\ [0 files][  2.3 GiB/  3.0 GiB]   61.6 MiB/s                                   
|
/
/ [0 files][  2.4 GiB/  3.0 GiB]   61.0 MiB/s                                   
-
- [0 files][  2.5 GiB/  3.0 GiB]   62.4 MiB/s                                   
\
|
| [0 files][  2.5 GiB/  3.0 GiB]   61.8 MiB/s                                   
/
-
- [0 files][  2.6 GiB/  3.0 GiB]   66.2 MiB/s                                   
\
\ [0 files][  2.6 GiB/  3.0 GiB]   65.3 MiB/s                                   
|
/
/ [0 files][  2.7 GiB/  3.0 GiB]   64.7 MiB/s                                   
-
\
\ [0 files][  2.8 GiB/  3.0 GiB]   66.2 MiB/s                                   
|
| [0 files][  2.9 GiB/  3.0 GiB]   68.8 MiB/s                                   
/
-
- [0 files][  2.9 GiB/  3.0 GiB]   71.3 MiB/s                                   
\
|
| [0 files][  3.0 GiB/  3.0 GiB]   75.2 MiB/s                                   
Operation completed over 1 objects/3.0 GiB.                                      
| [1 files][  3.0 GiB/  3.0 GiB]   75.2 MiB/s                                   
Copying gs://genomics-public-data/resources/broad/hg38/v0/Homo_sapiens_assembly38.fasta.fai...
| [1 files][  3.0 GiB/  3.0 GiB]   75.2 MiB/s                                   
| [2 files][  3.0 GiB/  3.0 GiB]   73.7 MiB/s                                   
2022/10/12 05:10:25 Localization script execution complete.
2022/10/12 05:10:30 Done localization.
2022/10/12 05:10:31 Running user action: docker run -v /mnt/local-disk:/cromwell_root --entrypoint=/bin/bash us.gcr.io/broad-gatk/gatk@sha256:18146e79d06787483310e5de666502090a480e10ac0fad06a36a5e7a5c9bb1dc /cromwell_root/script
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.1133802e
05:10:35.862 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.1.8.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
05:10:36.130 INFO  GenomicsDBImport - ------------------------------------------------------------
05:10:36.130 INFO  GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.1.8.0
05:10:36.131 INFO  GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
05:10:36.131 INFO  GenomicsDBImport - Executing as root@61f90d3bccac on Linux v5.15.65+ amd64
05:10:36.131 INFO  GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08
05:10:36.131 INFO  GenomicsDBImport - Start Date/Time: October 12, 2022 5:10:35 AM GMT
05:10:36.131 INFO  GenomicsDBImport - ------------------------------------------------------------
05:10:36.131 INFO  GenomicsDBImport - ------------------------------------------------------------
05:10:36.132 INFO  GenomicsDBImport - HTSJDK Version: 2.22.0
05:10:36.132 INFO  GenomicsDBImport - Picard Version: 2.22.8
05:10:36.132 INFO  GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 2
05:10:36.132 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
05:10:36.133 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
05:10:36.133 INFO  GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
05:10:36.133 INFO  GenomicsDBImport - Deflater: IntelDeflater
05:10:36.133 INFO  GenomicsDBImport - Inflater: IntelInflater
05:10:36.133 INFO  GenomicsDBImport - GCS max retries/reopens: 20
05:10:36.133 INFO  GenomicsDBImport - Requester pays: enabled. Billed to: t2t-nhgri
05:10:36.134 INFO  GenomicsDBImport - Initializing engine
05:10:37.825 INFO  FeatureManager - Using codec IntervalListCodec to read file file:///cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-SplitIntervalList/cacheCopy/glob-d928cd0f5fb17b6bd5e635f48c18ccfb/0008-scattered.interval_list
05:10:38.070 INFO  IntervalArgumentCollection - Processing 342683386 bp from intervals
05:10:38.084 INFO  GenomicsDBImport - Done initializing engine
05:10:38.570 INFO  GenomicsDBLibLoader - GenomicsDB native library version : 1.3.0-e701905
05:10:38.570 INFO  GenomicsDBImport - Vid Map JSON file will be written to /cromwell_root/genomicsdb/vidmap.json
05:10:38.570 INFO  GenomicsDBImport - Callset Map JSON file will be written to /cromwell_root/genomicsdb/callset.json
05:10:38.571 INFO  GenomicsDBImport - Complete VCF Header will be written to /cromwell_root/genomicsdb/vcfheader.vcf
05:10:38.571 INFO  GenomicsDBImport - Importing to workspace - /cromwell_root/genomicsdb
05:10:38.571 WARN  GenomicsDBImport - GenomicsDBImport cannot use multiple VCF reader threads for initialization when the number of intervals is greater than 1. Falling back to serial VCF reader initialization.
05:10:38.571 INFO  ProgressMeter - Starting traversal
05:10:38.571 INFO  ProgressMeter -        Current Locus  Elapsed Minutes     Batches Processed   Batches/Minute
05:11:09.822 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
07:04:30.407 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
08:55:18.982 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
10:55:29.471 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
20:03:46.979 INFO  GenomicsDBImport - Importing batch 1 with 50 samples
20:44:12.092 INFO  ProgressMeter -       chr20:29229966            933.6                     1              0.0
20:44:12.092 INFO  GenomicsDBImport - Done importing batch 1/2
20:44:16.834 INFO  GenomicsDBImport - Importing batch 2 with 7 samples
21:58:30.143 INFO  GenomicsDBImport - Importing batch 2 with 7 samples
23:11:16.348 INFO  GenomicsDBImport - Importing batch 2 with 7 samples
00:28:41.364 INFO  GenomicsDBImport - Importing batch 2 with 7 samples
Using GATK jar /gatk/gatk-package-4.1.8.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xms8g -jar /gatk/gatk-package-4.1.8.0-local.jar GenomicsDBImport --genomicsdb-workspace-path genomicsdb --batch-size 50 -L /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-SplitIntervalList/cacheCopy/glob-d928cd0f5fb17b6bd5e635f48c18ccfb/0008-scattered.interval_list --sample-name-map /cromwell_root/fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/1511c824-d248-4d69-86c3-469425952d44/GenerateSampleMap/08e26b9d-d47c-4408-a65c-51321dd20bc5/call-GenerateSampleMapFile/generate-sample-map_2022-10-12T04-05-19.sample_map --reader-threads 5 --merge-input-intervals --consolidate --gcs-project-for-requester-pays t2t-nhgri
2022/10/13 06:36:53 Starting delocalization.
2022/10/13 06:36:54 Delocalization script execution started...
2022/10/13 06:36:54 Delocalizing output /cromwell_root/memory_retry_rc -> gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/memory_retry_rc
2022/10/13 06:36:58 Delocalizing output /cromwell_root/rc -> gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/rc
2022/10/13 06:37:00 Delocalizing output /cromwell_root/stdout -> gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/stdout
2022/10/13 06:37:02 Delocalizing output /cromwell_root/stderr -> gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/stderr
2022/10/13 06:37:04 Delocalizing output /cromwell_root/genomicsdb.tar -> gs://fc-secure-4329a05b-7ebf-4931-86a6-1e3b2a0a9f0e/submissions/a88c50b9-c2b0-4e98-9f8e-8de2964f3308/JointGenotyping/41a5ab73-a203-4bae-9c39-953baed3d6bd/call-ImportGVCFs/shard-8/attempt-2/genomicsdb.tar
Required file output '/cromwell_root/genomicsdb.tar' does not exist.
1

1 comment

Sort by Date Votes
  • Avatar
    Samantha (she/her)

    HI Jason Ni,

    Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with GATK/Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Toggle the "Share with support" button to "Yes"
    2. Click Save

    Please provide us with

    1. A link to your workspace
    2. The relevant submission ID
    3. The relevant workflow ID

    We’ll be happy to take a closer look as soon as we can!

    Kind regards,

    Samantha

    0
    Comment actions Permalink

Please sign in to leave a comment.

Welcome

Forum Guidelines

GATK Announcements

Become a GATK Guru

Read Before You Post

Get Email Notifications

Didn't find what you were looking for?

New post

Quick Links

GATK Latest Releases

Known Issues & Troubleshooting

Submitting a Bug Report

How to Track a Github issue

GATK on the Cloud with Terra

Recent GATK News

GATK Blog Posts

GATK Twitter Page

About the GATK community

  • 133,119 Total Users
  • 0 Comments
  • 0 Posts
  • 0 Votes
Powered by Zendesk