erro running "gatk seq-format-validation workflow"
Hello
I would be extremely grateful if you could help me find the problem, I ran the following command inside docker GATK container in a local machine mac pro, and cannot find the " test.validation_.txt" file which is supposed to be the expected output in the execution directories
Thank you in advanced
Best
Can you please provide
a) GATK version used : gatk-4.1.7.0
b) Exact GATK commands used
c) The entire error log if applicable.
(gatk) root@bc3c6aca6231:/gatk/my_data/tools# java -jar cromwell-51.jar run /gatk/my_data/seq-format-validation/validate-bam.wdl --inputs /gatk/my_data/seq-format-validation/validate-bam.inputs.json
[2020-07-14 05:09:22,78] [info] Running with database db.url = jdbc:hsqldb:mem:f10b64bd-d8ca-4428-917b-311fca24c372;shutdown=false;hsqldb.tx=mvcc
[2020-07-14 05:09:29,36] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2020-07-14 05:09:29,37] [info] [RenameWorkflowOptionsInMetadata] 100%
[2020-07-14 05:09:29,47] [info] Running with database db.url = jdbc:hsqldb:mem:e337a356-2f0c-4389-92c5-255465180f24;shutdown=false;hsqldb.tx=mvcc
[2020-07-14 05:09:29,89] [info] Slf4jLogger started
[2020-07-14 05:09:30,10] [info] Workflow heartbeat configuration:
{
"cromwellId" : "cromid-ca5c695",
"heartbeatInterval" : "2 minutes",
"ttl" : "10 minutes",
"failureShutdownDuration" : "5 minutes",
"writeBatchSize" : 10000,
"writeThreshold" : 10000
}
[2020-07-14 05:09:30,23] [info] Metadata summary refreshing every 1 second.
[2020-07-14 05:09:30,23] [warn] 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
[2020-07-14 05:09:30,25] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2020-07-14 05:09:30,26] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2020-07-14 05:09:30,26] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2020-07-14 05:09:30,36] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2020-07-14 05:09:30,46] [info] SingleWorkflowRunnerActor: Version 51
[2020-07-14 05:09:30,48] [info] SingleWorkflowRunnerActor: Submitting workflow
[2020-07-14 05:09:30,55] [info] Unspecified type (Unspecified version) workflow 968be82c-eef3-4bdb-a1ab-3d4e2ca70674 submitted
[2020-07-14 05:09:30,66] [info] SingleWorkflowRunnerActor: Workflow submitted 968be82c-eef3-4bdb-a1ab-3d4e2ca70674
[2020-07-14 05:09:30,67] [info] 1 new workflows fetched by cromid-ca5c695: 968be82c-eef3-4bdb-a1ab-3d4e2ca70674
[2020-07-14 05:09:30,68] [info] WorkflowManagerActor Starting workflow 968be82c-eef3-4bdb-a1ab-3d4e2ca70674
[2020-07-14 05:09:30,69] [info] WorkflowManagerActor Successfully started WorkflowActor-968be82c-eef3-4bdb-a1ab-3d4e2ca70674
[2020-07-14 05:09:30,69] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2020-07-14 05:09:30,72] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2020-07-14 05:09:30,83] [info] MaterializeWorkflowDescriptorActor [968be82c]: Parsing workflow as WDL 1.0
[2020-07-14 05:09:31,60] [info] MaterializeWorkflowDescriptorActor [968be82c]: Call-to-Backend assignments: ValidateBamsWf.ValidateBAM -> Local
[2020-07-14 05:09:31,82] [warn] Local [968be82c]: Key/s [memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2020-07-14 05:09:35,38] [info] Not triggering log of token queue status. Effective log interval = None
[2020-07-14 05:09:37,15] [info] WorkflowExecutionActor-968be82c-eef3-4bdb-a1ab-3d4e2ca70674 [968be82c]: Starting ValidateBamsWf.ValidateBAM
[2020-07-14 05:09:37,39] [info] Assigned new job execution tokens to the following groups: 968be82c: 1
[2020-07-14 05:09:41,61] [warn] BackgroundConfigAsyncJobExecutionActor [968be82cValidateBamsWf.ValidateBAM:0:1]: Unrecognized runtime attribute keys: disks, memory
[2020-07-14 05:09:41,71] [info] BackgroundConfigAsyncJobExecutionActor [968be82cValidateBamsWf.ValidateBAM:0:1]: /gatk/gatk \
ValidateSamFile \
--INPUT /cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/inputs/-1942028726/test.bam \
--OUTPUT test.validation_.txt \
--MODE SUMMARY
[2020-07-14 05:09:41,76] [info] BackgroundConfigAsyncJobExecutionActor [968be82cValidateBamsWf.ValidateBAM:0:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
--cidfile /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/docker_cid \
-i \
\
--entrypoint /bin/bash \
-v /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0:/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0:delegated \
broadinstitute/gatk@sha256:18146e79d06787483310e5de666502090a480e10ac0fad06a36a5e7a5c9bb1dc /cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/script
# get the return code (working even if the container was detached)
rc=$(docker wait cat /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/docker_cid)
# remove the container after waiting
docker rm cat /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/docker_cid
# return exit code
exit $rc
[2020-07-14 05:09:45,29] [info] BackgroundConfigAsyncJobExecutionActor [968be82cValidateBamsWf.ValidateBAM:0:1]: job id: 243
[2020-07-14 05:09:45,29] [info] BackgroundConfigAsyncJobExecutionActor [968be82cValidateBamsWf.ValidateBAM:0:1]: Status change from - to Done
[2020-07-14 05:09:46,38] [info] WorkflowManagerActor Workflow 968be82c-eef3-4bdb-a1ab-3d4e2ca70674 failed (during ExecutingWorkflowState): Job ValidateBamsWf.ValidateBAM:0:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/stderr.
Could not retrieve content: Could not read from /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/stderr: /gatk/my_data/tools/cromwell-executions/ValidateBamsWf/968be82c-eef3-4bdb-a1ab-3d4e2ca70674/call-ValidateBAM/shard-0/execution/stderr
[2020-07-14 05:09:46,38] [info] WorkflowManagerActor WorkflowActor-968be82c-eef3-4bdb-a1ab-3d4e2ca70674 is in a terminal state: WorkflowFailedState
[2020-07-14 05:09:51,97] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2020-07-14 05:09:55,28] [info] Workflow polling stopped
[2020-07-14 05:09:55,30] [info] 0 workflows released by cromid-ca5c695
[2020-07-14 05:09:55,30] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2020-07-14 05:09:55,30] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2020-07-14 05:09:55,30] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2020-07-14 05:09:55,31] [info] JobExecutionTokenDispenser stopped
[2020-07-14 05:09:55,31] [info] Aborting all running workflows.
[2020-07-14 05:09:55,31] [info] WorkflowStoreActor stopped
[2020-07-14 05:09:55,31] [info] WorkflowLogCopyRouter stopped
[2020-07-14 05:09:55,31] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2020-07-14 05:09:55,32] [info] WorkflowManagerActor All workflows finished
[2020-07-14 05:09:55,32] [info] WorkflowManagerActor stopped
[2020-07-14 05:09:55,53] [info] Connection pools shut down
[2020-07-14 05:09:55,53] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2020-07-14 05:09:55,53] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2020-07-14 05:09:55,53] [info] SubWorkflowStoreActor stopped
[2020-07-14 05:09:55,54] [info] JobStoreActor stopped
[2020-07-14 05:09:55,53] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2020-07-14 05:09:55,54] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2020-07-14 05:09:55,54] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2020-07-14 05:09:55,54] [info] CallCacheWriteActor stopped
[2020-07-14 05:09:55,54] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2020-07-14 05:09:55,54] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2020-07-14 05:09:55,54] [info] IoProxy stopped
[2020-07-14 05:09:55,54] [info] DockerHashActor stopped
[2020-07-14 05:09:55,55] [info] Shutting down connection pool: curAllocated=1 idleQueues.size=1 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2020-07-14 05:09:55,55] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2020-07-14 05:09:55,55] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2020-07-14 05:09:55,55] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2020-07-14 05:09:55,55] [info] KvWriteActor Shutting down: 0 queued messages to process
[2020-07-14 05:09:55,55] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2020-07-14 05:09:55,55] [info] ServiceRegistryActor stopped
[2020-07-14 05:09:55,58] [info] Database closed
[2020-07-14 05:09:55,58] [info] Stream materializer shut down
[2020-07-14 05:09:55,58] [info] WDL HTTP import resolver closed
Workflow 968be82c-eef3-4bdb-a1ab-3d4e2ca70674 transitioned to state Failed
-
Hi zdr j,
Happy to see if I'm able to help. Do you have any .log files for the job you can share? stdout and stderr files may help as well.
Can you also share your validate-bam.wdl and validate-bam.inputs.json files? If you got the WDL and json files from somewhere, please point me to their origin(s).
Kind regards,
Jason
-
Thanks a lot Jason Cerrato,
below is the stderr, and in the stdout it just says 230. I attached the image of validate-bam.wdl and validate-bam.inputs.json files. I appreciate.
Best
Zara
-
Hi zdr j,
Thank you for those details. Would you be able to share the .log, WDL, and inputs.json as downloadable files? I recommend Dropbox, Google Drive, or public Google bucket files. Downloadable files will be much easier to use for troubleshooting purposes.
Many thanks,
Jason
-
Hello
Thanks for you help
I tried many things with the json format and wdl, but the problem still exists. The link below has the files and folders for json, wdl and the input file I am using.
Thank you so much
https://drive.google.com/drive/folders/10CJNhgegAnu1Sp-DQBw11YK5qVZrV5Ys?usp=sharing
-
Hi zdr j,
Happy to continue assisting. I see the cromwell-workflow-logs folder is empty. Were logs not generated from your recent run?
I also see that your inputs.json file does not include disk_size and mem_size. I'm curious—are you following the the steps in this article? https://gatk.broadinstitute.org/hc/en-us/articles/360035530952--How-to-Execute-Workflows-from-the-gatk-workflows-Git-Organization
I'm curious to know if it will provide you with some helpful insight for your case.
Kind regards,
Jason
-
Thank you for your reply. Yes, the log folder is empty after the execution. And for the json file I tried with different formats and none of them worked. I added another json to the shared google drive link below. And yes, I am following exactly the guidelines in the article you mentioned.
I am thinking maybe the problem is my input file? can you please share a tested sample bam file here so that I check with that. And I have also shared my input bam.file, in case you prefer to test it on your machine.
I am new at GATK, and I would appreciate if you could help me figure out what is this problem. I am frustrated with it not working.
Thank you so much
Best
https://drive.google.com/drive/folders/1MtWN6o6ZIdLmy67xSXvP7fU3To1A6bGS
-
Hi zdr j,
Happy to see if we can get you unstuck here.
Can you revise your inputs.json file to have the full path for the bam rather than the relative path, similar to how its shown in the link? You can also test with a different bam by going to the bucket as described in the link: https://console.cloud.google.com/storage/browser/gatk-test-data/wgs_bam?organizationId=548622027621&project=broad-dsde-outreach&prefix=
I recommend selecting a small bam, like the NA12878_24RG_small.hg38.bam.
Please provide the full output of the run if it fails to work again. We may want to run through the steps from scratch if so to make sure we're not missing anything.
Kind regards,
Jason
-
Thanks for your guidance. I changed the path in the Json file as you mentioned, unfortunately the workflow still fails. Please see files in the folder named" error gatk-workflows new" in the following link for details. I also tried with NA12878_24RG_small.hg38.bam, and did not resolve the issue meaning that most probably my input was fine. It seems the problem is json format or cromwell? what do you think? And also is debugging the json related to this at all? should I run in debug mode?
Thank you so much
Best
https://drive.google.com/drive/folders/1VrJWUNtOsWkyNibQ6hIahAfVy7GQafAN
-
Hi zdr j,
I'm working through the steps myself and I've identified a few issues with the current documentation. I'll let you know once I get a working configuration on my end.
Kind regards,
Jason
-
Hi zdr j,
I took a look at your files and I see that you actually did get a stderr, but it was written as stderr.background. See: https://drive.google.com/file/d/1f4N_EVNgAkWdaJmcHxYiWU5rWu_ln0ny/view?usp=sharing
The contents are as follows:
/gatk/my_data/cromwell-executions/ValidateBamsWf/d5757999-1bef-4ce4-a316-77b3ddcf84e5/call-ValidateBAM/shard-0/execution/script.submit: line 5: docker: command not found
cat: /gatk/my_data/cromwell-executions/ValidateBamsWf/d5757999-1bef-4ce4-a316-77b3ddcf84e5/call-ValidateBAM/shard-0/execution/docker_cid: No such file or directory
/gatk/my_data/cromwell-executions/ValidateBamsWf/d5757999-1bef-4ce4-a316-77b3ddcf84e5/call-ValidateBAM/shard-0/execution/script.submit: line 14: docker: command not found
cat: /gatk/my_data/cromwell-executions/ValidateBamsWf/d5757999-1bef-4ce4-a316-77b3ddcf84e5/call-ValidateBAM/shard-0/execution/docker_cid: No such file or directory
/gatk/my_data/cromwell-executions/ValidateBamsWf/d5757999-1bef-4ce4-a316-77b3ddcf84e5/call-ValidateBAM/shard-0/execution/script.submit: line 17: docker: command not foundCan you confirm you have docker installed and try again?
Kind regards,
Jason
-
Thank you so much for your time. Yes, I would appreciate of you could try the workflow on your machine. Honestly, I think there is something missing in the instructions for local machine in the paper. Yes, I do have docker latest version installed and that is how I pulled the GATK image. However, one problem might be the fact that docker cannot be run when inside a docker container; when in the terminal and inside home directory I ask for "docker --version", there is an answer :"Docker version 19.03.12, build 48a66213fe". But when I change directory to gatk image or to the gatk-workflows directory, it says : " docker command not found". But I heard it is not a good idea to install docker inside a docker image. After the image is pulled by docker, do we still need docker when running the workflow? I look forward to your reply on this matter. Thank you so much
Best
zdrj
-
Hi zdr j,
I was able to successfully run this locally. You do not need to run this inside a docker to run it locally—the job itself will use the docker.
Can you try this again running following the directions in the Running Workflows Locally section? For your inputs.json, please edit it to look like the following:
{ "ValidateBamsWf.bam_array": [ "inputs/4.bam"] }
You can use your original bam, or the sample bam. Try with the full path if this doesn't work.
If you don't already have the cromwell jar on your local machine you can simply download it and move it to the desired directory if your computer doesn't have wget.
I believe the Google Cloud version should also work without being inside the GATK docker, if that's what you prefer to do.
Let me know if this works for you!
Kind regards,
Jason
-
Hi zdr j,
This should also work with the addition of
"ValidateBamsWf.ValidateBAM.validation_mode": "SUMMARY"
in the json if you are interested in using the SUMMARY mode.
Jason
-
Thank you so much Jason Cerrato
When I edited the json file, it says failed state again at the end. But this time in the output directory, the output file exists and says:
## HISTOGRAM java.lang.String
Error Type Count
ERROR:MATE_NOT_FOUND 73Does this mean that the workflow worked and this is an error the workflow found in my bam file? :)))))) The output directory contains these files now:
- 4.validation_.txt
- docker_cid
- rc
- script
- script.background
- script.submit
- stderr
- stderr.background
- stdout
- stdout.background
I shared stdout and stderr in the link below.
Thank you so much
Best
https://drive.google.com/drive/folders/1KnAKSJ2qyRG-w1xeH8KVZFPc_RGUjvAg
-
Hi zdr j,
Yes, the contents of the validation_.txt file would reveal any issues with the provided bam. You can confirm by testing with one of the known valid sample bams to see if you get a different result.
Kind regards,
Jason
-
Yes, it seems working although says failed state at the end. Thank you so much. I really really appreciate your time and grate help.
Best Regards,
zdrj
-
I'm very glad to help! I'll reach out to our User Education team to make the appropriate changes in our documentation to make it clearer. Thank you for writing in and for helping us improve our documentation!
Jason
-
Hello guys, I have followed your debate because something similar is happening to me but I have not understood what was the final solution to the problem, could you please provide us with information on how you solved your problem,
Greetings Jose
-
Problem solved with the help of Beri Shifaw, thanks for everything.
Kind regards,
Jose
-
Hello guys,
Could you please provide us with information on how you solved your problem ? I have the same problem and this type of error:
[2022-03-22 13:38:43,22] [error] WorkflowManagerActor Workflow 1754f8db-da10-440c-bd04-591832dd8e44 failed (during ExecutingWorkflowState): Job ValidateBamsWf.ValidateBAM:0:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /gatk/cromwell-executions/ValidateBamsWf/1754f8db-da10-440c-bd04-591832dd8e44/call-ValidateBAM/shard-0/execution/stderr.
Could not retrieve content: /gatk/cromwell-executions/ValidateBamsWf/1754f8db-da10-440c-bd04-591832dd8e44/call-ValidateBAM/shard-0/execution/stderrThanks in advance
Tom
-
Hi Tomasz .P,
Can you please check the contents of the "/gatk/cromwell-executions/ValidateBamsWf/1754f8db-da10-440c-bd04-591832dd8e44/call-ValidateBAM/shard-0/execution/stderr" file. This file should provide more information about why this task failed.
Kind regards,
Pamela
Please sign in to leave a comment.
21 comments