Cromwell stuck at WaitingForReturnCode on Mac
I am using Cromwell version 55 on a MacBook Pro running OS 10.15.7 (Catalina). I thought I'd learn to use Cromwell, so I started with the 5-minute introduction tutorial found here:
https://cromwell.readthedocs.io/en/stable/tutorials/FiveMinuteIntro/
Unfortunately it isn't working quite as advertised. I followed the instructions exactly, and it does seem to be working in the sense that it produces a directory structure, and a few levels down in that directory structure there is a file "stdout" that contains the words "hello world". But in the terminal window, I never get my command line back. The following is the complete output:
[2021-02-01 12:33:10,09] [info] Running with database db.url = jdbc:hsqldb:mem:4b62140f-ee89-4787-b6e5-7d025fe17d5d;shutdown=false;hsqldb.tx=mvcc
[2021-02-01 12:33:21,26] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2021-02-01 12:33:21,29] [info] [RenameWorkflowOptionsInMetadata] 100%
[2021-02-01 12:33:21,52] [info] Running with database db.url = jdbc:hsqldb:mem:fe4f4034-7ee8-427b-9719-579ee2f6618f;shutdown=false;hsqldb.tx=mvcc
[2021-02-01 12:33:22,24] [info] Slf4jLogger started
[2021-02-01 12:33:22,55] [info] Workflow heartbeat configuration:
{
"cromwellId" : "cromid-af22feb",
"heartbeatInterval" : "2 minutes",
"ttl" : "10 minutes",
"failureShutdownDuration" : "5 minutes",
"writeBatchSize" : 10000,
"writeThreshold" : 10000
}
[2021-02-01 12:33:22,60] [info] Metadata summary refreshing every 1 second.
[2021-02-01 12:33:22,62] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2021-02-01 12:33:22,62] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2021-02-01 12:33:22,72] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2021-02-01 12:33:22,72] [warn] 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
[2021-02-01 12:33:22,86] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2021-02-01 12:33:22,92] [info] SingleWorkflowRunnerActor: Version 55
[2021-02-01 12:33:22,94] [info] SingleWorkflowRunnerActor: Submitting workflow
[2021-02-01 12:33:23,00] [info] Unspecified type (Unspecified version) workflow b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a submitted
[2021-02-01 12:33:23,04] [info] SingleWorkflowRunnerActor: Workflow submitted b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a
[2021-02-01 12:33:23,05] [info] 1 new workflows fetched by cromid-af22feb: b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a
[2021-02-01 12:33:23,07] [info] WorkflowManagerActor Starting workflow b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a
[2021-02-01 12:33:23,09] [info] WorkflowManagerActor Successfully started WorkflowActor-b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a
[2021-02-01 12:33:23,09] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2021-02-01 12:33:23,11] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2021-02-01 12:33:23,29] [info] MaterializeWorkflowDescriptorActor [b9e4ed10]: Parsing workflow as WDL draft-2
[2021-02-01 12:33:24,75] [info] MaterializeWorkflowDescriptorActor [b9e4ed10]: Call-to-Backend assignments: myWorkflow.myTask -> Local
[2021-02-01 12:33:26,12] [info] WorkflowExecutionActor-b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a [b9e4ed10]: Starting myWorkflow.myTask
[2021-02-01 12:33:26,89] [info] Assigned new job execution tokens to the following groups: b9e4ed10: 1
[2021-02-01 12:33:27,10] [info] BackgroundConfigAsyncJobExecutionActor [b9e4ed10myWorkflow.myTask:NA:1]: echo "hello world"
[2021-02-01 12:33:27,21] [info] BackgroundConfigAsyncJobExecutionActor [b9e4ed10myWorkflow.myTask:NA:1]: executing: /bin/bash /Users/mcrepeau/working/cromwell/cromwell-executions/myWorkflow/b9e4ed10-1e46-4b1f-b5b4-9075fa736f3a/call-myTask/execution/script
[2021-02-01 12:33:27,88] [info] Not triggering log of token queue status. Effective log interval = None
[2021-02-01 12:33:32,65] [info] BackgroundConfigAsyncJobExecutionActor [b9e4ed10myWorkflow.myTask:NA:1]: job id: 31514
[2021-02-01 12:33:32,70] [info] BackgroundConfigAsyncJobExecutionActor [b9e4ed10myWorkflow.myTask:NA:1]: Status change from - to WaitingForReturnCode
At that point it stalls forever. Using top I can see the bash PID 31514 still present with state "sleeping", plus a java PID 31511, also with state sleeping. As far as my java -version, the output I get is:
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode
I tried the same commands on our Linux server, and they finished and exited properly. Just not working on my Mac.
-
Hi , our GATK support team is focused on questions involving GATK issues or abnormal results. You can see our support policy here. For questions regarding Cromwell, we encourage other users to help each other find solutions. You can also look at these resources for more information:
- Bioinformatics Stack Exchange
- Cromwell slack organization: cromwellhq.slack.com
- Cromwell Documentation
-
Okay, this was my bad. The problem was, in fact, the java version. It worked fine when I switched to java version 1.8. To roll my Mac back to that version I used the excellent instructions here:
-
Glad you were able to solve your issue! Thank you for the update Marc Crepeau, I am sure other users will find this helpful.
Please sign in to leave a comment.
3 comments