gatk4-genome-processing-pipeline fails for 30x-coverage Wholegenome data on default GCP custom machinetype
Answeredversion: GATK4.
https://github.com/gatk-workflows/gatk4-genome-processing-pipeline runs fine out of the box for the downsampled data. However when we change the dataset to 30x-coveragewholegenome data,https://github.com/priyankasebastian/gatk4-genome-processing-pipeline/blob/30x-original/WholeGenomeGermlineSingleSample.inputs.json , the work flow fails in two tasks.
1. UnmappedBamToAlignedBam.CheckContamination:NA:4 failed. The job was stopped before the command finished. PAPI error code 10. The assigned worker has failed to complete the operation.
We got this issue resolved by bumping up runtime memory to 105.5GB in task GenerateSubsettedContaminationResources. Is this kind of failure expected, or is this the right way to resolve it?
2.BamToCram.ValidateCram fails, we tried couple of workaround with increasing the resources, and changing the preemption. Attaching the metadata for preemptible True/False
2.1.Workflow with preemptible: false :
Metadata:
"message": "Task BamToCram.ValidateCram:NA:1 failed. The job was stopped before the command finished. PAPI error code 10. The assigned worker has failed to complete the operation"
Cromwell log:
2020-08-05 04:59:44,082 cromwell-system-akka.dispatchers.backend-dispatcher-189 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(e76a826b)BamToCram.ValidateCram:NA:1]: Status change from Running to Failed
2020-08-05 04:59:44,859 cromwell-system-akka.dispatchers.backend-dispatcher-189 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(e76a826b)BamToCram.ValidateCram:NA:1]: Job Failed with Error Code 10 for a machine where Preemptible is set to false
2020-08-05 04:59:47,023 cromwell-system-akka.dispatchers.engine-dispatcher-83 INFO - WorkflowManagerActor Workflow 0aaf1c81-4542-42bf-8ce2-4c0b2daa5de2 failed (during ExecutingWorkflowState): java.lang.Exception: Task BamToCram.ValidateCram:NA:1 failed. The job was stopped before the command finished. PAPI error code 10. The assigned worker has failed to complete the operation
2.2Workflow with preemptible: true :
Metadata:
"message": "The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration."
cromwell log:
WorkflowManagerActor Workflow 9ed3ca0b-a816-44e2-9b39-ad9bfe3b7b48 failed (during ExecutingWorkflowState): java.lang.Exception: The compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute instance, or exceeding the backend's maximum job duration
https://github.com/priyankasebastian/gatk4-genome-processing-pipeline/blob/master/metadata-preempt0
https://github.com/priyankasebastian/gatk4-genome-processing-pipeline/blob/master/metadata-preempt3
-
Thanks for writing in. Based on your message, I believe you are running these workflows on your own Cromwell instance rather than on Terra, is that correct?
Even if you are not running on Terra, there is an article about PAPI Error Code 10 that may help: https://support.terra.bio/hc/en-us/articles/360039010292-Error-message-PAPI-error-code-10
If you have any further questions after reading through this, please let us know.
Kind regards,
Jason
-
I get this error below and PAPI is NOT mentioned once in the logs. I have only seen people post this error and in addition with something to do with PAPI. Why do I only get this error? I am running on cromwell-52.
java.lang.Exception: The
compute backend terminated the job. If this termination is unexpected, examine likely causes such as preemption, running out of disk or memory on the compute inst
ance, or exceeding the backend's maximum job duration. -
Hi Brian Wiley, our GATK support team is focused on questions involving GATK issues or abnormal results. You can see our support policy here. For questions regarding Cromwell, we encourage other users to help each other find solutions. You can also look at these resources for more information:
- Bioinformatics Stack Exchange
- Cromwell slack organization: cromwellhq.slack.com
- Cromwell Documentation
Please sign in to leave a comment.
3 comments