Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

error in mitochondria pipeline

Follow
Avatar
man Sac

Hello. I am currently trying to detect mutations and estimate haplogroups using mitochondrial DNA reads sequenced with MiSeq.

I prepared my data like this.

bwa index human_ref/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta

bwa mem -Y human_ref/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta read1.fastq.gz read2.fastq.gz > mt-aln-pe.sam

samtools view -bS mt-aln-pe.sam > mt-aln-pe.bam

samtools sort -o mt-aln-pe.sorted.bam mt-aln-pe.bam

samtools index mt-aln-pe.sorted.bam

a) GATK version used: 4.5.0.0

b) Exact command used:

java -jar /home/User/cromwell/cromwell-87.jar run /home/User/mitochondria_m2/MitochondriaPipeline.wdl --inputs ExampleInputsMitochondriaPipeline.json

 

c) Entire program log:

[2024-05-19 11:42:28,59] [info] Running with database db.url = jdbc:hsqldb:mem:d88c929e-8a0d-4315-a2fd-98ff52c7cb42;shutdown=false;hsqldb.tx=mvcc
[2024-05-19 11:42:31,05] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2024-05-19 11:42:31,07] [info] [RenameWorkflowOptionsInMetadata] 100%
[2024-05-19 11:42:31,44] [info] Running with database db.url = jdbc:hsqldb:mem:42c17c5b-ba0c-4d66-b106-63d6902970cf;shutdown=false;hsqldb.tx=mvcc
[2024-05-19 11:42:31,88] [info] Slf4jLogger started
[2024-05-19 11:42:32,09] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-bd0d4db",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "failureShutdownDuration" : "5 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2024-05-19 11:42:32,13] [info] Metadata summary refreshing every 1 second.
[2024-05-19 11:42:32,13] [info] No metadata archiver defined in config
[2024-05-19 11:42:32,13] [info] No metadata deleter defined in config
[2024-05-19 11:42:32,14] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2024-05-19 11:42:32,14] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2024-05-19 11:42:32,15] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2024-05-19 11:42:32,40] [info] JobRestartCheckTokenDispenser - Distribution rate: 50 per 1 seconds.
[2024-05-19 11:42:32,41] [info] JobExecutionTokenDispenser - Distribution rate: 20 per 10 seconds.
[2024-05-19 11:42:32,44] [info] SingleWorkflowRunnerActor: Version 87
[2024-05-19 11:42:32,45] [info] SingleWorkflowRunnerActor: Submitting workflow
[2024-05-19 11:42:32,47] [info] Unspecified type (Unspecified version) workflow f575b192-8041-4592-9a51-c19217f02662 submitted
[2024-05-19 11:42:32,49] [info] SingleWorkflowRunnerActor: Workflow submitted [38;5;2mf575b192-8041-4592-9a51-c19217f02662[0m
[2024-05-19 11:42:32,49] [info] 1 new workflows fetched by cromid-bd0d4db: f575b192-8041-4592-9a51-c19217f02662
[2024-05-19 11:42:32,50] [info] WorkflowManagerActor: Starting workflow [38;5;2mf575b192-8041-4592-9a51-c19217f02662[0m
[2024-05-19 11:42:32,50] [info] WorkflowManagerActor: Successfully started WorkflowActor-f575b192-8041-4592-9a51-c19217f02662
[2024-05-19 11:42:32,50] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2024-05-19 11:42:32,52] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2024-05-19 11:42:32,62] [info] MaterializeWorkflowDescriptorActor [[38;5;2mf575b192[0m]: Parsing workflow as WDL 1.0
[2024-05-19 11:42:33,42] [info] MaterializeWorkflowDescriptorActor [[38;5;2mf575b192[0m]: Call-to-Backend assignments: MitochondriaPipeline.SubsetBamToChrM -> Local, AlignAndCall.MergeStats -> Local, MitochondriaPipeline.CoverageAtEveryBase -> Local, AlignAndCall.CollectWgsMetrics -> Local, AlignAndCall.CallMt -> Local, AlignmentPipeline.AlignAndMarkDuplicates -> Local, AlignmentPipeline.GetBwaVersion -> Local, AlignAndCall.FilterContamination -> Local, AlignAndCall.LiftoverAndCombineVcfs -> Local, AlignAndCall.GetContamination -> Local, AlignAndCall.InitialFilter -> Local, MitochondriaPipeline.SplitMultiAllelicSites -> Local, AlignAndCall.CallShiftedMt -> Local, AlignAndCall.SplitMultiAllelicsAndRemoveNonPassSites -> Local, MitochondriaPipeline.RevertSam -> Local
[2024-05-19 11:42:33,59] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [memory, disks, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,59] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [memory, disks, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,59] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [disks, memory, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,59] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,59] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [disks, memory, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, memory, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [memory, disks, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [preemptible, disks, cpu, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [memory, disks, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:33,60] [[38;5;220mwarn[0m] Local [[38;5;2mf575b192[0m]: Key/s [disks, memory, preemptible] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2024-05-19 11:42:35,72] [info] WorkflowExecutionActor-f575b192-8041-4592-9a51-c19217f02662 [[38;5;2mf575b192[0m]: Starting MitochondriaPipeline.SubsetBamToChrM
[2024-05-19 11:42:37,42] [info] Not triggering log of restart checking token queue status. Effective log interval = None
[2024-05-19 11:42:37,43] [info] Not triggering log of execution token queue status. Effective log interval = None
[2024-05-19 11:42:42,43] [info] Assigned new job execution tokens to the following groups: f575b192: 1
[2024-05-19 11:42:44,22] [[38;5;220mwarn[0m] BackgroundConfigAsyncJobExecutionActor [[38;5;2mf575b192[0mMitochondriaPipeline.SubsetBamToChrM:NA:1]: Unrecognized runtime attribute keys: preemptible, disks, memory
[2024-05-19 11:42:44,32] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2mf575b192[0mMitochondriaPipeline.SubsetBamToChrM:NA:1]: [38;5;5mset -e
export GATK_LOCAL_JAR=/root/gatk.jar

gatk PrintReads \
  -R /cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/inputs/-730740457/hg38_v0_Homo_sapiens_assembly38.fasta \
  -L chrM \
  --read-filter MateOnSameContigOrNoMappedMateReadFilter \
  --read-filter MateUnmappedAndUnmappedReadFilter \
   \
  -I /cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/inputs/-1376513808/mt-aln-pe.sorted.bam \
  --read-index /cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/inputs/-1376513808/mt-aln-pe.sorted.bam.bai \
  -O mt-aln-pe.sorted.bam[0m
[2024-05-19 11:42:44,35] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2mf575b192[0mMitochondriaPipeline.SubsetBamToChrM:NA:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM:/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM:delegated \
  us.gcr.io/broad-gatk/gatk@sha256:192fedf9b9d65809da4a2954030269e3f311d296e6b5e4c6c7deec12c7fe84b2 /cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait [38;5;5mcat /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid[0m)

# remove the container after waiting
docker rm [38;5;5mcat /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid[0m

# return exit code
exit $rc
[2024-05-19 11:42:47,17] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2mf575b192[0mMitochondriaPipeline.SubsetBamToChrM:NA:1]: job id: 480409
[2024-05-19 11:42:47,17] [info] BackgroundConfigAsyncJobExecutionActor [[38;5;2mf575b192[0mMitochondriaPipeline.SubsetBamToChrM:NA:1]: Status change from - to Done
[2024-05-19 11:42:48,01] [info] WorkflowManagerActor: Workflow f575b192-8041-4592-9a51-c19217f02662 failed (during ExecutingWorkflowState): Job MitochondriaPipeline.SubsetBamToChrM:NA:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/stderr.
 Could not retrieve content: Could not read from /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/stderr: /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/stderr
[2024-05-19 11:42:52,17] [info] WorkflowManagerActor: Workflow actor for f575b192-8041-4592-9a51-c19217f02662 completed with status 'Failed'. The workflow will be removed from the workflow store.
[2024-05-19 11:42:58,31] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2024-05-19 11:43:02,16] [info] Workflow polling stopped
[2024-05-19 11:43:02,16] [info] 0 workflows released by cromid-bd0d4db
[2024-05-19 11:43:02,17] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2024-05-19 11:43:02,17] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2024-05-19 11:43:02,17] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2024-05-19 11:43:02,17] [info] JobExecutionTokenDispenser stopped
[2024-05-19 11:43:02,17] [info] Aborting all running workflows.
[2024-05-19 11:43:02,17] [info] WorkflowStoreActor stopped
[2024-05-19 11:43:02,17] [info] WorkflowLogCopyRouter stopped
[2024-05-19 11:43:02,17] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2024-05-19 11:43:02,17] [info] WorkflowManagerActor: All workflows finished
[2024-05-19 11:43:02,17] [info] WorkflowManagerActor stopped
[2024-05-19 11:43:02,40] [info] Connection pools shut down
[2024-05-19 11:43:02,40] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] SubWorkflowStoreActor stopped
[2024-05-19 11:43:02,40] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2024-05-19 11:43:02,40] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2024-05-19 11:43:02,40] [info] JobStoreActor stopped
[2024-05-19 11:43:02,40] [info] CallCacheWriteActor stopped
[2024-05-19 11:43:02,40] [info] KvWriteActor Shutting down: 0 queued messages to process
[2024-05-19 11:43:02,40] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2024-05-19 11:43:02,40] [info] IoProxy stopped
[2024-05-19 11:43:02,40] [info] DockerHashActor stopped
[2024-05-19 11:43:02,41] [info] ServiceRegistryActor stopped
[2024-05-19 11:43:02,41] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2024-05-19 11:43:02,41] [info] Shutting down connection pool: curAllocated=1 idleQueues.size=1 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2024-05-19 11:43:02,41] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2024-05-19 11:43:02,41] [info] Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
[2024-05-19 11:43:02,41] [info] Database closed
[2024-05-19 11:43:02,41] [info] Stream materializer shut down
[2024-05-19 11:43:02,41] [info] WDL HTTP import resolver closed


error log

Workflow f575b192-8041-4592-9a51-c19217f02662 transitioned to state Failed

 

/lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/stderr

is

docker: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create": dial unix /var/run/docker.sock: connect: permission denied.
See 'docker run --help'.
cat: /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid: No such file or directory
"docker wait" requires at least 1 argument.
See 'docker wait --help'.

Usage:  docker wait CONTAINER [CONTAINER...]

Block until one or more containers stop, then print their exit codes
cat: /lustre7/home/User/MTDNA-prospective/gatkmt/mt/cromwell-executions/MitochondriaPipeline/f575b192-8041-4592-9a51-c19217f02662/call-SubsetBamToChrM/execution/docker_cid: No such file or directory
"docker rm" requires at least 1 argument.
See 'docker rm --help'.

Usage:  docker rm [OPTIONS] CONTAINER [CONTAINER...]

Remove one or more containers

I can also provide input files. Could you please give me some advice?

 

Best regards, 

0

6 comments

Sort by Date Votes
  • Avatar
    Gökalp Çelik

    Hi

    Looks like your user account does not have permission to use the docker socket therefore script cannot execute. If you are using an infrastructure then you may need to get help from your local IT or you may modify the wdl file to use local GATK installation instead of docker images. This may take a few experimentation on your end since your paths will be your own. 

    Otherwise if you have a chance to add your user to docker usergroup then you should be able to call into docker socket therefore this script will run properly as is. 

    Regards. 

    1
    Comment actions Permalink
  • Avatar
    man Sac

    Thank you for your advice.

    As you mentioned, it seems that Docker is not allowed in my analysis environment. Singularity is available, but as indicated in other questions, this issue is not supported by GATK. I will consider using a different analysis environment or using GATK locally.

    Best regards.

    0
    Comment actions Permalink
  • Avatar
    Gökalp Çelik

    Mitochondria Variant calling workflow does not use too many resources and should be fairly easy to run on local environments. 

    1
    Comment actions Permalink
  • Avatar
    man Sac

    By "local environment," do you mean a local machine like a Mac rather than an HPC? Is it easy to run using Docker on a local machine?

    0
    Comment actions Permalink
  • Avatar
    Gökalp Çelik

    You can run it on any POSIX compatible environment including Intel Mac or Linux machines.

    If you have an Apple Silicon Mac, our intel gkl dependencies are not aarch64 compatible therefore you may experience unexpected behavior in some cases. 

     

    1
    Comment actions Permalink
  • Avatar
    man Sac

    Thank you very much.

    I appreciate the thorough support.

    0
    Comment actions Permalink

Please sign in to leave a comment.

Welcome

Forum Guidelines

GATK Announcements

Become a GATK Guru

Read Before You Post

Get Email Notifications

Didn't find what you were looking for?

New post

Quick Links

GATK Latest Releases

Known Issues & Troubleshooting

Submitting a Bug Report

How to Track a Github issue

GATK on the Cloud with Terra

Recent GATK News

GATK Blog Posts

GATK Twitter Page

About the GATK community

  • 133,119 Total Users
  • 0 Comments
  • 0 Posts
  • 0 Votes
Powered by Zendesk