Mutect2 Funcotator error
-
Hi Mia,
I'll be happy to take a look at this. Is this workspace where you ran the workflow currently shared with GROUP_FireCloud-Support@firecloud.org? If not, can you share the workspace where you are seeing this issue with GROUP_FireCloud-Support@firecloud.org by clicking the Share button in your workspace (see the icon with the three dots at the top-right)?
1. Add GROUP_FireCloud-Support@firecloud.org to the User email field and press Enter on your keyboard
2. Click SaveLet us know the name of the workspace, or a share a link to the workspace, along with the associated submission ID.
Many thanks,
Jason
-
Hi Jason,
That sounds great. Exactly, it is the same workspace that is in question called 661-Clonal hematopoiesis.
Submission ID 20aa1262-1a68-4cbe-9a26-abdff0c672aa
Thanks a million,
Mia
-
Hi Mia,
Is there any particular reason you are using GATK version 4.1.2.0 for your submission rather than the latest 4.1.6.0? I don't know that this is the reason it failed, but I'm curious to know if there's a particular reason to use this version.
Kind regards,
Jason
-
Hi Jason,
I am using 4.1.6.0 (see the link in my first post) and this is the version from which I get an error:
github.com/broadinstitute/gatk/mutect2:4.1.6.0
Not sure why you think I was using 4.1.2.0?
Thanks,
Mia
-
Hi Mia,
I see in your method configuration that gatk:4.1.2.0 was provided as the gatk_docker. This also shows as the input for the job you provided.
I'm curious to know whether using an old version of GATK as the docker has an effect on the end result.
Kind regards,
Jason
-
Hi Jason,
Thanks for clarifying ! I forgot I linked this as it has now been a while - I did not know it would affect results and I did not know how to source a docker for the new workflow.
For now, should I change:
"broadinstitute/gatk:4.1.2.0"
to:
"broadinstitute/gatk:4.1.6.0"
For future:
Are there instructions specific to Terra workflows on how to source required arguments for json files? It seems that many of the Mutect ones are standard and that many are available internally at the Broad, but I am never sure how to find links to those?
Many thanks,
Mia
-
Hi Mia,
I would recommend changing it to gatk:4.1.6.0, yes. As previously mentioned, I don't know for certain whether this is the reason it failed, but it looks like there have been a few changes to Mutect2 since 4.1.2.0: https://github.com/broadinstitute/gatk/releases
Can you provide some additional details for what you mean by "how to source required arguments for json files?"
Kind regards,
Jason -
Thanks Jason, I will try this and update.
What I meant was how can I identify the files that are required to run various workflows ?
When I import an individual workflow, input json is empty and needs to be filled. I always struggle to identify the required files. Github often specifies the type of the file needed, but not also where that file is and how can I source the correct file that matches the workflow version. The way I currently do it is by digging around Terra to try to find exemplar workflows where input json has been pre-filled. However, I cannot often find such pre-filled workflows for all versions and so I struggle to know how to fill in jsons.
Perhaps I am missing something major, please let me know.
Many thanks
Mia
-
Hi Jason,
I ran this with gatk:4.1.6.0 docker, but I still get the same error related to Funcotator:
Can you please have a look ?
Thanks,
Mia
-
Hi Mia,
As you mentioned, workflows in the featured workspace have their configuration already filled, though we may only have specific versions set up.
We also often have template json files you can use in Dockstore. There is normally a template json along with each version of the workflow. In Dockstore you can see it by clicking on the Files tab, then selecting the test Parameter Files sub tab.
In Github it's a bit trickier—sometimes authors have the template json in the same directory as the WDL and other times it's in a subfolder. In the case of Mutect2, that one exists here: https://github.com/broadinstitute/gatk/tree/master/scripts/m2_cromwell_tests
I hope this helps!
Kind regards,
Jason
-
Hi Jason,
That does actually help a lot, thank you for explaining !
Not sure whether you saw my update above on the Functator error (there is a possibility that we were typing at the same time) - can you please follow up?
I am basically still getting the same error, with missing the idx file (see post above with link to an err file)
Thanks
Mia
-
Hi Mia,
Upon closer inspection of the log file for your job, we noticed that your task isn't showing an indication that Funcotator is shutting down, how long the run lasted, how much memory it used, etc.
For example this is what you would see from a successful run.08:10:26.311 INFO ProgressMeter - Starting traversal
08:10:26.311 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute
08:16:04.255 INFO ProgressMeter - unmapped 5.6 215 38.2
08:16:04.255 INFO ProgressMeter - Traversal complete. Processed 215 total variants in 5.6 minutes.
08:16:04.256 INFO VcfFuncotationFactory - dbSNP 9606_b150 cache hits/total: 0/70
08:16:04.257 INFO Funcotator - Shutting down engine
[June 15, 2020 8:16:04 AM UTC] org.broadinstitute.hellbender.tools.funcotator.Funcotator done. Elapsed time: 6.12 minutes.
Runtime.totalMemory()=1789394944
Tool returned:
true
Using GATK jar /root/gatk.jar defined in environment variable GATK_LOCAL_JAR
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx3500m -jar /root/gatk.jar Funcotator --data-sources-path /cromwell_root/datasources_dir --ref-version hg19 --output-file-format MAF -R gs://gcp-public-data--broad-references/hg19/v0/Homo_sapiens_assembly19.fasta -V gs://fc-3336442e-1ba8-4f6d-aff6-5a9bbd6678cb/606b6527-b130-41cc-b43b-c2f3a0b13df3/Mutect2/8bfe751e-ac30-4e84-9d14-a761339685e4/call-Filter/HCC1143-filtered.vcf -O HCC1143-filtered.annotated.maf --annotation-default normal_barcode:HCC1143_BL --annotation-default tumor_barcode:HCC1143 --annotation-default Center:Unknown --annotation-default source:Unknown --transcript-list /cromwell_root/broad-public-datasets/funcotator/transcriptList.exact_uniprot_matches.AKT1_CRLF2_FGFR1.txt --remove-filtered-variantsBut yours looks like the following:
12:13:58.482 WARN MafOutputRenderer - MAF typically does not support multiple transcripts per variant, though this should be able to render (grouped by transcript). No user action needed.
Using GATK jar /root/gatk.jar defined in environment variable GATK_LOCAL_JAR
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx3500m -jar /root/gatk.jar Funcotator --data-sources-path /cromwell_root/datasources_dir --ref-version hg19 --output-file-format MAF -R gs://gatk-best-practices/somatic-b37/Homo_sapiens_assembly19.fasta -V gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/2c80c646-bf80-4376-9d28-3565a2f969b7/Mutect2/7484bb22-80b2-4bb3-ab63-a4911dedf003/call-Filter/attempt-2/COLFR0376C121_T001-filtered.vcf -O COLFR0376C121_T001-filtered.annotated.maf --annotation-default normal_barcode: --annotation-default tumor_barcode:COLFR0376C121_T001 --annotation-default Center:Unknown --annotation-default source:Unknown --transcript-selection-mode ALL --remove-filtered-variantsIt seems to end abruptly, which likely indicates inadequate memory or disk space. Would you be able to try again increasing either or both of these to see if you get to the Traversal complete and Shutting down engine messages?
I'm awaiting word from a member of the GATK team regarding an example for how one should calculate the disk space required. I'll post it here once I get it.
Kind regards,
Jason
-
Thank you, Jason.
Where can I increase those are there the values you might suggest I try before you hear back?
Many thanks,
Mia
-
Hi Mia,
You can set the disk space and memory in the Funcotate task in the method configuration.
I've also heard back, and in the standalone Funcotator WDL you can see the way they calculate disk space: https://github.com/broadinstitute/gatk/blob/master/scripts/funcotator_wdl/funcotator.wdl#L166-L170
Kind regards,
Jason
-
Hi Jason,
I managed to run this successfully by increasing the values you suggested.
Then, the next day I went to rerun the job to use as a control for a different run, and it failed !!
Although I used all of the same settings. I really do not understand what is going on...
Submission IDis 8a7feef8-ce06-4e99-896f-9603fbe408d4
The error:
Task Mutect2.Funcotate:NA:5 failed. The job was stopped before the command finished. PAPI error code 10. The assigned worker has failed to complete the operation
Please let me know and many thanks,
Mia
-
Hi Mia,
I'll take a closer look at the differences between the successful run and the unsuccessful run. One thing I'm noticing right off the bat is that the runtime parameters for the task don't seem to actually reflect the default_disk_space_gb and default_ram_mb inputs.
I can see you set these at 1000 and 30000, respectively.
But here are the runtime params:
Taking a closer look at the WDL, you can see in the call for Funcotate the standard_runtime is being provided to the task, as well as a calculated disk_space.
This seems consistent with a test workspace for the workflow—the default_disk_space_gb and default_ram_mb values aren't the ones being used for the runtime, but the standard_runtime memory and the calculated disk space.
I'm going to follow up with the original authors of the workflow to get a better sense of how this is constructed and the best way to adjust these parameters for larger inputs. It may be as simple as adding a value to
large_input_to_output_multiplier or emergency_extra_disk, but I want to make sure I get a clear answer from them before making any suggestions to you.Kind regards,
Jason
-
Thanks Jason, although I am still not sure why did the same run complete once, and then failed later - you would expect that the runtime parameters did not match my specifications both times, but the first run was completed successfully ...
I guess please keep this in mind when troubleshooting - exactly the same thing ran once, and failed on the second attempt. The only thing that I possible changed (although I am not 100% sure ) was to tick off the 'use call caching box' on one of the attempts.
Thanks
Mia
-
Hi Mia,
Did you end up running it on the same exactly pair, or a different pair? Can you provide the successful run's submission ID just so I'm definitely looking at the right jobs?
I was curious to know if the success was a coincidence, though you mentioning call caching being used—that could affected the outcome. If call caching was used on the one that succeeded, it may not have had to utilize as many resources (disk space and memory) to get to a success, since it uses any part of a previous run that already ran successfully rather than having to run everything from scratch.
Let me know the submission ID and I can do a more thorough investigation.
Kind regards,
Jason
-
Thanks Jason,
Successful ID: 454be392-8120-4c7c-abe6-393cb83490e7 (unfortunately I deleted all the outputs when I was cleaning the bucket, because I never expected this to fail again)
Failed ID: 8a7feef8-ce06-4e99-896f-9603fbe408d4
Of note, it is really difficult and not really 'user-friendly' to have to predict disc space and runtime for Funcotator, which seem to depend (based on calculations you copied above from other Functotator workflows) on outputs of Mutect2 (eg vcf sizes), when here Mutect and Funcotator and bundled together. So I cannot see output of Mutect to predict values for Funcotator - especially not when I get to run this over hundreds of samples. It is also pricey to have jobs failing because of this. Can this please be raised with the authors? It would be much better to have these variables encoded, so that the algorithm uses Mutect outputs to predict memory etc. that it will need to run Funcotator downstream. If this is really how things work (and this is my current understanding), I really do not know how to estimate this for many samples without 'trial and error' that is both costly and it will take extremely long time....
Thanks,
Mia
-
Hi Mia,
Thank you. I agree that the workflow can be more user-friendly, and I'll be bringing this up with the authors.
Kind regards,
Jason
-
Thanks Jason, I would also still like to run the workflow asap - can you please help me address the issue in short-term, while the team is presumably looking to make long-term changes to the workflow etc. that will presumably take time?
-
P.S. I reran the workflow with call caching enabled, and it failed. So the issue remains: the workflow that was run successfully keeps failing, across the same samples where it worked before. All of the submission IDs are in the thread above. New failure is :
bbd9a591-a069-4811-a027-3243d2a96eba
Same error:
Task Mutect2.Funcotate:NA:5 failed. Job exit code 1. Check gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/bbd9a591-a069-4811-a027-3243d2a96eba/Mutect2/12fff1b3-b2ab-43fc-9089-2fe6e80a6682/call-Funcotate/attempt-5/stderr for more information. PAPI error code 9. Please check the log file for more details: gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/bbd9a591-a069-4811-a027-3243d2a96eba/Mutect2/12fff1b3-b2ab-43fc-9089-2fe6e80a6682/call-Funcotate/attempt-5/Funcotate.log.
-
Looking at the runtime block for the funcotator task in the Mutect2 WDL workflow, it doesn't look like `default_disk_space_gb` or `default_ram_mb` has any role in changing the VM settings. I don’t see them being used at all in the rest of the task block so maybe they are not usable. I don't see any other parameter here that can be used to adjust the VM memory or disk settings so if your workflow is failing for these reasons then it won't be possible to complete the task for this workflow without editing the WDL script directly to increase the memory or diskspace.
Another simpler option may be to disable the funcotator task for this mutect2 workflow, and instead just have it output the resulting mutect2 bams. Once thats done, import the Funcotator workflow from this workspace : Variant-Functional-Annotation-With-Funcotator , and run the funcotator workflow on the bam files. The separate Funcotator workflow is a bit more usable in terms of being able to adjust the memory and diskspace for the funcatator tasks. You'll can change adjust the memory/disk using these workflow parameters:
memdisk_space_gb -
On second thought,
Can you try increasing the following variables: `small_task_mem` and `small_task_disk`
Looks like they are used in the `standard_runtime` dictionary which ends up being used to set the Funcotator runtime block.Fyi, the standard_runtime dictionary is used in several other tasks so this will increase the memory/diskspace for tasks other than Funcotator.
-
Hi Beri,
Thanks a million for helping. The workflow that I am using is copy-paste version of the GATK Mutect2 WDL (https://dockstore.org/workflows/github.com/broadinstitute/gatk/mutect2:4.1.6.0). The only reason I copied and re-uploaded is not to lose the workflow if it gets deleted, which happened to me before and I was unable to perform analyses on additional samples I wanted to pursue with the same workflow.
If what you are saying is correct, then GATK's Mutect2 workflow cited above does not seem to be optimized to run Funcotator, which is its core part. It seems that other Fucotator workflows calculate the required space/memory etc. automatically, so I would hope that this can also be incorporated here. It has been both costly and timely to me projects to try to run a workflow which does not seem to be optimized to run to completion without manual modifications, which as it sounds need to be given the 'best guess' to get them to work on hundreds of samples. Can these changes please be incorporated ?
I can try to increase the variables you suggest, but can you please let me know where to do so (they are not part of the standard json from what i see) and which values to try? I don't know what these were currently, nor what would, again, seem to be the best guess.
I only really expected to be able to run the workflow to completion by populating originally given jsons and following instructions in the documentation. Manual modifications and 'best guesses', which seem to be required, are not really user-friendly, and it currently seems that it is not possible to run this workflow without them.
Please let me know how to proceed, I am happy to try things, but please note I would need more specific directions as I am not able to code in WDLs to optimize, which is why I resourced to using workflows in the first place.
Many thanks for helping,
Mia
-
Hey MPetlj,
Here is a screenshot of the variables that can be used to change the resource usage for the Funcotator task:
Why:
The Funcotator task in the mutect workflow uses a map called runtime_params which is used to set the values in the runtime block. The `runtime_params` is defined by `standard_runtime` map in the Funcatator call block. The standard_runtime map is defined earlier in the workflow and it includes the variables mentioned above : `small_task_mem` and `small_task_disk`. Since these variables are defined in the workflows input block with default values you have access to adjust the default in the Terra workflow config.Hope the screenshot and explanation helps.
---
No, mutect2 workflow does not automatically adjust the disk usage for the Funcotator task based on the size of the reference/resources and inputs. If you want this done automatically then you can opt to use the Funcotator workflow. Thanks for your feedback, it has been passed to GATK git team.
-
Hi Beri,
Thank you for raising with GATK team, is there somewhere I can keep track of the ticket ?
Thank you also for the explanations, I do see now 'small_task_mem' and 'small_task_disk' - it looks like the 'search' option failed on me when I looked for those, apologies.
Could you please help me further by having a look at the failed job (29b915c1-0c50-40f3-ab20-9ced3d0274ea), you now have access to my workspace (661-Clonal hematopoiesis), and please suggest values for me to try for both of these parameters? I am not sure where to start guessing and I suspect you have more experience in these sorts of workflows... any advice on this would be helpful, as I really do not know by how much to increment, nor how to calculate what might work.
Thanks,
Mia
-
You can track ticket here.
I don't see a job ID'ed as job 29b915c1-0c50-40f3-ab20-9ced3d0274ea ?
For the values you can first try doubling the current defaults, so try 8 for memory and 200 for diskspace.
-
Hi Mia,
Looks like the job was successful so I'll close out this ticket.
-
Hi Beri,
128/129 jobs were successful. For the one that failed I get error in the bottom, could this also be due to space/memory requirements? I should note, I ran this with "16" for memory and "400" for diskspace.
2020/07/01 16:05:27 Starting container setup. 2020/07/01 16:05:31 Done container setup. 2020/07/01 16:05:32 Starting localization. 2020/07/01 16:05:40 Localization script execution started... 2020/07/01 16:05:40 Localizing input gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/script -> /cromwell_root/script 2020/07/01 16:05:41 Localization script execution complete. 2020/07/01 16:05:47 Done localization. 2020/07/01 16:05:48 Running user action: docker run -v /mnt/local-disk:/cromwell_root --entrypoint= broadinstitute/gatk@sha256:2c0e2ba20c9beb58842ba2149efc29059bc52a5178ce05debf0f38238c0bde86 /bin/bash /cromwell_root/script Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.479e376e 16:05:59.463 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.1.6.0-local.jar!/com/intel/gkl/native/libgkl_compression.so 16:06:00.537 INFO GetSampleName - ------------------------------------------------------------ 16:06:00.538 INFO GetSampleName - The Genome Analysis Toolkit (GATK) v4.1.6.0 16:06:00.538 INFO GetSampleName - For support and documentation go to https://software.broadinstitute.org/gatk/ 16:06:00.539 INFO GetSampleName - Executing as root@d2d48a310a77 on Linux v4.19.112+ amd64 16:06:00.540 INFO GetSampleName - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03 16:06:00.541 INFO GetSampleName - Start Date/Time: July 1, 2020 4:05:59 PM UTC 16:06:00.541 INFO GetSampleName - ------------------------------------------------------------ 16:06:00.541 INFO GetSampleName - ------------------------------------------------------------ 16:06:00.542 INFO GetSampleName - HTSJDK Version: 2.21.2 16:06:00.542 INFO GetSampleName - Picard Version: 2.21.9 16:06:00.543 INFO GetSampleName - HTSJDK Defaults.COMPRESSION_LEVEL : 2 16:06:00.543 INFO GetSampleName - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 16:06:00.543 INFO GetSampleName - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 16:06:00.543 INFO GetSampleName - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 16:06:00.544 INFO GetSampleName - Deflater: IntelDeflater 16:06:00.547 INFO GetSampleName - Inflater: IntelInflater 16:06:00.548 INFO GetSampleName - GCS max retries/reopens: 20 16:06:00.548 INFO GetSampleName - Requester pays: disabled 16:06:00.548 INFO GetSampleName - Initializing engine 16:06:06.526 INFO GetSampleName - Done initializing engine 16:06:06.532 INFO ProgressMeter - Starting traversal 16:06:06.534 INFO ProgressMeter - Current Locus Elapsed Minutes Records Processed Records/Minute 16:06:06.549 INFO ProgressMeter - unmapped 0.0 0 NaN 16:06:06.549 INFO ProgressMeter - Traversal complete. Processed 0 total records in 0.0 minutes. 16:06:06.549 INFO GetSampleName - Shutting down engine [July 1, 2020 4:06:07 PM UTC] org.broadinstitute.hellbender.tools.GetSampleName done. Elapsed time: 0.13 minutes. Runtime.totalMemory()=226246656 Using GATK jar /root/gatk.jar defined in environment variable GATK_LOCAL_JAR Running: java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx3000m -jar /root/gatk.jar GetSampleName -R gs://gatk-best-practices/somatic-b37/Homo_sapiens_assembly19.fasta -I gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/HTAPP_WES/MBC/WES_GRCh37/05246_CCPM_0300532_T1.bam -O tumor_name.txt -encode Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.479e376e 16:06:13.275 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.1.6.0-local.jar!/com/intel/gkl/native/libgkl_compression.so 16:06:13.685 INFO Mutect2 - ------------------------------------------------------------ 16:06:13.690 INFO Mutect2 - The Genome Analysis Toolkit (GATK) v4.1.6.0 16:06:13.690 INFO Mutect2 - For support and documentation go to https://software.broadinstitute.org/gatk/ 16:06:13.693 INFO Mutect2 - Executing as root@d2d48a310a77 on Linux v4.19.112+ amd64 16:06:13.695 INFO Mutect2 - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03 16:06:13.699 INFO Mutect2 - Start Date/Time: July 1, 2020 4:06:13 PM UTC 16:06:13.700 INFO Mutect2 - ------------------------------------------------------------ 16:06:13.700 INFO Mutect2 - ------------------------------------------------------------ 16:06:13.701 INFO Mutect2 - HTSJDK Version: 2.21.2 16:06:13.702 INFO Mutect2 - Picard Version: 2.21.9 16:06:13.702 INFO Mutect2 - HTSJDK Defaults.COMPRESSION_LEVEL : 2 16:06:13.702 INFO Mutect2 - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 16:06:13.703 INFO Mutect2 - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 16:06:13.703 INFO Mutect2 - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 16:06:13.703 INFO Mutect2 - Deflater: IntelDeflater 16:06:13.703 INFO Mutect2 - Inflater: IntelInflater 16:06:13.704 INFO Mutect2 - GCS max retries/reopens: 20 16:06:13.704 INFO Mutect2 - Requester pays: disabled 16:06:13.704 INFO Mutect2 - Initializing engine 16:06:19.883 INFO FeatureManager - Using codec IntervalListCodec to read file gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-SplitIntervals/cacheCopy/glob-0fc990c5ca95eebc97c4c204e3e303e1/0000-scattered.interval_list 16:06:20.252 INFO IntervalArgumentCollection - Processing 62039531 bp from intervals 16:06:20.281 INFO Mutect2 - Done initializing engine 16:06:20.952 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/gatk/gatk-package-4.1.6.0-local.jar!/com/intel/gkl/native/libgkl_utils.so 16:06:21.014 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/gatk/gatk-package-4.1.6.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so 16:06:21.109 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM 16:06:21.110 INFO IntelPairHmm - Available threads: 1 16:06:21.111 INFO IntelPairHmm - Requested threads: 4 16:06:21.111 WARN IntelPairHmm - Using 1 available threads, but 4 were requested 16:06:21.112 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation 16:06:21.374 INFO ProgressMeter - Starting traversal 16:06:21.375 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute 16:06:22.322 INFO VectorLoglessPairHMM - Time spent in setup for JNI call : 0.0 16:06:22.323 INFO PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 0.0 16:06:22.323 INFO SmithWatermanAligner - Total compute time in java Smith-Waterman : 0.00 sec 16:06:22.324 INFO Mutect2 - Shutting down engine [July 1, 2020 4:06:22 PM UTC] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 0.15 minutes. Runtime.totalMemory()=229560320 java.lang.IllegalArgumentException: Reference name for '1279721472' not found in sequence dictionary. at htsjdk.samtools.SAMRecord.resolveNameFromIndex(SAMRecord.java:573) at htsjdk.samtools.SAMRecord.setReferenceIndex(SAMRecord.java:426) at htsjdk.samtools.BAMRecord.<init>(BAMRecord.java:110) at htsjdk.samtools.DefaultSAMRecordFactory.createBAMRecord(DefaultSAMRecordFactory.java:42) at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:283) at htsjdk.samtools.BAMFileReader$BAMFileIterator.getNextRecord(BAMFileReader.java:866) at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:1026) at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:840) at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.<init>(BAMFileReader.java:1008) at htsjdk.samtools.BAMFileReader.createIndexIterator(BAMFileReader.java:955) at htsjdk.samtools.BAMFileReader.query(BAMFileReader.java:612) at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.query(SamReader.java:533) at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.queryOverlapping(SamReader.java:405) at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.loadNextIterator(SamReaderQueryingIterator.java:125) at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.<init>(SamReaderQueryingIterator.java:66) at org.broadinstitute.hellbender.engine.ReadsDataSource.prepareIteratorsForTraversal(ReadsDataSource.java:416) at org.broadinstitute.hellbender.engine.ReadsDataSource.iterator(ReadsDataSource.java:342) at org.broadinstitute.hellbender.engine.MultiIntervalLocalReadShard.iterator(MultiIntervalLocalReadShard.java:134) at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.<init>(AssemblyRegionIterator.java:86) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:188) at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:173) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1048) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206) at org.broadinstitute.hellbender.Main.main(Main.java:292) Using GATK jar /root/gatk.jar defined in environment variable GATK_LOCAL_JAR Running: java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx3000m -jar /root/gatk.jar Mutect2 -R gs://gatk-best-practices/somatic-b37/Homo_sapiens_assembly19.fasta -I gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/HTAPP_WES/MBC/WES_GRCh37/05246_CCPM_0300532_T1.bam -tumor 05246_CCPM_0300532_T1 -L gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-SplitIntervals/cacheCopy/glob-0fc990c5ca95eebc97c4c204e3e303e1/0000-scattered.interval_list -O output.vcf --f1r2-tar-gz f1r2.tar.gz ln: failed to access '/cromwell_root/*normal-pileups.table': No such file or directory ln: failed to access '/cromwell_root/*tumor-pileups.table': No such file or directory 2020/07/01 16:06:25 Starting delocalization. 2020/07/01 16:06:26 Delocalization script execution started... 2020/07/01 16:06:26 Delocalizing output /cromwell_root/memory_retry_rc -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/memory_retry_rc 2020/07/01 16:06:26 Delocalizing output /cromwell_root/rc -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/rc 2020/07/01 16:06:28 Delocalizing output /cromwell_root/stdout -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/stdout 2020/07/01 16:06:29 Delocalizing output /cromwell_root/stderr -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/stderr 2020/07/01 16:06:30 Delocalizing output /cromwell_root/output.vcf -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/output.vcf 2020/07/01 16:06:31 Delocalizing output /cromwell_root/glob-33dc6fdaacf3c597568558628abb6bca -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/glob-33dc6fdaacf3c597568558628abb6bca/ 2020/07/01 16:06:33 Delocalizing output /cromwell_root/output.vcf.idx -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/output.vcf.idx 2020/07/01 16:06:34 Delocalizing output /cromwell_root/output.vcf.stats -> gs://fc-secure-a46c7502-d26e-4217-b1d4-7d80a20d7456/a427e4bf-ba51-4847-9f50-0ece16e9015e/Mutect2/2a12d693-5a5f-4269-af28-8fc2bf3adb12/call-M2/shard-0/output.vcf.stats Required file output '/cromwell_root/output.vcf.stats' does not exist.
Please sign in to leave a comment.
31 comments