Troubleshooting VCF Output Truncation Issue during GATK CombineGVCFs Process
I'm running gatk-4.1.2.0 where I combine the VCF outputs for my six samples for proceeding steps such as Cigar split, base recalibration, sample naming, etc to generate my genotype VCF outputs. Unfortunately, combining the vcf output truncates chromosomal reads from 25 to 12. Interestingly, all six samples have 25 chromosomal read data in them.
gatk CombineGVCFs -R /home/akandeba/Desktop/Zebrafish_reference/GCF_000002035.6_GRCz11_genomic.fna --variant *_sampnamed.g.vcf.g.gz -O Pen_wsamps.g.vcf.gz
A look into the output is shown below: zgrep -v "^#" Pen_wsamps.g.vcf.gz | cut -f 1 | sort | uniq -c
gzip: Pen_wsamps.g.vcf.gz: invalid compressed data--format violated
3180475 NC_007112.7
4073215 NC_007113.7
3812157 NC_007114.7
3619687 NC_007115.7
4122713 NC_007116.7
3324278 NC_007117.7
3953942 NC_007118.7
3483638 NC_007119.7
3122617 NC_007120.7
2771400 NC_007121.7
2731475 NC_007122.7
2732872 NC_007123.7
3064453 NC_007124.7
2476811 NC_007125.7
2820480 NC_007126.7
The NC_ are ReqSeq ID for the respective chromosomes. I am interested in data on chromosome 19 which is most likely committed by this approach. I would greatly appreciate any help to resolve this issue.
-
Can you share the log of CombineGVCFs tool? Did it run to completion without any issues?
-
Log output
00:35:14.125 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute 00:35:14.232 WARN ReferenceConfidenceVariantContextMerger - Detected invalid annotations: When trying to merge variant contexts at location NC_007112.7:3724 the annotation MLEAC=[1, 0] was not a numerical value and was ignored 00:35:24.136 INFO ProgressMeter - NC_007112.7:8651384 0.2 941000 5640359.6 00:35:34.140 INFO ProgressMeter - NC_007112.7:23373934 0.3 2029000 6082438.2 00:35:44.148 INFO ProgressMeter - NC_007112.7:41286119 0.5 3162000 6319155.3 00:35:54.148 INFO ProgressMeter - NC_007112.7:50390179 0.7 4313000 6465782.2 00:36:04.157 INFO ProgressMeter - NC_007112.7:57348148 0.8 5483000 6575391.7 00:36:14.159 INFO ProgressMeter - NC_007113.7:4043775 1.0 6601000 6597261.6 00:36:24.163 INFO ProgressMeter - NC_007113.7:11999200 1.2 7741000 6631542.9 00:36:34.175 INFO ProgressMeter - NC_007113.7:22845123 1.3 8890000 6663335.4 00:36:44.183 INFO ProgressMeter - NC_007113.7:31864533 1.5 10036000 6686357.7 00:36:54.187 INFO ProgressMeter - NC_007113.7:42232206 1.7 11172000 6699113.5 00:37:04.193 INFO ProgressMeter - NC_007113.7:52404138 1.8 12283000 6695679.0 00:37:14.198 INFO ProgressMeter - NC_007114.7:301545 2.0 13415000 6703477.9 00:37:24.205 INFO ProgressMeter - NC_007114.7:13441394 2.2 14519000 6696955.7 00:37:34.208 INFO ProgressMeter - NC_007114.7:23436485 2.3 15739000 6741289.1 00:37:44.210 INFO ProgressMeter - NC_007114.7:33635094 2.5 16924000 6765766.1 00:37:54.211 INFO ProgressMeter - NC_007114.7:42087109 2.7 18120000 6791349.6 00:38:04.211 INFO ProgressMeter - NC_007114.7:55773977 2.8 19211000 6776924.6 00:38:14.216 INFO ProgressMeter - NC_007115.7:526605 3.0 20362000 6783903.7 00:38:24.222 INFO ProgressMeter - NC_007115.7:8638509 3.2 21564000 6806209.5 00:38:34.224 INFO ProgressMeter - NC_007115.7:17528666 3.3 22754000 6822822.7 00:38:44.226 INFO ProgressMeter - NC_007115.7:29792183 3.5 23903000 6826145.5 00:38:54.228 INFO ProgressMeter - NC_007115.7:56249712 3.7 24845000 6772738.2 00:39:04.233 INFO ProgressMeter - NC_007115.7:74239538 3.8 25837000 6736923.5 00:39:14.237 INFO ProgressMeter - NC_007116.7:4118485 4.0 26991000 6744602.5 00:39:24.241 INFO ProgressMeter - NC_007116.7:17848793 4.2 28163000 6755985.2 00:39:34.246 INFO ProgressMeter - NC_007116.7:26207624 4.3 29378000 6776384.8 00:39:44.246 INFO ProgressMeter - NC_007116.7:37991247 4.5 30527000 6780739.0 00:39:54.248 INFO ProgressMeter - NC_007116.7:54576443 4.7 31712000 6792444.7 00:40:04.256 INFO ProgressMeter - NC_007116.7:65953311 4.8 32850000 6793506.4 00:40:14.258 INFO ProgressMeter - NC_007117.7:121166 5.0 34010000 6798985.8 00:40:24.261 INFO ProgressMeter - NC_007117.7:9618967 5.2 35190000 6807981.0 00:40:34.261 INFO ProgressMeter - NC_007117.7:21430802 5.3 36355000 6813666.7 00:40:44.262 INFO ProgressMeter - NC_007117.7:37733694 5.5 37466000 6809173.2 00:40:54.277 INFO ProgressMeter - NC_007117.7:48873259 5.7 38576000 6804487.4 00:41:04.278 INFO ProgressMeter - NC_007117.7:58869792 5.8 39710000 6804454.1 00:41:14.283 INFO ProgressMeter - NC_007118.7:10395919 6.0 40834000 6802699.9 00:41:24.283 INFO ProgressMeter - NC_007118.7:24053337 6.2 41995000 6807093.2 00:41:34.291 INFO ProgressMeter - NC_007118.7:33130010 6.3 43162000 6812076.8 00:41:44.294 INFO ProgressMeter - NC_007118.7:48256890 6.5 44337000 6818139.9 00:41:54.303 INFO ProgressMeter - NC_007118.7:58989326 6.7 45470000 6817466.2 00:42:04.308 INFO ProgressMeter - NC_007118.7:69956467 6.8 46528000 6805937.8 00:42:14.314 INFO ProgressMeter - NC_007119.7:2430939 7.0 47635000 6801939.1 00:42:24.321 INFO ProgressMeter - NC_007119.7:9795343 7.2 48706000 6793089.7 00:42:34.322 INFO ProgressMeter - NC_007119.7:18639548 7.3 49805000 6788551.5 00:42:44.331 INFO ProgressMeter - NC_007119.7:28713083 7.5 50942000 6789158.7 00:42:54.335 INFO ProgressMeter - NC_007119.7:44769555 7.7 52036000 6784207.2 00:43:04.344 INFO ProgressMeter - NC_007119.7:52650292 7.8 53182000 6786029.5 00:43:14.347 INFO ProgressMeter - NC_007120.7:3209199 8.0 54362000 6792122.8 00:43:24.347 INFO ProgressMeter - NC_007120.7:14390342 8.2 55497000 6792473.6 00:43:34.353 INFO ProgressMeter - NC_007120.7:31576855 8.3 56565000 6784774.0 00:43:44.348 INFO ProgressMeter - NC_007120.7:40797318 8.5 57595000 6772920.9 00:43:54.356 INFO ProgressMeter - NC_007120.7:51049725 8.7 58582000 6756460.1 00:44:04.356 INFO ProgressMeter - NC_007121.7:1598455 8.8 59635000 6748190.9 00:44:14.357 INFO ProgressMeter - NC_007121.7:9406737 9.0 60658000 6736883.4 00:44:24.358 INFO ProgressMeter - NC_007121.7:21681014 9.2 61678000 6725659.9 00:44:34.363 INFO ProgressMeter - NC_007121.7:32500363 9.3 62576000 6701723.2 00:44:44.368 INFO ProgressMeter - NC_007121.7:40875704 9.5 63616000 6693567.5 00:44:54.369 INFO ProgressMeter - NC_007122.7:678864 9.7 64643000 6684394.8 00:45:04.381 INFO ProgressMeter - NC_007122.7:6877010 9.8 65621000 6670427.7 00:45:14.387 INFO ProgressMeter - NC_007122.7:20031846 10.0 66522000 6649296.5 00:45:24.396 INFO ProgressMeter - NC_007122.7:30077087 10.2 67461000 6632561.6 00:45:34.405 INFO ProgressMeter - NC_007122.7:39168392 10.3 68437000 6619945.8 00:45:44.405 INFO ProgressMeter - NC_007122.7:45416613 10.5 69446000 6610966.6 00:45:54.409 INFO ProgressMeter - NC_007123.7:5216760 10.7 70427000 6599602.7 00:46:04.420 INFO ProgressMeter - NC_007123.7:15581021 10.8 71389000 6586774.6 00:46:14.432 INFO ProgressMeter - NC_007123.7:26074823 11.0 72312000 6570771.7 00:46:24.432 INFO ProgressMeter - NC_007123.7:35379738 11.2 73198000 6552042.6 00:46:34.437 INFO ProgressMeter - NC_007123.7:47495556 11.3 74063000 6531973.6 00:46:44.437 INFO ProgressMeter - NC_007124.7:2499214 11.5 74990000 6517922.3 00:46:54.446 INFO ProgressMeter - NC_007124.7:11583988 11.7 75814000 6495364.3 00:47:04.446 INFO ProgressMeter - NC_007124.7:23370050 11.8 76633000 6473101.6 00:47:14.449 INFO ProgressMeter - NC_007124.7:29436952 12.0 77446000 6450930.4 00:47:24.450 INFO ProgressMeter - NC_007124.7:35801539 12.2 78272000 6430452.2 00:47:34.459 INFO ProgressMeter - NC_007124.7:43219263 12.3 79098000 6410458.0 00:47:44.466 INFO ProgressMeter - NC_007124.7:52007038 12.5 79977000 6395252.3 00:47:54.477 INFO ProgressMeter - NC_007125.7:6245988 12.7 80902000 6384043.2 00:48:04.484 INFO ProgressMeter - NC_007125.7:16790395 12.8 81800000 6371055.6 00:48:14.488 INFO ProgressMeter - NC_007125.7:30802943 13.0 82689000 6357733.5 00:48:24.497 INFO ProgressMeter - NC_007125.7:45421943 13.2 83595000 6346007.1 00:48:34.499 INFO ProgressMeter - NC_007125.7:52657785 13.3 84501000 6334613.6 00:48:44.521 INFO ProgressMeter - NC_007126.7:4816143 13.5 85435000 6325426.1 00:48:54.524 INFO ProgressMeter - NC_007126.7:15482901 13.7 86261000 6308710.8 00:49:04.534 INFO ProgressMeter - NC_007126.7:22443102 13.8 86966000 6283602.4 00:49:14.540 INFO ProgressMeter - NC_007126.7:29886887 14.0 87875000 6273686.2 00:49:24.555 INFO ProgressMeter - NC_007126.7:38049509 14.2 88761000 6262314.4 00:49:34.559 INFO ProgressMeter - NC_007126.7:45141489 14.3 89636000 6250527.4 00:49:44.566 INFO ProgressMeter - NC_007127.7:1969833 14.5 90357000 6228360.1 00:49:54.572 INFO ProgressMeter - NC_007127.7:7680072 14.7 91084000 6207119.8 00:50:04.572 INFO ProgressMeter - NC_007127.7:13167424 14.8 91790000 6184983.5 00:50:14.575 INFO ProgressMeter - NC_007127.7:21695436 15.0 92619000 6171521.1 00:50:24.584 INFO ProgressMeter - NC_007127.7:27518048 15.2 93501000 6161793.1 00:50:34.591 INFO ProgressMeter - NC_007127.7:34915664 15.3 94373000 6151644.9 00:50:44.592 INFO ProgressMeter - NC_007127.7:43113181 15.5 95220000 6140142.5 00:50:54.607 INFO ProgressMeter - NC_007127.7:50063011 15.7 96021000 6125858.9 00:51:04.608 INFO ProgressMeter - NC_007127.7:55203502 15.8 96843000 6113291.9 00:51:14.610 INFO ProgressMeter - NC_007128.7:7561999 16.0 97738000 6105540.4 00:51:24.622 INFO ProgressMeter - NC_007128.7:15312082 16.2 98659000 6099493.4 00:51:34.632 INFO ProgressMeter - NC_007128.7:25596650 16.3 99545000 6091440.4 00:51:44.640 INFO ProgressMeter - NC_007128.7:33839812 16.5 100410000 6082290.5 00:51:54.643 INFO ProgressMeter - NC_007128.7:43755493 16.7 101328000 6076532.4 00:52:04.649 INFO ProgressMeter - NC_007128.7:51201278 16.8 102233000 6070098.3 00:52:14.652 INFO ProgressMeter - NC_007129.7:3360769 17.0 103168000 6065572.0 00:52:24.653 INFO ProgressMeter - NC_007129.7:13087137 17.2 104085000 6060103.2 00:52:34.657 INFO ProgressMeter - NC_007129.7:24994528 17.3 105013000 6055344.8 00:52:44.672 INFO ProgressMeter - NC_007129.7:36042376 17.5 105694000 6036512.4 00:52:54.672 INFO ProgressMeter - NC_007129.7:46117452 17.7 106525000 6026607.0 00:53:04.680 INFO ProgressMeter - NC_007130.7:508738 17.8 107307000 6014095.5 00:53:14.684 INFO ProgressMeter - NC_007130.7:6340115 18.0 108252000 6010888.8 00:53:24.691 INFO ProgressMeter - NC_007130.7:10864007 18.2 109047000 5999471.8 00:53:34.705 INFO ProgressMeter - NC_007130.7:18739394 18.3 109764000 5983977.5 00:53:44.711 INFO ProgressMeter - NC_007130.7:26961921 18.5 110662000 5978573.5 00:53:54.716 INFO ProgressMeter - NC_007130.7:36113598 18.7 111565000 5973544.3 00:54:04.722 INFO ProgressMeter - NC_007130.7:43638146 18.8 112468000 5968598.9 00:54:14.722 INFO ProgressMeter - NC_007131.7:1281221 19.0 113388000 5964665.9 00:54:24.722 INFO ProgressMeter - NC_007131.7:7109693 19.2 114285000 5959601.8 00:54:34.723 INFO ProgressMeter - NC_007131.7:18307324 19.3 115044000 5947485.7 00:54:44.732 INFO ProgressMeter - NC_007131.7:25508537 19.5 115790000 5934869.7 00:54:54.748 INFO ProgressMeter - NC_007131.7:29623587 19.7 116374000 5914204.5 00:55:04.760 INFO ProgressMeter - NC_007131.7:34235151 19.8 116949000 5893443.4 00:55:14.760 INFO ProgressMeter - NC_007131.7:39537534 20.0 117707000 5882237.3 00:55:24.769 INFO ProgressMeter - NC_007131.7:47694596 20.2 118467000 5871276.7 00:55:34.778 INFO ProgressMeter - NC_007131.7:52561685 20.3 119138000 5856111.4 00:55:44.779 INFO ProgressMeter - NC_007132.7:228196 20.5 119700000 5835921.4 00:55:54.780 INFO ProgressMeter - NC_007132.7:4687643 20.7 120520000 5828534.1 00:56:04.791 INFO ProgressMeter - NC_007132.7:10049149 20.8 121163000 5812727.0 00:56:14.806 INFO ProgressMeter - NC_007132.7:14035851 21.0 121756000 5794772.8 00:56:24.818 INFO ProgressMeter - NC_007132.7:21407038 21.2 122330000 5776222.7 00:56:34.821 INFO ProgressMeter - NC_007132.7:27220972 21.3 123026000 5763709.7 00:56:44.824 INFO ProgressMeter - NC_007132.7:35302978 21.5 123771000 5753673.0 00:56:54.830 INFO ProgressMeter - NC_007132.7:40546822 21.7 124390000 5737965.2 00:57:04.837 INFO ProgressMeter - NC_007132.7:45116887 21.8 125090000 5726200.7 00:57:14.844 INFO ProgressMeter - NC_007133.7:1525066 22.0 125823000 5716113.7 00:57:24.848 INFO ProgressMeter - NC_007133.7:3385989 22.2 126481000 5702809.7 00:57:34.858 INFO ProgressMeter - NC_007133.7:10559116 22.3 127381000 5700508.6 00:57:44.866 INFO ProgressMeter - NC_007133.7:18171141 22.5 128252000 5696961.9 00:57:54.874 INFO ProgressMeter - NC_007133.7:31733690 22.7 129032000 5689454.9 00:58:04.877 INFO ProgressMeter - NC_007134.7:214359 22.8 129831000 5682909.8 00:58:14.885 INFO ProgressMeter - NC_007134.7:4793362 23.0 130653000 5677438.5 00:58:24.893 INFO ProgressMeter - NC_007134.7:11451796 23.2 131318000 5665272.7 00:58:34.893 INFO ProgressMeter - NC_007134.7:24131902 23.3 132201000 5662650.8 00:58:44.902 INFO ProgressMeter - NC_007134.7:31909193 23.5 133161000 5663304.7 00:58:54.905 INFO ProgressMeter - NC_007134.7:39584036 23.7 134031000 5660172.6 00:59:04.911 INFO ProgressMeter - NC_007134.7:45157846 23.8 134846000 5654766.0 00:59:14.912 INFO ProgressMeter - NC_007135.7:6162178 24.0 135689000 5650620.1 00:59:24.920 INFO ProgressMeter - NC_007135.7:17260579 24.2 136616000 5649978.1 00:59:34.925 INFO ProgressMeter - NC_007135.7:24564179 24.3 137211000 5635720.2 00:59:44.926 INFO ProgressMeter - NC_007135.7:29522820 24.5 137762000 5619876.5 00:59:54.929 INFO ProgressMeter - NC_007135.7:36179991 24.7 138499000 5611775.8 01:00:04.943 INFO ProgressMeter - NC_007135.7:40298897 24.8 139216000 5602937.4 01:00:14.947 INFO ProgressMeter - NC_007136.7:1693574 25.0 139794000 5588697.4 01:00:24.955 INFO ProgressMeter - NC_007136.7:5986806 25.2 140473000 5578642.2 01:00:34.961 INFO ProgressMeter - NC_007136.7:13659138 25.3 141244000 5572356.3 01:00:44.965 INFO ProgressMeter - NC_007136.7:20672572 25.5 142049000 5567492.4 01:00:54.968 INFO ProgressMeter - NC_007136.7:31103060 25.7 142843000 5562266.9 01:01:04.970 INFO ProgressMeter - NC_007136.7:36083943 25.8 143728000 5560633.1 01:01:14.979 INFO ProgressMeter - NW_003337057.1:17351 26.0 144618000 5559187.5 01:01:24.981 INFO ProgressMeter - NW_003336732.1:33728 26.2 145514000 5558014.2 01:01:34.988 INFO ProgressMeter - NW_001884427.4:217007 26.3 146379000 5555665.2 01:01:45.004 INFO ProgressMeter - NW_018394527.1:103534 26.5 147210000 5552025.0 01:01:55.009 INFO ProgressMeter - NW_018394599.1:180864 26.7 147993000 5546673.0 01:02:05.021 INFO ProgressMeter - NW_018394660.1:165602 26.8 148657000 5536931.0 01:02:15.023 INFO ProgressMeter - NW_018394720.1:178871 27.0 149315000 5527121.4 01:02:25.027 INFO ProgressMeter - NW_018394785.1:437433 27.2 149966000 5517167.8 01:02:35.034 INFO ProgressMeter - NW_018394842.1:140309 27.3 150621000 5507471.8 01:02:45.037 INFO ProgressMeter - NW_018394903.1:47231 27.5 151292000 5498491.4 01:02:55.049 INFO ProgressMeter - NW_018394956.1:37031 27.7 151954000 5489257.8 01:03:05.056 INFO ProgressMeter - NW_018395011.1:100776 27.8 152861000 5488955.3 01:03:15.064 INFO ProgressMeter - NW_018395091.1:267210 28.0 153731000 5487325.8 01:03:25.066 INFO ProgressMeter - NW_018395162.1:97947 28.2 154560000 5484283.6 01:03:35.068 INFO ProgressMeter - NW_018395236.1:144415 28.3 155374000 5480748.0 01:03:45.071 INFO ProgressMeter - NW_018395293.1:158482 28.5 156080000 5473463.2 01:03:55.078 INFO ProgressMeter - NW_018395361.1:121721 28.7 156806000 5466950.9 01:03:59.028 INFO ProgressMeter - NC_002333.2:10964 28.7 157102302 5464735.2 01:03:59.028 INFO ProgressMeter - Traversal complete. Processed 157102302 total variants in 28.7 minutes. 01:03:59.043 INFO CombineGVCFs - Shutting down engine [April 13, 2024 1:03:59 AM EDT] org.broadinstitute.hellbender.tools.walkers.CombineGVCFs done. Elapsed time: 29.35 minutes. Runtime.totalMemory()=2430599168 htsjdk.samtools.util.RuntimeIOException: Unable to close index for file:///home/akandeba/Desktop/BAM_files/Add_Group/Dup_filtered/Cigar_Split/Prelim_VCFcalling/Pen_wsamps.g.vcf.gz at htsjdk.variant.variantcontext.writer.IndexingVariantContextWriter.close(IndexingVariantContextWriter.java:183) at htsjdk.variant.variantcontext.writer.VCFWriter.close(VCFWriter.java:231) at org.broadinstitute.hellbender.tools.walkers.CombineGVCFs.closeTool(CombineGVCFs.java:495) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1043) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205) at org.broadinstitute.hellbender.Main.main(Main.java:291) Caused by: java.io.IOException: Terminator block not found after closing BGZF file Pen_wsamps.g.vcf.gz at htsjdk.samtools.util.BlockCompressedOutputStream.close(BlockCompressedOutputStream.java:329) at htsjdk.variant.variantcontext.writer.IndexingVariantContextWriter.close(IndexingVariantContextWriter.java:172) ... 9 more
-
Looks like there is a problem when closing the file. Can you try running this tool with the latest version 4.5 to see if the issue persists?
Also can you include the below parameter for a more detailed error message
--java-options '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true'
One final issue that caught my attention is that the --variant parameter must be specified for each input file separately. Can you fix that in the command line as well?
Regards.
-
Hi Gökalp Çelik
I will re-run the command and get back to you with the outcome. Also, what do you intend by specifying each file's variant parameters? Can you help me navigate that?
Best
-
The parameter --variant must be used individually for each GVCF file to be combined by this tool such as
--variant 1.g.vcf.gz --variant 2.g.vcf.gz ...
Please sign in to leave a comment.
5 comments