Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

"Did not inflate expected amount" Error

Answered
Follow
Avatar
Jacob Wang

Hi! I'm doing WGS analysis of a pedigree of three individuals using GATK 4.2.0.0. Everything went on well for the first individual. However, in the step of generating gvcf file from bam file, I encountered the error [htsjdk.samtools.SAMFormatException: Did not inflate expected amount] in the other two of the individuals. Please help me! Thank you in advance!


a) GATK version used:

GATK 4.2.0.0


b) Exact command used:

java -jar /home/ngs/biosoft/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar \
HaplotypeCaller \
-R /media/ngs/NGS0/Database/RefSeq/Homo_sapiens_NCBI_GRCh38Decoy/Homo_sapiens/NCBI/GRCh38Decoy/Sequence/WholeGenomeFasta/NewIndex/genome.fa \
-I /media/ngs/BAM5T/WGS_analysis/Data/9_BQSRBam/Ped-San-3_merged_realigned_bqsr.bam \
-ERC GVCF \
-O /media/ngs/BAM5T/WGS_analysis/Data/10_gvcf/Ped-San-3_merged_realigned_bqsr.g.vcf


c) Entire error log:

14:14:32.075 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/ngs/biosoft/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 01, 2021 2:14:32 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
14:14:32.573 INFO HaplotypeCaller - ------------------------------------------------------------
14:14:32.573 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.0.0
14:14:32.573 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
14:14:32.573 INFO HaplotypeCaller - Executing as ngs@ngs-linux on Linux v5.8.0-59-generic amd64
14:14:32.573 INFO HaplotypeCaller - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
14:14:32.573 INFO HaplotypeCaller - Start Date/Time: 2021年11月1日 下午02时14分31秒
14:14:32.573 INFO HaplotypeCaller - ------------------------------------------------------------
14:14:32.573 INFO HaplotypeCaller - ------------------------------------------------------------
14:14:32.574 INFO HaplotypeCaller - HTSJDK Version: 2.24.0
14:14:32.574 INFO HaplotypeCaller - Picard Version: 2.25.0
14:14:32.574 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
14:14:32.574 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
14:14:32.574 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:14:32.574 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:14:32.574 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:14:32.574 INFO HaplotypeCaller - Deflater: IntelDeflater
14:14:32.574 INFO HaplotypeCaller - Inflater: IntelInflater
14:14:32.574 INFO HaplotypeCaller - GCS max retries/reopens: 20
14:14:32.574 INFO HaplotypeCaller - Requester pays: disabled
14:14:32.574 INFO HaplotypeCaller - Initializing engine
14:14:36.824 INFO HaplotypeCaller - Done initializing engine
14:14:36.826 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
14:14:36.866 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
14:14:36.866 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
14:14:36.876 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/home/ngs/biosoft/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
14:14:36.878 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/home/ngs/biosoft/gatk-4.2.0.0/gatk-package-4.2.0.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
14:14:36.927 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
14:14:36.928 INFO IntelPairHmm - Available threads: 8
14:14:36.928 INFO IntelPairHmm - Requested threads: 4
14:14:36.928 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
14:14:37.228 INFO ProgressMeter - Starting traversal
14:14:37.228 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
14:14:38.715 WARN InbreedingCoeff - InbreedingCoeff will not be calculated at position chr1:10439 and possibly subsequent; at least 10 samples must have called genotypes
14:14:47.243 INFO ProgressMeter - chr1:186172 0.2 920 5511.7
14:14:57.278 INFO ProgressMeter - chr1:830665 0.3 3650 10922.7
14:15:05.692 WARN DepthPerSampleHC - Annotation will not be calculated at position chr1:977935 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
14:15:05.692 WARN StrandBiasBySample - Annotation will not be calculated at position chr1:977935 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
14:15:07.307 INFO ProgressMeter - chr1:1001168 0.5 4640 9255.6
14:15:17.364 INFO ProgressMeter - chr1:1225127 0.7 5890 8805.1
...................................................................................

23:42:13.843 INFO ProgressMeter - chr6:61885746 567.6 5518770 9722.8
23:42:23.883 INFO ProgressMeter - chr6:62348074 567.8 5520940 9723.8
23:42:33.905 INFO ProgressMeter - chr6:63002985 567.9 5523550 9725.5
23:42:43.959 INFO ProgressMeter - chr6:63445408 568.1 5525630 9726.3
23:42:53.980 INFO ProgressMeter - chr6:64097380 568.3 5528160 9727.9
23:43:03.981 INFO ProgressMeter - chr6:64620666 568.4 5530460 9729.1
23:43:14.001 INFO ProgressMeter - chr6:65092981 568.6 5532720 9730.2
23:43:24.023 INFO ProgressMeter - chr6:65484919 568.8 5534570 9730.6
23:43:34.123 INFO ProgressMeter - chr6:65881517 568.9 5536530 9731.2
23:43:44.137 INFO ProgressMeter - chr6:66376712 569.1 5538770 9732.2
23:43:54.146 INFO ProgressMeter - chr6:66759727 569.3 5540620 9732.6
23:44:04.211 INFO ProgressMeter - chr6:67127862 569.4 5542540 9733.2
23:44:10.555 WARN DepthPerSampleHC - Annotation will not be calculated at position chr6:67407399 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
23:44:10.555 WARN StrandBiasBySample - Annotation will not be calculated at position chr6:67407399 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
23:44:10.556 WARN DepthPerSampleHC - Annotation will not be calculated at position chr6:67407415 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
23:44:10.556 WARN StrandBiasBySample - Annotation will not be calculated at position chr6:67407415 and possibly subsequent; genotype for sample 8939{JXM}-3 is not called
23:44:14.224 INFO ProgressMeter - chr6:67607778 569.6 5544800 9734.3
23:44:24.280 INFO ProgressMeter - chr6:68147283 569.8 5547230 9735.7
23:44:30.026 INFO VectorLoglessPairHMM - Time spent in setup for JNI call : 27.307544954
23:44:30.027 INFO PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 4768.198119518001
23:44:30.027 INFO SmithWatermanAligner - Total compute time in java Smith-Waterman : 4695.36 sec
23:44:30.027 INFO HaplotypeCaller - Shutting down engine
[2021年11月1日 下午11时44分30秒] org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCaller done. Elapsed time: 569.97 minutes.
Runtime.totalMemory()=742916096
htsjdk.samtools.SAMFormatException: Did not inflate expected amount
at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:147)
at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
at java.io.DataInputStream.read(DataInputStream.java:149)
at htsjdk.samtools.util.BinaryCodec.readBytesOrFewer(BinaryCodec.java:421)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:394)
at htsjdk.samtools.util.BinaryCodec.readBytes(BinaryCodec.java:380)
at htsjdk.samtools.BAMRecordCodec.decode(BAMRecordCodec.java:282)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.getNextRecord(BAMFileReader.java:866)
at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:1005)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:840)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:834)
at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:802)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1058)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1048)
at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1012)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:591)
at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:570)
at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.loadNextRecord(SamReaderQueryingIterator.java:119)
at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.next(SamReaderQueryingIterator.java:156)
at org.broadinstitute.hellbender.utils.iterators.SamReaderQueryingIterator.next(SamReaderQueryingIterator.java:29)
at org.broadinstitute.hellbender.utils.iterators.SAMRecordToReadIterator.next(SAMRecordToReadIterator.java:27)
at org.broadinstitute.hellbender.utils.iterators.SAMRecordToReadIterator.next(SAMRecordToReadIterator.java:13)
at org.broadinstitute.hellbender.utils.iterators.ReadTransformingIterator.next(ReadTransformingIterator.java:42)
at org.broadinstitute.hellbender.utils.iterators.ReadTransformingIterator.next(ReadTransformingIterator.java:14)
at org.broadinstitute.hellbender.utils.iterators.ReadFilteringIterator.loadNextRead(ReadFilteringIterator.java:53)
at org.broadinstitute.hellbender.utils.iterators.ReadFilteringIterator.next(ReadFilteringIterator.java:47)
at org.broadinstitute.hellbender.utils.iterators.ReadFilteringIterator.next(ReadFilteringIterator.java:13)
at org.broadinstitute.hellbender.utils.iterators.ReadTransformingIterator.next(ReadTransformingIterator.java:42)
at org.broadinstitute.hellbender.utils.iterators.ReadTransformingIterator.next(ReadTransformingIterator.java:14)
at org.broadinstitute.hellbender.utils.iterators.PushToPullIterator.fillCache(PushToPullIterator.java:72)
at org.broadinstitute.hellbender.utils.iterators.PushToPullIterator.advanceToNextElement(PushToPullIterator.java:58)
at org.broadinstitute.hellbender.utils.iterators.PushToPullIterator.next(PushToPullIterator.java:52)
at org.broadinstitute.hellbender.utils.iterators.ReadCachingIterator.next(ReadCachingIterator.java:42)
at org.broadinstitute.hellbender.utils.iterators.ReadCachingIterator.next(ReadCachingIterator.java:17)
at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
at org.broadinstitute.hellbender.utils.locusiterator.ReadStateManager.collectPendingReads(ReadStateManager.java:160)
at org.broadinstitute.hellbender.utils.locusiterator.LocusIteratorByState.lazyLoadNextAlignmentContext(LocusIteratorByState.java:315)
at org.broadinstitute.hellbender.utils.locusiterator.LocusIteratorByState.hasNext(LocusIteratorByState.java:252)
at org.broadinstitute.hellbender.utils.locusiterator.IntervalAlignmentContextIterator.advanceAlignmentContext(IntervalAlignmentContextIterator.java:104)
at org.broadinstitute.hellbender.utils.locusiterator.IntervalAlignmentContextIterator.advanceAlignmentContextToCurrentInterval(IntervalAlignmentContextIterator.java:99)
at org.broadinstitute.hellbender.utils.locusiterator.IntervalAlignmentContextIterator.next(IntervalAlignmentContextIterator.java:69)
at org.broadinstitute.hellbender.utils.locusiterator.IntervalAlignmentContextIterator.next(IntervalAlignmentContextIterator.java:21)
at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.loadNextAssemblyRegion(AssemblyRegionIterator.java:120)
at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.next(AssemblyRegionIterator.java:112)
at org.broadinstitute.hellbender.engine.AssemblyRegionIterator.next(AssemblyRegionIterator.java:35)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.processReadShard(AssemblyRegionWalker.java:192)
at org.broadinstitute.hellbender.engine.AssemblyRegionWalker.traverse(AssemblyRegionWalker.java:173)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1058)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)



0

18 comments

Sort by Date Votes
  • Avatar
    Pamela Bretscher

    Hi Jacob Wang,

    Could you take a look at this previous forum post on a similar issue and let me know if the suggestions in it are helpful? It seems that using either the option --use-jdk-inflater or --use-jdk-deflater was successful in solving this error for the previous individual. 

    Kind regards,

    Pamela

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Pamela Bretscher ,

        Thanks for your reply! I tried adding those two options as you suggested, but it seems that this method did not work. The generation of gvcf stoped at the same position as using the command without those two options.

         Here is the command I tried:

    for i in {3..4} ;
    do
    java -Djava.io.tmpdir=/mnt/nas/TMP \
    -jar ${RJLJ}/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar \
    HaplotypeCaller \
    --tmp-dir /mnt/nas/TMP \
    --use-jdk-inflater true \
    --use-jdk-deflater true \
    -R ${CKXLhg38} \
    -I ${ZLJ}/9_BQSRBam/${JXM}-${i}_merged_realigned_bqsr.bam \
    -ERC GVCF \
    -O ${ZLJ}/10_gvcf/${JXM}-${i}_merged_realigned_bqsr.g.vcf

    done

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Pamela Bretscher ,

         Could you give me any other suggestions? I can provide any other information to solve this bug? Does this error only been reported after GATK 4.0?

         Thank you very much in advance!

         Best regards!

    WANG

    PS: for the previous post you mentioned, I think the error was not really solved because the author said there was still this error in log.

        

    0
    Comment actions Permalink
  • Avatar
    Pamela Bretscher

    Hi Jacob Wang,

    Thank you for trying that suggestion. Given that you received the same error, it's possible that your input file is somehow malformed which is causing the error. Could you please try running PrintBGZFBlockInformation on your input bam file to pinpoint if there are issues with the file?

    Kind regards,

    Pamela

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Pamela,

    Thank you for your reply! I will try PrintBGZFBlockInformation as you suggested. I noticed from its help doc that PrintBGZFBlockInformation itself has parameters: --use-jdk-deflater & --use-jdk-inflater. Should I set them as true? Or just use the most simple command (my defaults are IntelDeflater & IntelInflater):

    java -jar /home/ngs/biosoft/gatk-4.2.2.0/gatk-package-4.2.2.0-local.jar PrintBGZFBlockInformation \
    --bgzf-file /mnt/nas/202101_Macrogen_Genome/5_SortedBam/${JXM}-${i}_aligned_fixmated_sorted.bam \
    -O /mnt/nas/202101_Macrogen_Genome/5_SortedBam/${JXM}-${i}_aligned_fixmated_sorted.bam.BlockInfo

    0
    Comment actions Permalink
  • Avatar
    Pamela Bretscher

    Hi Jacob Wang,

    I would try running with the default options first.

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Pamela,

        Thank you for the suggestion. I have run PrintBGZFBlockInformation for the .bam files (after the step of BQSR) of the three samples with the default parameters. The files generated by PrintBGZFBlockInformation are similar in size (~500 Mb each). Each file contains ~ 5 million of lines. I don't know how to check whether there are errors somewhere in bams from those files. So I uploaded to my google drive, could you please take a look of those files?

    https://drive.google.com/drive/folders/1Ls2sPyHcigUiFv-Cb57baFoWKWScOd3k?usp=sharing

    Thank you very much in advance!

    <PS: I encountered inflate error in the samples numbered 3 & 4; the sample numbered 2 went on well in getting gvcf>

     

    0
    Comment actions Permalink
  • Avatar
    Pamela Bretscher

    Hi Jacob Wang,

    Thank you for running that and providing the output. The output looks good, but it still seems that the issue may be with the input bam files. Can you try running samtools view as well as ValidateSamFile on the two bam files causing the issue? If both of these fail then there is an issue with your bam file whereas if ValidateSamFile works, then this is likely a bug with htsjdk. You could also try running samtools index on the bam files and re-running HaplotypeCaller to see if this solves the problem. 

    Kind regards,

    Pamela

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Pamela Bretscher,
    I have run the two commands on my bam files as you suggested. I kept the intermediate bam files (after steps 3.Alignment, 4.Fixmate, 5.Sorting, 6.Deduplication and 9.BQSR) so I have five bam files for each sample.

    1) I ran ValidateSamFile on all those BAM files. It seems that the error only occured in the bam files after BQSR for those two samples.
    Here are the last parts of the log and the error information:
    Jansen-San-2 (the sample of which gvcf can be generated)
    INFO 2021-11-02 20:14:14 SamFileValidator Validated Read 920,000,000 records. Elapsed time: 01:19:10s. Time for last 10,000,000: 57s. Last read position: chr14_GL000225v1_random:137,645
    INFO 2021-11-02 20:15:23 SamFileValidator Validated Read 930,000,000 records. Elapsed time: 01:20:20s. Time for last 10,000,000: 69s. Last read position: KN707862.1:11,951
    [Tue Nov 02 20:42:33 JST 2021] picard.sam.ValidateSamFile done. Elapsed time: 107.51 minutes.
    Runtime.totalMemory()=3863478272

    Jansen-San-3
    INFO 2021-11-02 18:01:19 SamFileValidator Validated Read 370,000,000 records. Elapsed time: 00:32:55s. Time for last 10,000,000: 51s. Last read position: chr6:66,281,342
    ERROR 2021-11-02 18:01:21 ValidateSamFile java.util.zip.DataFormatException: invalid stored block lengths

    Jansen-San-4
    INFO 2021-11-02 18:54:52 SamFileValidator Validated Read 600,000,000 records. Elapsed time: 00:53:26s. Time for last 10,000,000: 51s. Last read position: chr11:94,132,177
    ERROR 2021-11-02 18:55:00 ValidateSamFile java.util.zip.DataFormatException: invalid stored block lengths
    [Tue Nov 02 18:55:00 JST 2021] picard.sam.ValidateSamFile done. Elapsed time: 53.59 minutes.
    Runtime.totalMemory()=2488795136

    For the other bam files of all the samples, there was no error information.

    2) I ran samtools view using the following commend to test:
    for i in {2..4}
    do
    for j in {3..4}
    do
    samtools view -hb -L ErrorRegions_${j}.bed Jansen-San-${i}_merged_realigned_bqsr.bam> Jansen-San-${i}_merged_realigned_bqsr.ErrorRegion${j}.bam
    done
    done

    ErrorRegions_${j}.bed are the bed files covering the chromosome region where the error occured.

    for San-3 and San-4, I got the similar error as generating gvcf files.
    [E::bgzf_uncompress] Inflate operation failed: invalid stored block lengths
    [E::bgzf_read] Read block operation failed with error 1 after 50 of 309 bytes
    samtools view: error reading file "/mnt/nas/202101_Macrogen_Genome/9_BQSRBam/Jansen-San-3_merged_realigned_bqsr.bam"
    samtools view: error closing "/mnt/nas/202101_Macrogen_Genome/9_BQSRBam/Jansen-San-3_merged_realigned_bqsr.bam": -1

    [E::bgzf_uncompress] Inflate operation failed: invalid stored block lengths
    [E::bgzf_read] Read block operation failed with error 1 after 156 of 296 bytes
    samtools view: error reading file "/mnt/nas/202101_Macrogen_Genome/9_BQSRBam/Jansen-San-4_merged_realigned_bqsr.bam"
    samtools view: error closing "/mnt/nas/202101_Macrogen_Genome/9_BQSRBam/Jansen-San-4_merged_realigned_bqsr.bam": -1

    for all the bams of San-2, and other bams of San-3 and San-4 (eg. after step 6.Deduplication), there are no error.

    I suppose that it mean that the error was actually occured during BQSR step? Should I excute BQSR again with the --use-jdk-inflater or --use-jdk-deflater options?

    0
    Comment actions Permalink
  • Avatar
    Pamela Bretscher

    Hi Jacob Wang,

    Thank you for running the validation and providing this output. Yes, it does appear that there is an issue with the block lengths in the bam files rather than a bug causing the issue. Considering one of your files is working properly and the errors are only occurring after the BQSR step, I would recommend re-running BQSR as you suggested. Please let me know if this is successful.

    Kind regards,

    Pamela

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi, Pamela!

         Good news! I think we have solved the problem. I re-run for the two individuals both of the steps of BQSR and HaplotypeCaller with the --use-jdk-inflater or --use-jdk-deflater options, and got gvcf files in correct size with all the chromosomes called.

         It seems that with default setting of Intel inflater/deflater, BQSR could generate an complete bam file,  but some blocks may actually have compression errors. Therefore when HaplotypeCaller tries to read those blocks, the program terminated. 

        I think some other people have also encountered the same problem without an solution. Shall we report this bug somewhere?

        Best regards!

    WANG

     

    0
    Comment actions Permalink
  • Avatar
    Pamela Bretscher
    • Edited

    Hi Jacob,

    I'm glad to hear that it is working for you now! Yes, I can create a github ticket for this issue so the GATK team can look into it further! In the meantime, thank you for sharing your solution here and please let me know if you have any additional questions. Here is the link to the issue report so you can follow along: https://github.com/broadinstitute/gatk/issues/7582 

    Kind regards,

    Pamela

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Thanks, Pamela. I ran the following steps (from merging gvcfs to annotation) using the gvcfs in the past weekend. No problem happened. I think for now I can say the problem has been fixed. I will post my solution in the git page!  Thanks again! Nice to work together with you!

    0
    Comment actions Permalink
  • Avatar
    Yuxiang Yang

    Sorry to trouble you here. I have some problems here. Even when I use the parameter

    --use-jdk-deflater true; --use-jdk-inflater true. The comprehensive command is as below with gatke 4.*:

    ############

    time $bwa mem -t 16 -M -R "@RG\tID:$RGID\tPL:ILLUMINA\tPU:$PU\tLB:$library\tSM:$sample" $reference \
    $fq1 $fq2 | $samtools view -Sb - > $outdir/bwa/${sample}.bam && \
    echo "** BWA MEM done **" && \
    time $samtools sort -@ 16 -m 2G -O bam -o $outdir/bwa/${sample}.sorted.bam $outdir/bwa/${sample}.bam && echo "** sorted raw bamfile done **"

    time $samtools index $outdir/bwa/${sample}.sorted.bam && echo "** ${sample}.sorted.bam index done **"


    $gatk MarkDuplicates \
    -I $outdir/bwa/${sample}.sorted.bam \
    -M $outdir/bwa/${sample}.markdup_metrics.txt \
    -O $outdir/bwa/${sample}.sorted.markdup.bam && echo "** ${sample}.sorted.bam MarkDuplicates done **"


    time $samtools index $outdir/bwa/${sample}.sorted.markdup.bam && echo "** ${sample}.sorted.markdup.bam index done **"


    time $gatk BaseRecalibrator \
    -R $reference \
    -I $outdir/bwa/${sample}.sorted.markdup.bam \
    --use-jdk-deflater true \
    --use-jdk-inflater true \
    --known-sites $GATK_bundle/gallus_gallus.vcf \
    --known-sites $GATK_bundle/gallus_gallus_incl_consequences.vcf \
    -O $outdir/bwa/${sample}.sorted.markdup.recal_data.table && echo "** ${sample}.sorted.markdup.recal_data.table done **"


    time $gatk ApplyBQSR \
    --bqsr-recal-file $outdir/bwa/${sample}.sorted.markdup.recal_data.table \
    -R $reference \
    --use-jdk-deflater true \
    --use-jdk-inflater true \
    -I $outdir/bwa/${sample}.sorted.markdup.bam \
    -O $outdir/bwa/${sample}.sorted.markdup.BQSR.bam && echo "** ApplyBQSR done **"


    time $samtools index $outdir/bwa/${sample}.sorted.markdup.BQSR.bam && echo "** ${sample}.sorted.markdup.BQSR.bam index done **"


    time $gatk HaplotypeCaller \
    --emit-ref-confidence GVCF \
    --native-pair-hmm-threads 16 \
    -R $reference \
    --use-jdk-deflater true \
    --use-jdk-inflater true \
    -I $outdir/bwa/${sample}.sorted.markdup.BQSR.bam \
    -O $outdir/gatk/${sample}.HC.g.vcf.gz && echo "** GVCF ${sample}.HC.g.vcf.gz done **"

    ########################

    It stills has great chance to fail in ValidateVariants check, output the error as below:

    ##########

    [sc94039@bscc-a3:/public1/home/sc94039 $~]gatk ValidateVariants -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -V /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD043E05/gatk/CD043E05.HC.g.vcf.gz --validation-type-to-exclude ALLELES --dbsnp /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ValidateVariants -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -V /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD043E05/gatk/CD043E05.HC.g.vcf.gz --validation-type-to-exclude ALLELES --dbsnp /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    14:52:56.213 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    Jan 15, 2022 2:53:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    14:53:06.525 INFO ValidateVariants - ------------------------------------------------------------
    14:53:06.525 INFO ValidateVariants - The Genome Analysis Toolkit (GATK) v4.2.4.0
    14:53:06.526 INFO ValidateVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
    14:53:06.527 INFO ValidateVariants - Executing as sc94039@eb0802.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    14:53:06.527 INFO ValidateVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    14:53:06.527 INFO ValidateVariants - Start Date/Time: January 15, 2022 at 2:52:56 PM CST
    14:53:06.527 INFO ValidateVariants - ------------------------------------------------------------
    14:53:06.527 INFO ValidateVariants - ------------------------------------------------------------
    14:53:06.539 INFO ValidateVariants - HTSJDK Version: 2.24.1
    14:53:06.539 INFO ValidateVariants - Picard Version: 2.25.4
    14:53:06.539 INFO ValidateVariants - Built for Spark Version: 2.4.5
    14:53:06.539 INFO ValidateVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    14:53:06.540 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    14:53:06.540 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    14:53:06.540 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    14:53:06.540 INFO ValidateVariants - Deflater: IntelDeflater
    14:53:06.540 INFO ValidateVariants - Inflater: IntelInflater
    14:53:06.540 INFO ValidateVariants - GCS max retries/reopens: 20
    14:53:06.540 INFO ValidateVariants - Requester pays: disabled
    14:53:06.540 INFO ValidateVariants - Initializing engine
    14:53:06.807 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    14:53:06.997 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD043E05/gatk/CD043E05.HC.g.vcf.gz
    14:53:07.247 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    14:53:07.262 INFO ValidateVariants - Done initializing engine
    14:53:07.273 INFO ProgressMeter - Starting traversal
    14:53:07.274 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute
    14:53:13.226 INFO ValidateVariants - Shutting down engine
    [January 15, 2022 at 2:53:13 PM CST] org.broadinstitute.hellbender.tools.walkers.variantutils.ValidateVariants done. Elapsed time: 0.29 minutes.
    Runtime.totalMemory()=2147483648
    java.lang.RuntimeException: Invalid deflate block found.
    at com.intel.gkl.compression.IntelInflater.inflateNative(Native Method)
    at com.intel.gkl.compression.IntelInflater.inflate(IntelInflater.java:174)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:145)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
    at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
    at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
    at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
    at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:257)
    at htsjdk.tribble.readers.PositionalBufferedStream.fill(PositionalBufferedStream.java:132)
    at htsjdk.tribble.readers.PositionalBufferedStream.read(PositionalBufferedStream.java:84)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
    at htsjdk.tribble.readers.LongLineBufferedReader.fill(LongLineBufferedReader.java:140)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:300)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:356)
    at htsjdk.tribble.readers.SynchronousLineReader.readLine(SynchronousLineReader.java:51)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:205)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:149)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at org.broadinstitute.hellbender.engine.VariantWalker.traverse(VariantWalker.java:102)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1085)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
    at org.broadinstitute.hellbender.Main.main(Main.java:289)

    #################

    Also when I check with ValidateSamFile

    ##########

    [sc94039@bscc-a3:/public1/home/sc94039 $~]java -jar /public1/home/sc94039/software/picard.jar ValidateSamFile -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A03/bwa/CD044A03.sorted.markdup.BQSR.bam -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -MODE SUMMARY
    15:09:47.725 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/public1/home/sc94039/software/picard.jar!/com/intel/gkl/native/libgkl_compression.so
    [Sat Jan 15 15:09:47 CST 2022] ValidateSamFile --INPUT /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A03/bwa/CD044A03.sorted.markdup.BQSR.bam --MODE SUMMARY --REFERENCE_SEQUENCE /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa --MAX_OUTPUT 100 --IGNORE_WARNINGS false --VALIDATE_INDEX true --INDEX_VALIDATION_STRINGENCY EXHAUSTIVE --IS_BISULFITE_SEQUENCED false --MAX_OPEN_TEMP_FILES 8000 --SKIP_MATE_VALIDATION false --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
    [Sat Jan 15 15:09:47 CST 2022] Executing as sc94039@ea0315.para.bscc on Linux 3.10.0-1160.11.1.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 11.0.13+10-LTS-370; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: Version:2.26.10

    #################

    It refers here with "Deflater: Intel; Inflater: Intel", why it still output with Deflater intel? Can you help me on it?

    Thank you!

    0
    Comment actions Permalink
  • Avatar
    Genevieve Brandt (she/her)

    Hi Yuxiang Yang,

    Can you isolate what point in the pipeline this error comes up? You can use ValidateSamFile to check your files before and after each pre-processing step.

    Best,

    Genevieve

    0
    Comment actions Permalink
  • Avatar
    Jacob Wang

    Hi Yuxiang

        I read your post and I want to confirm with you on the following points.

       1) You used "--use-jdk-deflater true --use-jdk-inflater true " in your GATK pipeline (BaseRecalibrator, ApplyBQSR, HaplotypeCaller). Did you got any error in those steps? If you kept the log of processing those step, please check which Deflater/Inflater were used.

       2) The error logs in your post were from ValidateVariants & ValidateSamFile. It seems that in your commands (those two commands) you did not add "--use-jdk-deflater true --use-jdk-inflater true ". I think that was why Intel Deflater/Inflater were used and got errors. Please add those two options for those two commands and run again.

        Best,

    WANG

    0
    Comment actions Permalink
  • Avatar
    Yuxiang Yang

    Thank you for your kind help here.

    For your first reply, I check that I have used the JDK inflater and deflater, please see the log below:

    ##################

    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar MarkDuplicates -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.bam -M /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.markdup_metrics.txt -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam

    real 0m21.410s
    user 0m21.055s
    sys 0m0.271s

    real 0m21.341s
    user 0m21.059s
    sys 0m0.264s
    Jan 20, 2022 11:29:10 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    Jan 20, 2022 11:29:10 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:29:10.101 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.102 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.103 INFO BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:29:10.103 INFO BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:29:10.104 INFO BaseRecalibrator - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:29:10.104 INFO BaseRecalibrator - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:29:10.104 INFO BaseRecalibrator - Start Date/Time: January 20, 2022 at 11:29:09 PM CST
    23:29:10.104 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.104 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.104 INFO BaseRecalibrator - HTSJDK Version: 2.24.1
    23:29:10.104 INFO BaseRecalibrator - Picard Version: 2.25.4
    23:29:10.104 INFO BaseRecalibrator - Built for Spark Version: 2.4.5
    23:29:10.104 INFO BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:29:10.105 INFO BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:29:10.105 INFO BaseRecalibrator - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:29:10.105 INFO BaseRecalibrator - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:29:10.105 INFO BaseRecalibrator - Start Date/Time: January 20, 2022 at 11:29:08 PM CST
    23:29:10.105 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.105 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:10.105 INFO BaseRecalibrator - HTSJDK Version: 2.24.1
    23:29:10.106 INFO BaseRecalibrator - Picard Version: 2.25.4
    23:29:10.106 INFO BaseRecalibrator - Built for Spark Version: 2.4.5
    23:29:10.106 INFO BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:29:10.106 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:29:10.106 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:29:10.106 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:29:10.107 INFO BaseRecalibrator - Deflater: JdkDeflater
    23:29:10.107 INFO BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:29:10.107 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:29:10.107 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:29:10.107 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:29:10.107 INFO BaseRecalibrator - Inflater: JdkInflater
    23:29:10.107 INFO BaseRecalibrator - GCS max retries/reopens: 20
    23:29:10.107 INFO BaseRecalibrator - Requester pays: disabled
    23:29:10.108 INFO BaseRecalibrator - Initializing engine
    23:29:10.108 INFO BaseRecalibrator - Deflater: JdkDeflater
    23:29:10.108 INFO BaseRecalibrator - Inflater: JdkInflater
    23:29:10.108 INFO BaseRecalibrator - GCS max retries/reopens: 20
    23:29:10.108 INFO BaseRecalibrator - Requester pays: disabled
    23:29:10.109 INFO BaseRecalibrator - Initializing engine
    23:29:10.265 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    23:29:10.266 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    23:29:10.299 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf
    23:29:10.299 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf
    23:29:10.316 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:10.316 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:10.324 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:10.325 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:10.339 INFO BaseRecalibrator - Done initializing engine
    23:29:10.340 INFO BaseRecalibrator - Done initializing engine
    23:29:10.341 INFO BaseRecalibrationEngine - The covariates being used here:
    23:29:10.341 INFO BaseRecalibrationEngine - ReadGroupCovariate
    23:29:10.341 INFO BaseRecalibrationEngine - QualityScoreCovariate
    23:29:10.341 INFO BaseRecalibrationEngine - ContextCovariate
    23:29:10.341 INFO BaseRecalibrationEngine - CycleCovariate
    23:29:10.342 INFO BaseRecalibrationEngine - The covariates being used here:
    23:29:10.342 INFO BaseRecalibrationEngine - ReadGroupCovariate
    23:29:10.342 INFO BaseRecalibrationEngine - QualityScoreCovariate
    23:29:10.342 INFO BaseRecalibrationEngine - ContextCovariate
    23:29:10.342 INFO BaseRecalibrationEngine - CycleCovariate
    23:29:10.349 INFO ProgressMeter - Starting traversal
    23:29:10.349 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:29:10.349 INFO ProgressMeter - Starting traversal
    23:29:10.350 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:29:20.353 INFO ProgressMeter - Z:44777516 0.2 233000 1397720.5
    23:29:20.360 INFO ProgressMeter - Z:44993179 0.2 234000 1402597.4

    real 0m21.556s
    user 0m21.178s
    sys 0m0.332s

    real 0m21.546s
    user 0m21.189s
    sys 0m0.343s
    Jan 20, 2022 11:29:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:29:26.858 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.861 INFO BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:29:26.861 INFO BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:29:26.861 INFO BaseRecalibrator - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:29:26.861 INFO BaseRecalibrator - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:29:26.861 INFO BaseRecalibrator - Start Date/Time: January 20, 2022 at 11:29:23 PM CST
    Jan 20, 2022 11:29:26 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:29:26.861 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.861 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.862 INFO BaseRecalibrator - HTSJDK Version: 2.24.1
    23:29:26.862 INFO BaseRecalibrator - Picard Version: 2.25.4
    23:29:26.862 INFO BaseRecalibrator - Built for Spark Version: 2.4.5
    23:29:26.863 INFO BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:29:26.864 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:29:26.864 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:29:26.864 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:29:26.863 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.864 INFO BaseRecalibrator - Deflater: JdkDeflater
    23:29:26.865 INFO BaseRecalibrator - Inflater: JdkInflater
    23:29:26.865 INFO BaseRecalibrator - GCS max retries/reopens: 20
    23:29:26.865 INFO BaseRecalibrator - Requester pays: disabled
    23:29:26.865 INFO BaseRecalibrator - Initializing engine
    23:29:26.866 INFO BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:29:26.866 INFO BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:29:26.867 INFO BaseRecalibrator - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:29:26.867 INFO BaseRecalibrator - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:29:26.867 INFO BaseRecalibrator - Start Date/Time: January 20, 2022 at 11:29:22 PM CST
    23:29:26.867 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.867 INFO BaseRecalibrator - ------------------------------------------------------------
    23:29:26.867 INFO BaseRecalibrator - HTSJDK Version: 2.24.1
    23:29:26.868 INFO BaseRecalibrator - Picard Version: 2.25.4
    23:29:26.868 INFO BaseRecalibrator - Built for Spark Version: 2.4.5
    23:29:26.869 INFO BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:29:26.869 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:29:26.869 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:29:26.869 INFO BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:29:26.870 INFO BaseRecalibrator - Deflater: JdkDeflater
    23:29:26.871 INFO BaseRecalibrator - Inflater: JdkInflater
    23:29:26.871 INFO BaseRecalibrator - GCS max retries/reopens: 20
    23:29:26.871 INFO BaseRecalibrator - Requester pays: disabled
    23:29:26.871 INFO BaseRecalibrator - Initializing engine
    23:29:27.024 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    23:29:27.027 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    23:29:27.052 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf
    23:29:27.055 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf
    23:29:27.066 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:27.068 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:27.074 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:27.077 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    23:29:27.091 INFO BaseRecalibrator - Done initializing engine
    23:29:27.093 INFO BaseRecalibrationEngine - The covariates being used here:
    23:29:27.093 INFO BaseRecalibrationEngine - ReadGroupCovariate
    23:29:27.093 INFO BaseRecalibrationEngine - QualityScoreCovariate
    23:29:27.093 INFO BaseRecalibrationEngine - ContextCovariate
    23:29:27.093 INFO BaseRecalibrationEngine - CycleCovariate
    23:29:27.093 INFO BaseRecalibrator - Done initializing engine
    23:29:27.095 INFO BaseRecalibrationEngine - The covariates being used here:
    23:29:27.096 INFO BaseRecalibrationEngine - ReadGroupCovariate
    23:29:27.096 INFO BaseRecalibrationEngine - QualityScoreCovariate
    23:29:27.096 INFO BaseRecalibrationEngine - ContextCovariate
    23:29:27.096 INFO BaseRecalibrationEngine - CycleCovariate
    23:29:27.100 INFO ProgressMeter - Starting traversal
    23:29:27.101 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:29:27.103 INFO ProgressMeter - Starting traversal
    23:29:27.103 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:29:30.353 INFO ProgressMeter - 1:14054124 0.3 527000 1580762.9
    23:29:30.372 INFO ProgressMeter - 1:14378921 0.3 530000 1588252.9
    23:29:37.118 INFO ProgressMeter - Z:45465596 0.2 236000 1413738.0
    23:29:37.118 INFO ProgressMeter - Z:43928706 0.2 229000 1372079.1
    23:29:40.359 INFO ProgressMeter - 1:65315070 0.5 1005000 2009397.2
    23:29:40.371 INFO ProgressMeter - 1:65984285 0.5 1011000 2020518.3
    23:29:47.121 INFO ProgressMeter - 1:14925473 0.3 535000 1603396.6
    23:29:47.132 INFO ProgressMeter - 1:12350135 0.3 511000 1530856.8
    23:29:50.359 INFO ProgressMeter - 1:119094602 0.7 1489000 2232997.6
    23:29:50.376 INFO ProgressMeter - 1:120192332 0.7 1499000 2246983.3
    23:29:57.133 INFO ProgressMeter - 1:63153763 0.5 984000 1966034.0
    23:29:57.136 INFO ProgressMeter - 1:65984285 0.5 1011000 2019643.7
    23:30:00.360 INFO ProgressMeter - 1:170801916 0.8 1974000 2368326.3
    23:30:00.377 INFO ProgressMeter - 1:172250872 0.8 1987000 2383065.5
    23:30:07.136 INFO ProgressMeter - 1:115839649 0.7 1458000 2185197.2
    23:30:07.145 INFO ProgressMeter - 1:118865479 0.7 1487000 2228049.1
    23:30:10.385 INFO ProgressMeter - 2:20134622 1.0 2397000 2395562.7
    23:30:10.385 INFO ProgressMeter - 2:19068868 1.0 2387000 2385608.4
    23:30:17.152 INFO ProgressMeter - 1:169915374 0.8 1966000 2356843.2
    23:30:17.154 INFO ProgressMeter - 1:166510623 0.8 1935000 2319634.0
    23:30:20.389 INFO ProgressMeter - 2:57028051 1.2 2740000 2347263.7
    23:30:20.404 INFO ProgressMeter - 2:58454704 1.2 2753000 2357895.3
    23:30:27.158 INFO ProgressMeter - 2:15032907 1.0 2350000 2347847.8
    23:30:27.179 INFO ProgressMeter - 2:18019532 1.0 2377000 2373913.9
    23:30:30.401 INFO ProgressMeter - 2:95086822 1.3 3098000 2322019.7
    23:30:30.412 INFO ProgressMeter - 2:96834903 1.3 3114000 2333662.2
    23:30:37.172 INFO ProgressMeter - 2:50973555 1.2 2687000 2300907.7
    23:30:37.199 INFO ProgressMeter - 2:55687415 1.2 2728000 2335016.7
    23:30:40.424 INFO ProgressMeter - 2:132551428 1.5 3450000 2298136.0
    23:30:40.428 INFO ProgressMeter - 2:134430915 1.5 3468000 2309972.4
    23:30:47.171 INFO ProgressMeter - 2:88543514 1.3 3036000 2275066.2
    23:30:47.216 INFO ProgressMeter - 2:93370241 1.3 3081000 2307433.1
    23:30:50.441 INFO ProgressMeter - 3:18939400 1.7 3774000 2262341.3
    23:30:50.450 INFO ProgressMeter - 3:17513883 1.7 3761000 2254368.2
    23:30:57.198 INFO ProgressMeter - 2:124506334 1.5 3378000 2249650.4
    23:30:57.245 INFO ProgressMeter - 2:129975895 1.5 3429000 2282373.6
    23:31:00.465 INFO ProgressMeter - 3:49389014 1.8 4059000 2211667.7
    23:31:00.474 INFO ProgressMeter - 3:47778213 1.8 4044000 2203334.4
    23:31:07.227 INFO ProgressMeter - 3:10377988 1.7 3696000 2214853.6
    23:31:07.264 INFO ProgressMeter - 3:15566448 1.7 3744000 2242744.3
    23:31:10.473 INFO ProgressMeter - 3:81145605 2.0 4361000 2178249.1
    23:31:10.507 INFO ProgressMeter - 3:79369392 2.0 4344000 2169162.0
    23:31:17.254 INFO ProgressMeter - 3:40047110 1.8 3971000 2163030.7
    23:31:17.271 INFO ProgressMeter - 3:45683425 1.8 4024000 2191522.2
    23:31:20.537 INFO ProgressMeter - 4:2128736 2.2 4653000 2144453.7
    23:31:20.548 INFO ProgressMeter - 4:636466 2.2 4642000 2139219.8
    23:31:27.281 INFO ProgressMeter - 3:69837666 2.0 4253000 2123350.4
    23:31:27.302 INFO ProgressMeter - 3:76903549 2.0 4321000 2156887.2
    23:31:30.553 INFO ProgressMeter - 4:28447637 2.3 4894000 2094376.8
    23:31:30.558 INFO ProgressMeter - 4:26402539 2.3 4874000 2085758.3
    23:31:37.293 INFO ProgressMeter - 3:100677609 2.2 4548000 2096029.6
    23:31:37.323 INFO ProgressMeter - 3:108368181 2.2 4615000 2126368.8
    23:31:40.559 INFO ProgressMeter - 4:52330271 2.5 5117000 2043938.5
    23:31:40.590 INFO ProgressMeter - 4:50828153 2.5 5103000 2037939.3
    23:31:47.301 INFO ProgressMeter - 4:15464910 2.3 4776000 2043966.4
    23:31:47.328 INFO ProgressMeter - 4:23708121 2.3 4848000 2074350.9
    23:31:50.566 INFO ProgressMeter - 4:77551575 2.7 5360000 2007277.6
    23:31:50.616 INFO ProgressMeter - 4:75961353 2.7 5345000 2001048.3
    23:31:57.302 INFO ProgressMeter - 4:40543101 2.5 5007000 2000146.5
    23:31:57.355 INFO ProgressMeter - 4:48053298 2.5 5078000 2027779.8
    23:32:00.577 INFO ProgressMeter - 5:12045635 2.8 5590000 1970298.7
    23:32:00.656 INFO ProgressMeter - 5:10486247 2.8 5575000 1964111.7
    23:32:07.329 INFO ProgressMeter - 4:63942401 2.7 5229000 1958109.2
    23:32:07.366 INFO ProgressMeter - 4:72463333 2.7 5311000 1988331.8
    23:32:10.597 INFO ProgressMeter - 5:37228207 3.0 5825000 1938995.2
    23:32:10.678 INFO ProgressMeter - 5:35499603 3.0 5808000 1932478.6
    23:32:17.363 INFO ProgressMeter - 4:87884861 2.8 5457000 1923058.9
    23:32:17.366 INFO ProgressMeter - 5:7416713 2.8 5546000 1954365.3
    23:32:20.648 INFO ProgressMeter - 6:3110712 3.2 6053000 1908470.4
    23:32:20.718 INFO ProgressMeter - 6:1422786 3.2 6037000 1902735.8
    23:32:27.384 INFO ProgressMeter - 5:31279591 3.0 5766000 1918993.6
    23:32:27.384 INFO ProgressMeter - 5:21601509 3.0 5677000 1889383.8
    23:32:30.662 INFO ProgressMeter - 6:26717131 3.3 6266000 1876862.7
    23:32:30.741 INFO ProgressMeter - 6:24994735 3.3 6250000 1871341.5
    23:32:37.400 INFO ProgressMeter - 5:45820646 3.2 5905000 1861826.5
    23:32:37.436 INFO ProgressMeter - 5:56370181 3.2 6003000 1892347.7
    23:32:40.694 INFO ProgressMeter - 7:18573595 3.5 6520000 1859801.8
    23:32:40.772 INFO ProgressMeter - 7:16215140 3.5 6498000 1852848.1
    23:32:47.441 INFO ProgressMeter - 6:8888234 3.3 6107000 1829018.1
    23:32:47.466 INFO ProgressMeter - 6:21151842 3.3 6216000 1861402.9
    23:32:50.717 INFO ProgressMeter - 8:10292616 3.7 6776000 1844922.3
    23:32:50.816 INFO ProgressMeter - 8:7657693 3.7 6755000 1838378.7
    23:32:57.457 INFO ProgressMeter - 6:32473689 3.5 6319000 1802390.3
    23:32:57.490 INFO ProgressMeter - 7:11400835 3.5 6453000 1840305.3
    23:33:00.730 INFO ProgressMeter - 9:6113588 3.8 7005000 1824369.2
    23:33:00.831 INFO ProgressMeter - 9:4216096 3.8 6989000 1819412.4
    23:33:07.459 INFO ProgressMeter - 7:25057430 3.7 6580000 1791646.2
    23:33:07.490 INFO ProgressMeter - 8:3663564 3.7 6721000 1829764.6
    23:33:10.755 INFO ProgressMeter - 10:8164813 4.0 7234000 1805445.8
    23:33:10.852 INFO ProgressMeter - 10:5750789 4.0 7211000 1798987.1
    23:33:17.481 INFO ProgressMeter - 8:15683218 3.8 6827000 1778034.4
    23:33:17.498 INFO ProgressMeter - 8:29348846 3.8 6951000 1810187.7
    23:33:20.779 INFO ProgressMeter - 11:18926293 4.2 7519000 1801461.5
    23:33:20.864 INFO ProgressMeter - 11:16512670 4.2 7499000 1796067.3
    23:33:27.485 INFO ProgressMeter - 9:9272317 4.0 7035000 1755955.1
    23:33:27.546 INFO ProgressMeter - 9:23511295 4.0 7164000 1787685.3
    23:33:30.787 INFO ProgressMeter - 13:4685872 4.3 7736000 1782235.2
    23:33:30.866 INFO ProgressMeter - 13:2922418 4.3 7721000 1778240.1
    23:33:37.512 INFO ProgressMeter - 10:11678730 4.2 7267000 1741231.3
    23:33:37.548 INFO ProgressMeter - 11:10439872 4.2 7441000 1782652.6
    23:33:40.819 INFO ProgressMeter - 14:10147357 4.5 7952000 1764040.4
    23:33:40.877 INFO ProgressMeter - 14:8205602 4.5 7934000 1759683.0
    23:33:47.545 INFO ProgressMeter - 12:1763233 4.3 7541000 1737277.4
    23:33:47.590 INFO ProgressMeter - 12:16987211 4.3 7677000 1768289.6
    23:33:50.841 INFO ProgressMeter - 17:8055564 4.7 8191000 1752135.5
    23:33:50.914 INFO ProgressMeter - 17:6303868 4.7 8177000 1748691.9
    23:33:57.545 INFO ProgressMeter - 13:5655305 4.5 7745000 1718298.2
    23:33:57.605 INFO ProgressMeter - 14:3431477 4.5 7894000 1750953.8
    23:34:00.862 INFO ProgressMeter - 20:1967578 4.8 8409000 1736726.9
    23:34:00.930 INFO ProgressMeter - 19:9435198 4.8 8386000 1731571.3
    23:34:07.585 INFO ProgressMeter - 14:10147357 4.7 7952000 1701071.7
    23:34:07.632 INFO ProgressMeter - 17:1450648 4.7 8137000 1740342.4
    23:34:10.866 INFO ProgressMeter - 23:4257492 5.0 8645000 1726031.2
    23:34:10.963 INFO ProgressMeter - 23:2425918 5.0 8630000 1722480.4
    23:34:17.614 INFO ProgressMeter - 17:7280744 4.8 8185000 1690469.6
    23:34:17.638 INFO ProgressMeter - 19:4172747 4.8 8342000 1722746.9
    23:34:20.887 INFO ProgressMeter - 28:4489547 5.2 8857000 1711288.2
    23:34:20.999 INFO ProgressMeter - 28:1945667 5.2 8839000 1707200.1
    23:34:24.923 INFO BaseRecalibrator - 354797 read(s) filtered by: MappingQualityNotZeroReadFilter
    0 read(s) filtered by: MappingQualityAvailableReadFilter
    0 read(s) filtered by: MappedReadFilter
    15338 read(s) filtered by: NotSecondaryAlignmentReadFilter
    208784 read(s) filtered by: NotDuplicateReadFilter
    0 read(s) filtered by: PassesVendorQualityCheckReadFilter
    0 read(s) filtered by: WellformedReadFilter
    578919 total reads filtered
    23:34:24.924 INFO ProgressMeter - AADN05001514.1:518 5.2 9023952 1721175.7
    23:34:24.924 INFO ProgressMeter - Traversal complete. Processed 9023952 total reads in 5.2 minutes.
    23:34:25.054 INFO BaseRecalibrator - Calculating quantized quality scores...
    23:34:25.062 INFO BaseRecalibrator - Writing recalibration report...
    23:34:25.859 INFO BaseRecalibrator - 354797 read(s) filtered by: MappingQualityNotZeroReadFilter
    0 read(s) filtered by: MappingQualityAvailableReadFilter
    0 read(s) filtered by: MappedReadFilter
    15338 read(s) filtered by: NotSecondaryAlignmentReadFilter
    208784 read(s) filtered by: NotDuplicateReadFilter
    0 read(s) filtered by: PassesVendorQualityCheckReadFilter
    0 read(s) filtered by: WellformedReadFilter
    578919 total reads filtered
    23:34:25.859 INFO ProgressMeter - AADN05001514.1:518 5.3 9023952 1716075.0
    23:34:25.859 INFO ProgressMeter - Traversal complete. Processed 9023952 total reads in 5.3 minutes.
    23:34:25.983 INFO BaseRecalibrator - Calculating quantized quality scores...
    23:34:25.992 INFO BaseRecalibrator - Writing recalibration report...
    23:34:26.069 INFO BaseRecalibrator - ...done!
    23:34:26.069 INFO BaseRecalibrator - BaseRecalibrator was able to recalibrate 9023952 reads
    23:34:26.069 INFO BaseRecalibrator - Shutting down engine
    [January 20, 2022 at 11:34:26 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 5.28 minutes.
    Runtime.totalMemory()=2579496960
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar BaseRecalibrator -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam --use-jdk-deflater true --use-jdk-inflater true --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table

    real 5m18.124s
    user 5m27.833s
    sys 0m6.844s
    23:34:27.023 INFO BaseRecalibrator - ...done!
    23:34:27.023 INFO BaseRecalibrator - BaseRecalibrator was able to recalibrate 9023952 reads
    23:34:27.024 INFO BaseRecalibrator - Shutting down engine
    [January 20, 2022 at 11:34:27 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 5.31 minutes.
    Runtime.totalMemory()=2579496960
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar BaseRecalibrator -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam --use-jdk-deflater true --use-jdk-inflater true --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table

    real 5m20.347s
    user 5m31.644s
    sys 0m5.287s
    23:34:27.636 INFO ProgressMeter - 19:9551572 5.0 8387000 1674425.1
    23:34:27.643 INFO ProgressMeter - 22:1194530 5.0 8579000 1712705.7
    23:34:37.655 INFO ProgressMeter - 27:3692766 5.2 8794000 1699028.2
    23:34:37.672 INFO ProgressMeter - 23:1778838 5.2 8626000 1666494.9
    Jan 20, 2022 11:34:37 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    Jan 20, 2022 11:34:37 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:34:37.751 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.752 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.754 INFO ApplyBQSR - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:34:37.754 INFO ApplyBQSR - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:34:37.754 INFO ApplyBQSR - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:34:37.754 INFO ApplyBQSR - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:34:37.754 INFO ApplyBQSR - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:34:37.755 INFO ApplyBQSR - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:34:37.755 INFO ApplyBQSR - Start Date/Time: January 20, 2022 at 11:34:27 PM CST
    23:34:37.755 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.755 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.755 INFO ApplyBQSR - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:34:37.755 INFO ApplyBQSR - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:34:37.755 INFO ApplyBQSR - HTSJDK Version: 2.24.1
    23:34:37.755 INFO ApplyBQSR - Picard Version: 2.25.4
    23:34:37.755 INFO ApplyBQSR - Start Date/Time: January 20, 2022 at 11:34:28 PM CST
    23:34:37.755 INFO ApplyBQSR - Built for Spark Version: 2.4.5
    23:34:37.755 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.755 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:37.756 INFO ApplyBQSR - HTSJDK Version: 2.24.1
    23:34:37.756 INFO ApplyBQSR - Picard Version: 2.25.4
    23:34:37.756 INFO ApplyBQSR - Built for Spark Version: 2.4.5
    23:34:37.757 INFO ApplyBQSR - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:34:37.757 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:34:37.757 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:34:37.757 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:34:37.758 INFO ApplyBQSR - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:34:37.758 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:34:37.758 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:34:37.758 INFO ApplyBQSR - Deflater: JdkDeflater
    23:34:37.758 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:34:37.758 INFO ApplyBQSR - Inflater: JdkInflater
    23:34:37.758 INFO ApplyBQSR - GCS max retries/reopens: 20
    23:34:37.758 INFO ApplyBQSR - Requester pays: disabled
    23:34:37.758 INFO ApplyBQSR - Initializing engine
    23:34:37.759 INFO ApplyBQSR - Deflater: JdkDeflater
    23:34:37.759 INFO ApplyBQSR - Inflater: JdkInflater
    23:34:37.759 INFO ApplyBQSR - GCS max retries/reopens: 20
    23:34:37.759 INFO ApplyBQSR - Requester pays: disabled
    23:34:37.759 INFO ApplyBQSR - Initializing engine
    23:34:37.897 INFO ApplyBQSR - Done initializing engine
    23:34:37.898 INFO ApplyBQSR - Done initializing engine
    23:34:38.310 INFO ProgressMeter - Starting traversal
    23:34:38.310 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:34:38.310 INFO ProgressMeter - Starting traversal
    23:34:38.311 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:34:44.734 INFO BaseRecalibrator - 354797 read(s) filtered by: MappingQualityNotZeroReadFilter
    0 read(s) filtered by: MappingQualityAvailableReadFilter
    0 read(s) filtered by: MappedReadFilter
    15338 read(s) filtered by: NotSecondaryAlignmentReadFilter
    208784 read(s) filtered by: NotDuplicateReadFilter
    0 read(s) filtered by: PassesVendorQualityCheckReadFilter
    0 read(s) filtered by: WellformedReadFilter
    578919 total reads filtered
    23:34:44.734 INFO ProgressMeter - AADN05001514.1:518 5.3 9023952 1704599.7
    23:34:44.734 INFO ProgressMeter - Traversal complete. Processed 9023952 total reads in 5.3 minutes.
    23:34:44.853 INFO BaseRecalibrator - Calculating quantized quality scores...
    23:34:44.862 INFO BaseRecalibrator - Writing recalibration report...
    23:34:45.848 INFO BaseRecalibrator - ...done!
    23:34:45.849 INFO BaseRecalibrator - BaseRecalibrator was able to recalibrate 9023952 reads
    23:34:45.849 INFO BaseRecalibrator - Shutting down engine
    [January 20, 2022 at 11:34:45 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 5.38 minutes.
    Runtime.totalMemory()=2650800128
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar BaseRecalibrator -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam --use-jdk-deflater true --use-jdk-inflater true --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table

    real 5m24.301s
    user 5m30.737s
    sys 0m6.257s
    23:34:47.678 INFO ProgressMeter - 27:7905264 5.3 8824000 1651532.4
    23:34:48.316 INFO ProgressMeter - 1:10215962 0.2 540000 3238704.5
    23:34:48.320 INFO ProgressMeter - 1:10422677 0.2 542000 3249400.5
    Jan 20, 2022 11:34:48 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:34:48.990 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:48.993 INFO ApplyBQSR - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:34:48.993 INFO ApplyBQSR - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:34:48.993 INFO ApplyBQSR - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:34:48.993 INFO ApplyBQSR - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:34:48.993 INFO ApplyBQSR - Start Date/Time: January 20, 2022 at 11:34:47 PM CST
    23:34:48.993 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:48.993 INFO ApplyBQSR - ------------------------------------------------------------
    23:34:48.994 INFO ApplyBQSR - HTSJDK Version: 2.24.1
    23:34:48.994 INFO ApplyBQSR - Picard Version: 2.25.4
    23:34:48.994 INFO ApplyBQSR - Built for Spark Version: 2.4.5
    23:34:48.996 INFO ApplyBQSR - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:34:48.996 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:34:48.996 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:34:48.996 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:34:48.996 INFO ApplyBQSR - Deflater: JdkDeflater
    23:34:48.997 INFO ApplyBQSR - Inflater: JdkInflater
    23:34:48.997 INFO ApplyBQSR - GCS max retries/reopens: 20
    23:34:48.997 INFO ApplyBQSR - Requester pays: disabled
    23:34:48.997 INFO ApplyBQSR - Initializing engine
    23:34:49.138 INFO ApplyBQSR - Done initializing engine
    23:34:49.189 INFO ProgressMeter - Starting traversal
    23:34:49.190 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:34:53.276 INFO BaseRecalibrator - 354797 read(s) filtered by: MappingQualityNotZeroReadFilter
    0 read(s) filtered by: MappingQualityAvailableReadFilter
    0 read(s) filtered by: MappedReadFilter
    15338 read(s) filtered by: NotSecondaryAlignmentReadFilter
    208784 read(s) filtered by: NotDuplicateReadFilter
    0 read(s) filtered by: PassesVendorQualityCheckReadFilter
    0 read(s) filtered by: WellformedReadFilter
    578919 total reads filtered
    23:34:53.276 INFO ProgressMeter - AADN05001514.1:518 5.4 9023952 1659969.2
    23:34:53.277 INFO ProgressMeter - Traversal complete. Processed 9023952 total reads in 5.4 minutes.
    23:34:53.409 INFO BaseRecalibrator - Calculating quantized quality scores...
    23:34:53.418 INFO BaseRecalibrator - Writing recalibration report...
    23:34:54.423 INFO BaseRecalibrator - ...done!
    23:34:54.423 INFO BaseRecalibrator - BaseRecalibrator was able to recalibrate 9023952 reads
    23:34:54.423 INFO BaseRecalibrator - Shutting down engine
    [January 20, 2022 at 11:34:54 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 5.53 minutes.
    Runtime.totalMemory()=2579496960
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar BaseRecalibrator -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam --use-jdk-deflater true --use-jdk-inflater true --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf --known-sites /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus_incl_consequences.vcf -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table

    real 5m33.493s
    user 5m40.929s
    sys 0m6.200s
    23:34:58.323 INFO ProgressMeter - 1:71232964 0.3 1128000 3381970.8
    23:34:58.324 INFO ProgressMeter - 1:70415914 0.3 1120000 3357985.2
    23:34:59.196 INFO ProgressMeter - 1:8461467 0.2 522000 3130434.8
    Jan 20, 2022 11:35:06 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:35:06.164 INFO ApplyBQSR - ------------------------------------------------------------
    23:35:06.167 INFO ApplyBQSR - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:35:06.167 INFO ApplyBQSR - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:35:06.168 INFO ApplyBQSR - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:35:06.168 INFO ApplyBQSR - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:35:06.168 INFO ApplyBQSR - Start Date/Time: January 20, 2022 at 11:34:56 PM CST
    23:35:06.168 INFO ApplyBQSR - ------------------------------------------------------------
    23:35:06.168 INFO ApplyBQSR - ------------------------------------------------------------
    23:35:06.168 INFO ApplyBQSR - HTSJDK Version: 2.24.1
    23:35:06.169 INFO ApplyBQSR - Picard Version: 2.25.4
    23:35:06.169 INFO ApplyBQSR - Built for Spark Version: 2.4.5
    23:35:06.170 INFO ApplyBQSR - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:35:06.170 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:35:06.170 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:35:06.170 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:35:06.171 INFO ApplyBQSR - Deflater: JdkDeflater
    23:35:06.171 INFO ApplyBQSR - Inflater: JdkInflater
    23:35:06.171 INFO ApplyBQSR - GCS max retries/reopens: 20
    23:35:06.171 INFO ApplyBQSR - Requester pays: disabled
    23:35:06.172 INFO ApplyBQSR - Initializing engine
    23:35:06.313 INFO ApplyBQSR - Done initializing engine
    23:35:06.434 INFO ProgressMeter - Starting traversal
    23:35:06.434 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
    23:35:08.326 INFO ProgressMeter - 1:130109529 0.5 1688000 3374312.8
    23:35:08.333 INFO ProgressMeter - 1:132867131 0.5 1715000 3427600.7
    23:35:09.210 INFO ProgressMeter - 1:67124828 0.3 1088000 3260902.1
    23:35:16.445 INFO ProgressMeter - 1:8461467 0.2 522000 3128871.1
    23:35:18.331 INFO ProgressMeter - 1:190717108 0.7 2275000 3410794.6
    23:35:18.339 INFO ProgressMeter - 1:196053338 0.7 2324000 3483648.5
    23:35:19.219 INFO ProgressMeter - 1:129678011 0.5 1684000 3364747.4
    23:35:26.446 INFO ProgressMeter - 1:68069498 0.3 1097000 3289190.9
    23:35:28.347 INFO ProgressMeter - 2:55594130 0.8 2865000 3435526.4
    23:35:28.353 INFO ProgressMeter - 2:61981885 0.8 2925000 3507054.1
    23:35:29.228 INFO ProgressMeter - 1:190615544 0.7 2274000 3407762.6
    23:35:36.460 INFO ProgressMeter - 1:127986512 0.5 1667000 3331224.0
    23:35:38.359 INFO ProgressMeter - 2:115766103 1.0 3450000 3447299.6
    23:35:38.362 INFO ProgressMeter - 2:121803594 1.0 3511000 3508018.2
    23:35:39.236 INFO ProgressMeter - 2:56564670 0.8 2874000 3445698.9
    23:35:46.460 INFO ProgressMeter - 1:186551836 0.7 2236000 3351821.3
    23:35:48.365 INFO ProgressMeter - 3:36060099 1.2 4114000 3523617.8
    23:35:48.374 INFO ProgressMeter - 3:28053285 1.2 4038000 3458030.6
    23:35:49.244 INFO ProgressMeter - 2:116828347 1.0 3461000 3457887.9
    23:35:56.469 INFO ProgressMeter - 2:49790175 0.8 2810000 3369708.6
    23:35:58.372 INFO ProgressMeter - 3:98026578 1.3 4721000 3538052.2
    23:35:58.381 INFO ProgressMeter - 3:87747235 1.3 4620000 3461970.8
    23:35:59.250 INFO ProgressMeter - 3:29597340 1.2 4053000 3471024.8
    23:36:06.472 INFO ProgressMeter - 2:108773541 1.0 3381000 3378916.3
    23:36:08.381 INFO ProgressMeter - 4:51350898 1.5 5325000 3547241.0
    23:36:08.399 INFO ProgressMeter - 4:38520622 1.5 5202000 3464612.4
    23:36:09.258 INFO ProgressMeter - 3:89563644 1.3 4638000 3475589.2
    23:36:16.483 INFO ProgressMeter - 3:19536153 1.2 3956000 3388485.2
    23:36:18.388 INFO ProgressMeter - 5:23785122 1.7 5933000 3557061.1
    23:36:18.408 INFO ProgressMeter - 5:4967062 1.7 5758000 3451452.1
    23:36:19.267 INFO ProgressMeter - 4:40531328 1.5 5221000 3477729.9
    23:36:26.486 INFO ProgressMeter - 3:79715801 1.3 4540000 3402788.2
    23:36:28.388 INFO ProgressMeter - 6:28609496 1.8 6541000 3565322.5
    23:36:28.422 INFO ProgressMeter - 6:4387234 1.8 6315000 3441073.1
    23:36:29.269 INFO ProgressMeter - 5:10444874 1.7 5807000 3481449.7
    23:36:36.489 INFO ProgressMeter - 4:28188183 1.5 5102000 3399256.0
    23:36:38.396 INFO ProgressMeter - 8:20121059 2.0 7150000 3572499.3
    23:36:38.424 INFO ProgressMeter - 7:27655230 2.0 6871000 3432267.9
    23:36:39.279 INFO ProgressMeter - 6:12920159 1.8 6395000 3485393.5
    23:36:46.497 INFO ProgressMeter - 4:85339568 1.7 5660000 3393895.8
    23:36:48.403 INFO ProgressMeter - 11:12062187 2.2 7760000 3579005.6
    23:36:48.424 INFO ProgressMeter - 9:20671291 2.2 7427000 3424869.2
    23:36:49.279 INFO ProgressMeter - 8:1369312 2.0 6970000 3482417.2
    23:36:56.501 INFO ProgressMeter - 5:52583065 1.8 6213000 3386846.2
    23:36:58.410 INFO ProgressMeter - 15:4416259 2.3 8371000 3585036.3
    23:36:58.436 INFO ProgressMeter - 12:16527455 2.3 7985000 3419090.1
    23:36:59.293 INFO ProgressMeter - 10:10042371 2.2 7546000 3480012.0
    23:37:06.518 INFO ProgressMeter - 7:13922829 2.0 6740000 3367670.7
    23:37:08.418 INFO ProgressMeter - 23:637554 2.5 8981000 3589839.2
    23:37:08.451 INFO ProgressMeter - 17:9209850 2.5 8542000 3413614.0
    23:37:09.308 INFO ProgressMeter - 13:12328923 2.3 8123000 3478354.0
    23:37:16.528 INFO ProgressMeter - 9:6877872 2.2 7298000 3365899.8
    23:37:18.428 INFO ProgressMeter - unmapped 2.7 9589000 3593247.4
    23:37:18.452 INFO ProgressMeter - 25:3081693 2.7 9099000 3409120.7
    23:37:18.646 INFO ApplyBQSR - 0 read(s) filtered by: WellformedReadFilter

    23:37:18.646 INFO ProgressMeter - unmapped 2.7 9602871 3593552.6
    23:37:18.646 INFO ProgressMeter - Traversal complete. Processed 9602871 total reads in 2.7 minutes.
    23:37:18.697 INFO ApplyBQSR - Shutting down engine
    [January 20, 2022 at 11:37:18 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.ApplyBQSR done. Elapsed time: 2.85 minutes.
    Runtime.totalMemory()=2147483648
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ApplyBQSR --bqsr-recal-file /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa --use-jdk-deflater true --use-jdk-inflater true -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bam

    real 2m52.626s
    user 4m22.785s
    sys 0m10.125s
    23:37:19.323 INFO ProgressMeter - 19:5105653 2.5 8700000 3476917.1
    23:37:26.532 INFO ProgressMeter - 12:2719591 2.3 7857000 3364930.3
    23:37:27.453 INFO ApplyBQSR - 0 read(s) filtered by: WellformedReadFilter

    23:37:27.453 INFO ProgressMeter - unmapped 2.8 9602871 3406441.1
    23:37:27.453 INFO ProgressMeter - Traversal complete. Processed 9602871 total reads in 2.8 minutes.
    23:37:27.503 INFO ApplyBQSR - Shutting down engine
    [January 20, 2022 at 11:37:27 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.ApplyBQSR done. Elapsed time: 2.98 minutes.
    Runtime.totalMemory()=2147483648
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ApplyBQSR --bqsr-recal-file /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa --use-jdk-deflater true --use-jdk-inflater true -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bam

    real 3m0.476s
    user 4m30.790s
    sys 0m10.260s
    23:37:29.335 INFO ProgressMeter - 31:1270014 2.7 9276000 3475350.5
    23:37:35.001 INFO ApplyBQSR - 0 read(s) filtered by: WellformedReadFilter

    23:37:35.001 INFO ProgressMeter - unmapped 2.8 9602871 3474873.6
    23:37:35.001 INFO ProgressMeter - Traversal complete. Processed 9602871 total reads in 2.8 minutes.
    23:37:35.051 INFO ApplyBQSR - Shutting down engine
    [January 20, 2022 at 11:37:35 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.ApplyBQSR done. Elapsed time: 2.79 minutes.
    Runtime.totalMemory()=2747269120
    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ApplyBQSR --bqsr-recal-file /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.recal_data.table -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa --use-jdk-deflater true --use-jdk-inflater true -I /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.bam -O /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bam

    real 2m49.199s
    user 4m29.010s
    sys 0m11.056s

    real 0m16.903s
    user 0m16.650s
    sys 0m0.207s
    23:37:36.548 INFO ProgressMeter - 15:9902504 2.5 8419000 3365042.6
    Jan 20, 2022 11:37:40 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:37:40.036 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:40.039 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:37:40.039 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:37:40.039 INFO HaplotypeCaller - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:37:40.039 INFO HaplotypeCaller - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:37:40.039 INFO HaplotypeCaller - Start Date/Time: January 20, 2022 at 11:37:37 PM CST
    23:37:40.039 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:40.039 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:40.040 INFO HaplotypeCaller - HTSJDK Version: 2.24.1
    23:37:40.040 INFO HaplotypeCaller - Picard Version: 2.25.4
    23:37:40.040 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
    23:37:40.041 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:37:40.041 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:37:40.041 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:37:40.041 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:37:40.042 INFO HaplotypeCaller - Deflater: JdkDeflater
    23:37:40.042 INFO HaplotypeCaller - Inflater: JdkInflater
    23:37:40.043 INFO HaplotypeCaller - GCS max retries/reopens: 20
    23:37:40.043 INFO HaplotypeCaller - Requester pays: disabled
    23:37:40.043 INFO HaplotypeCaller - Initializing engine
    WARNING: BAM index file /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bai is older than BAM /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bam
    23:37:40.200 INFO HaplotypeCaller - Done initializing engine
    23:37:40.201 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
    23:37:40.212 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
    23:37:40.212 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
    23:37:40.222 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
    23:37:40.233 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
    23:37:40.253 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
    23:37:40.253 INFO IntelPairHmm - Available threads: 16
    23:37:40.254 INFO IntelPairHmm - Requested threads: 16
    23:37:40.254 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
    23:37:40.749 INFO ProgressMeter - Starting traversal
    23:37:40.749 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
    23:37:41.109 WARN InbreedingCoeff - InbreedingCoeff will not be calculated at position MT:683 and possibly subsequent; at least 10 samples must have called genotypes
    23:37:43.109 WARN DepthPerSampleHC - Annotation will not be calculated at position W:277088 and possibly subsequent; genotype for sample CD044A07 is not called
    23:37:43.109 WARN StrandBiasBySample - Annotation will not be calculated at position W:277088 and possibly subsequent; genotype for sample CD044A07 is not called

    real 0m16.865s
    user 0m16.638s
    sys 0m0.205s
    23:37:46.562 INFO ProgressMeter - 23:637554 2.7 8981000 3365203.9
    23:37:51.481 INFO ProgressMeter - W:1843847 0.2 6780 37905.3

    real 0m17.013s
    user 0m16.675s
    sys 0m0.297s
    Jan 20, 2022 11:37:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    Jan 20, 2022 11:37:56 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    23:37:56.068 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.069 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.070 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:37:56.070 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:37:56.071 INFO HaplotypeCaller - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:37:56.071 INFO HaplotypeCaller - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:37:56.071 INFO HaplotypeCaller - Start Date/Time: January 20, 2022 at 11:37:45 PM CST
    23:37:56.072 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.072 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.072 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.4.0
    23:37:56.072 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
    23:37:56.072 INFO HaplotypeCaller - Executing as sc94039@ea1216.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    23:37:56.072 INFO HaplotypeCaller - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    23:37:56.072 INFO HaplotypeCaller - Start Date/Time: January 20, 2022 at 11:37:53 PM CST
    23:37:56.072 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.072 INFO HaplotypeCaller - HTSJDK Version: 2.24.1
    23:37:56.072 INFO HaplotypeCaller - ------------------------------------------------------------
    23:37:56.072 INFO HaplotypeCaller - Picard Version: 2.25.4
    23:37:56.072 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
    23:37:56.073 INFO HaplotypeCaller - HTSJDK Version: 2.24.1
    23:37:56.073 INFO HaplotypeCaller - Picard Version: 2.25.4
    23:37:56.073 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    23:37:56.074 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    23:37:56.075 INFO HaplotypeCaller - Deflater: JdkDeflater
    23:37:56.075 INFO HaplotypeCaller - Inflater: JdkInflater
    23:37:56.075 INFO HaplotypeCaller - Deflater: JdkDeflater
    23:37:56.075 INFO HaplotypeCaller - GCS max retries/reopens: 20
    23:37:56.075 INFO HaplotypeCaller - Requester pays: disabled
    23:37:56.075 INFO HaplotypeCaller - Initializing engine
    23:37:56.075 INFO HaplotypeCaller - Inflater: JdkInflater
    23:37:56.076 INFO HaplotypeCaller - GCS max retries/reopens: 20
    23:37:56.076 INFO HaplotypeCaller - Requester pays: disabled
    23:37:56.076 INFO HaplotypeCaller - Initializing engine
    23:37:56.232 INFO HaplotypeCaller - Done initializing engine
    23:37:56.233 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
    23:37:56.236 INFO HaplotypeCaller - Done initializing engine
    23:37:56.237 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
    23:37:56.244 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
    23:37:56.244 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
    23:37:56.248 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
    23:37:56.248 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
    23:37:56.254 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
    23:37:56.258 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
    23:37:56.264 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
    23:37:56.268 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
    23:37:56.285 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
    23:37:56.285 INFO IntelPairHmm - Available threads: 16
    23:37:56.285 INFO IntelPairHmm - Requested threads: 16
    23:37:56.285 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
    23:37:56.289 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
    23:37:56.289 INFO IntelPairHmm - Available threads: 16
    23:37:56.289 INFO IntelPairHmm - Requested threads: 16
    23:37:56.289 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
    23:37:56.318 INFO ProgressMeter - Starting traversal
    23:37:56.318 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
    23:37:56.322 INFO ProgressMeter - Starting traversal
    23:37:56.323 INFO ProgressMeter - Current Locus Elapsed Minutes Regions Processed Regions/Minute
    23:37:56.569 INFO ProgressMeter - AADN05001154.1:1405 2.8 9540000 3364387.1
    23:37:56.633 WARN InbreedingCoeff - InbreedingCoeff will not be calculated at position MT:683 and possibly subsequent; at least 10 samples must have called genotypes
    23:37:56.635 WARN InbreedingCoeff - InbreedingCoeff will not be calculated at position MT:683 and possibly subsequent; at least 10 samples must have called genotypes
    23:37:57.731 INFO ApplyBQSR - 0 read(s) filtered by: WellformedReadFilter

    23:37:57.731 INFO ProgressMeter - unmapped 2.9 9602871 3363586.4
    23:37:57.731 INFO ProgressMeter - Traversal complete. Processed 9602871 total reads in 2.9 minutes.
    23:37:57.782 INFO ApplyBQSR - Shutting down engine
    [January 20, 2022 at 11:37:57 PM CST] org.broadinstitute.hellbender.tools.walkers.bqsr.ApplyBQSR done. Elapsed time: 3.03 minutes.
    Runtime.totalMemory()=2147483648

    ##############################

    There is no error report while running, however, when I check the results with ValidateVariants, it turn out to be as follows:

    ############################

    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ValidateVariants -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -V /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/gatk/CD044A07.HC.g.vcf.gz --validation-type-to-exclude ALLELES --dbsnp /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    09:54:28.036 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
    Jan 21, 2022 9:54:38 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    09:54:38.538 INFO ValidateVariants - ------------------------------------------------------------
    09:54:38.538 INFO ValidateVariants - The Genome Analysis Toolkit (GATK) v4.2.4.0
    09:54:38.538 INFO ValidateVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
    09:54:38.539 INFO ValidateVariants - Executing as sc94039@eb1005.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    09:54:38.540 INFO ValidateVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    09:54:38.540 INFO ValidateVariants - Start Date/Time: January 21, 2022 at 9:54:27 AM CST
    09:54:38.540 INFO ValidateVariants - ------------------------------------------------------------
    09:54:38.540 INFO ValidateVariants - ------------------------------------------------------------
    09:54:38.542 INFO ValidateVariants - HTSJDK Version: 2.24.1
    09:54:38.542 INFO ValidateVariants - Picard Version: 2.25.4
    09:54:38.542 INFO ValidateVariants - Built for Spark Version: 2.4.5
    09:54:38.542 INFO ValidateVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    09:54:38.543 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    09:54:38.543 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    09:54:38.543 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    09:54:38.543 INFO ValidateVariants - Deflater: IntelDeflater
    09:54:38.543 INFO ValidateVariants - Inflater: IntelInflater
    09:54:38.543 INFO ValidateVariants - GCS max retries/reopens: 20
    09:54:38.543 INFO ValidateVariants - Requester pays: disabled
    09:54:38.543 INFO ValidateVariants - Initializing engine
    09:54:38.833 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    09:54:39.011 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/gatk/CD044A07.HC.g.vcf.gz
    09:54:39.167 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    09:54:39.186 INFO ValidateVariants - Done initializing engine
    09:54:39.197 INFO ProgressMeter - Starting traversal
    09:54:39.197 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute
    09:54:49.199 INFO ProgressMeter - 1:56297645 0.2 4160000 24960000.0
    09:54:59.198 INFO ProgressMeter - 2:2657818 0.3 10196000 30588000.0
    09:55:09.202 INFO ProgressMeter - 2:85493244 0.5 13712000 27421257.9
    09:55:19.200 INFO ProgressMeter - 3:30759066 0.7 17741000 26609504.3
    09:55:29.202 INFO ProgressMeter - 4:35015198 0.8 22634000 27158627.3
    09:55:39.210 INFO ProgressMeter - 5:23057708 1.0 26061000 26055354.7
    09:55:49.210 INFO ProgressMeter - 7:12631698 1.2 29692000 25445560.1
    09:55:59.210 INFO ProgressMeter - 11:7295985 1.3 34116000 25582842.8
    09:56:09.224 INFO ProgressMeter - 17:9018952 1.5 37860000 25232430.3
    09:56:17.775 INFO ValidateVariants - Shutting down engine
    [January 21, 2022 at 9:56:17 AM CST] org.broadinstitute.hellbender.tools.walkers.variantutils.ValidateVariants done. Elapsed time: 1.83 minutes.
    Runtime.totalMemory()=2579496960
    java.lang.RuntimeException: Invalid deflate block found.
    at com.intel.gkl.compression.IntelInflater.inflateNative(Native Method)
    at com.intel.gkl.compression.IntelInflater.inflate(IntelInflater.java:174)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:145)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
    at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
    at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
    at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
    at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:257)
    at htsjdk.tribble.readers.PositionalBufferedStream.fill(PositionalBufferedStream.java:132)
    at htsjdk.tribble.readers.PositionalBufferedStream.read(PositionalBufferedStream.java:84)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
    at htsjdk.tribble.readers.LongLineBufferedReader.fill(LongLineBufferedReader.java:140)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:300)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:356)
    at htsjdk.tribble.readers.SynchronousLineReader.readLine(SynchronousLineReader.java:51)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:205)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:149)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at org.broadinstitute.hellbender.engine.VariantWalker.traverse(VariantWalker.java:102)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1085)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
    at org.broadinstitute.hellbender.Main.main(Main.java:289)

    #########################################

    if I add "--use-jdk-deflater true --use-jdk-inflater true" in ValidateVariants, It still output error as below:

    ######################################

    Using GATK jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar
    Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /public1/home/sc94039/software/gatk-4.2.4.0/gatk-package-4.2.4.0-local.jar ValidateVariants -R /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa -V /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/gatk/CD044A07.HC.g.vcf.gz --use-jdk-deflater true --use-jdk-inflater true --validation-type-to-exclude ALLELES --dbsnp /public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    Jan 21, 2022 10:13:16 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
    INFO: Failed to detect whether we are running on Google Compute Engine.
    10:13:16.452 INFO ValidateVariants - ------------------------------------------------------------
    10:13:16.455 INFO ValidateVariants - The Genome Analysis Toolkit (GATK) v4.2.4.0
    10:13:16.455 INFO ValidateVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
    10:13:16.456 INFO ValidateVariants - Executing as sc94039@eb1005.para.bscc on Linux v3.10.0-1160.11.1.el7.x86_64 amd64
    10:13:16.456 INFO ValidateVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v11.0.13+10-LTS-370
    10:13:16.456 INFO ValidateVariants - Start Date/Time: January 21, 2022 at 10:13:06 AM CST
    10:13:16.456 INFO ValidateVariants - ------------------------------------------------------------
    10:13:16.456 INFO ValidateVariants - ------------------------------------------------------------
    10:13:16.457 INFO ValidateVariants - HTSJDK Version: 2.24.1
    10:13:16.457 INFO ValidateVariants - Picard Version: 2.25.4
    10:13:16.457 INFO ValidateVariants - Built for Spark Version: 2.4.5
    10:13:16.459 INFO ValidateVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
    10:13:16.459 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
    10:13:16.459 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
    10:13:16.459 INFO ValidateVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
    10:13:16.460 INFO ValidateVariants - Deflater: JdkDeflater
    10:13:16.460 INFO ValidateVariants - Inflater: JdkInflater
    10:13:16.460 INFO ValidateVariants - GCS max retries/reopens: 20
    10:13:16.461 INFO ValidateVariants - Requester pays: disabled
    10:13:16.461 INFO ValidateVariants - Initializing engine
    10:13:16.624 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf
    10:13:16.661 INFO FeatureManager - Using codec VCFCodec to read file file:///public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/gatk/CD044A07.HC.g.vcf.gz
    10:13:16.784 WARN IndexUtils - Feature file "file:///public1/home/sc94039/wgsAnalysis/reference/gallus_gallus.vcf" appears to contain no sequence dictionary. Attempting to retrieve a sequence dictionary from the associated index file
    10:13:16.799 INFO ValidateVariants - Done initializing engine
    10:13:16.811 INFO ProgressMeter - Starting traversal
    10:13:16.811 INFO ProgressMeter - Current Locus Elapsed Minutes Variants Processed Variants/Minute
    10:13:26.813 INFO ProgressMeter - 1:64402023 0.2 4507000 27042000.0
    10:13:36.812 INFO ProgressMeter - 2:3888840 0.3 10250000 30750000.0
    10:13:46.813 INFO ProgressMeter - 2:85998521 0.5 13735000 27468168.8
    10:13:56.814 INFO ProgressMeter - 3:33875103 0.7 17876000 26812659.4
    10:14:06.813 INFO ProgressMeter - 4:37342403 0.8 22735000 27280908.8
    10:14:16.813 INFO ProgressMeter - 5:26077630 1.0 26191000 26190127.0
    10:14:26.814 INFO ProgressMeter - 7:16622319 1.2 29869000 25600902.8
    10:14:36.814 INFO ProgressMeter - 11:8787039 1.3 34183000 25636288.6
    10:14:46.820 INFO ProgressMeter - 18:1825629 1.5 37994000 25326800.7
    10:14:54.931 INFO ValidateVariants - Shutting down engine
    [January 21, 2022 at 10:14:54 AM CST] org.broadinstitute.hellbender.tools.walkers.variantutils.ValidateVariants done. Elapsed time: 1.81 minutes.
    Runtime.totalMemory()=2579496960
    htsjdk.samtools.util.RuntimeIOException: java.util.zip.DataFormatException: invalid code lengths set
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:161)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:96)
    at htsjdk.samtools.util.BlockCompressedInputStream.inflateBlock(BlockCompressedInputStream.java:550)
    at htsjdk.samtools.util.BlockCompressedInputStream.processNextBlock(BlockCompressedInputStream.java:532)
    at htsjdk.samtools.util.BlockCompressedInputStream.nextBlock(BlockCompressedInputStream.java:468)
    at htsjdk.samtools.util.BlockCompressedInputStream.readBlock(BlockCompressedInputStream.java:458)
    at htsjdk.samtools.util.BlockCompressedInputStream.available(BlockCompressedInputStream.java:196)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:331)
    at htsjdk.samtools.util.BlockCompressedInputStream.read(BlockCompressedInputStream.java:257)
    at htsjdk.tribble.readers.PositionalBufferedStream.fill(PositionalBufferedStream.java:132)
    at htsjdk.tribble.readers.PositionalBufferedStream.read(PositionalBufferedStream.java:84)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181)
    at htsjdk.tribble.readers.LongLineBufferedReader.fill(LongLineBufferedReader.java:140)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:300)
    at htsjdk.tribble.readers.LongLineBufferedReader.readLine(LongLineBufferedReader.java:356)
    at htsjdk.tribble.readers.SynchronousLineReader.readLine(SynchronousLineReader.java:51)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.readNextRecord(TabixFeatureReader.java:170)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:205)
    at htsjdk.tribble.TabixFeatureReader$FeatureIterator.next(TabixFeatureReader.java:149)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
    at org.broadinstitute.hellbender.engine.VariantWalker.traverse(VariantWalker.java:102)
    at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1085)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
    at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
    at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
    at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
    at org.broadinstitute.hellbender.Main.main(Main.java:289)
    Caused by: java.util.zip.DataFormatException: invalid code lengths set
    at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
    at java.base/java.util.zip.Inflater.inflate(Inflater.java:385)
    at htsjdk.samtools.util.BlockGunzipper.unzipBlock(BlockGunzipper.java:145)
    ... 37 more

    ################################

    About check by ValidateSamFile, it outputs with:

    ############################

     

    [Fri Jan 21 10:22:12 CST 2022] ValidateSamFile --INPUT /public1/home/sc94039/wgsAnalysis/outputWgsPipe/CD044A07/bwa/CD044A07.sorted.markdup.BQSR.bam --MODE SUMMARY --REFERENCE_SEQUENCE /public1/home/sc94039/wgsAnalysis/reference/Gallus_gallus.GRCg6a.dna.toplevel.fa --USE_JDK_DEFLATER true --USE_JDK_INFLATER true --MAX_OUTPUT 100 --IGNORE_WARNINGS false --VALIDATE_INDEX true --INDEX_VALIDATION_STRINGENCY EXHAUSTIVE --IS_BISULFITE_SEQUENCED false --MAX_OPEN_TEMP_FILES 8000 --SKIP_MATE_VALIDATION false --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false
    [Fri Jan 21 10:22:12 CST 2022] Executing as sc94039@eb1005.para.bscc on Linux 3.10.0-1160.11.1.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 11.0.13+10-LTS-370; Deflater: Jdk; Inflater: Jdk; Provider GCS is not available; Picard version: Version:2.26.10


    ## HISTOGRAM java.lang.String
    Error Type Count
    ERROR:INVALID_TAG_NM 87

    [Fri Jan 21 10:23:38 CST 2022] picard.sam.ValidateSamFile done. Elapsed time: 1.43 minutes.
    Runtime.totalMemory()=3921674240
    To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp

    ###########################

    I do not know why it will still output wrong.

    I observed some interesting things, which is that it seems you can pass "ValidateVariants" check if you rerun your same program (GATK pipeline (BaseRecalibrator, ApplyBQSR, HaplotypeCaller)) for several times, it is likely to be chance.

    As for your second reply, I misunderstand the meaning of "Deflater: Intel; Inflater: Intel". I was thought that it tells me whether if I have used "--use-jdk-deflater true --use-jdk-inflater true" in gatk pipeline. Thank you !

     

     

    0
    Comment actions Permalink
  • Avatar
    Genevieve Brandt (she/her)

    Hi Yuxiang Yang,

    Are you running all these steps in some sort of pipeline? It looks like there might be some issues with how you have written the steps. Try running all these steps separately to verify if you are still getting the inflate error. Here are the issues I noticed:

    1. There seems to be two BaseRecalibrator runs running simultaneously. If they are writing to the same file, this is going to cause issues.
    2. You did not include the program log for MarkDuplicates (unless it is running in the background the whole time, which it should not be)
    3. There are also two ApplyBQSR runs running simultaneously, or more.

    Run these steps one at a time in the correct order and then see if you are still having the problem.

    Best,

    Genevieve

    0
    Comment actions Permalink

Please sign in to leave a comment.

Welcome

Forum Guidelines

GATK Announcements

Become a GATK Guru

Read Before You Post

Get Email Notifications

Didn't find what you were looking for?

New post

Quick Links

GATK Latest Releases

Known Issues & Troubleshooting

Submitting a Bug Report

How to Track a Github issue

GATK on the Cloud with Terra

Recent GATK News

GATK Blog Posts

GATK Twitter Page

About the GATK community

  • 133,119 Total Users
  • 0 Comments
  • 0 Posts
  • 0 Votes
Powered by Zendesk