Dear all,

I am currently using gatk4.1.9.0 to run BQSR on my sorted and deduped bam files, which I have had no problems with my previous 2 files, but having the error as stated below:

Command inserted:

./gatk ApplyBQSR -R cf3.1toplevel.fa -I Kuu_sorted_dedup.bam -bqsr recal_kuu.table -O recal_reads_kuu.bam

Error log:

$ ./gatk ApplyBQSR -R cf3.1toplevel.fa -I Kuu_sorted_dedup.bam -bqsr recal_kuu.table -O recal_reads_kuu.bam
Using GATK jar /mnt/d/Docker/WGS/gatk4/gatk-package-4.1.9.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /mnt/d/Docker/WGS/gatk4/gatk-package-4.1.9.0-local.jar ApplyBQSR -R cf3.1toplevel.fa -I Kuu_sorted_dedup.bam -bqsr recal_kuu.table -O recal_reads_kuu.bam
12:30:09.851 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/d/Docker/WGS/gatk4/gatk-package-4.1.9.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Jan 01, 2021 12:30:09 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
12:30:09.993 INFO ApplyBQSR - ------------------------------------------------------------
12:30:09.993 INFO ApplyBQSR - The Genome Analysis Toolkit (GATK) v4.1.9.0
12:30:09.993 INFO ApplyBQSR - For support and documentation go to https://software.broadinstitute.org/gatk/
12:30:09.993 INFO ApplyBQSR - Executing as naika@DESKTOP-TMKR2LT on Linux v4.19.128-microsoft-standard amd64
12:30:09.993 INFO ApplyBQSR - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_275-8u275-b01-0ubuntu1~20.04-b01
12:30:09.993 INFO ApplyBQSR - Start Date/Time: January 1, 2021 12:30:09 PM JST
12:30:09.993 INFO ApplyBQSR - ------------------------------------------------------------
12:30:09.993 INFO ApplyBQSR - ------------------------------------------------------------
12:30:09.994 INFO ApplyBQSR - HTSJDK Version: 2.23.0
12:30:09.994 INFO ApplyBQSR - Picard Version: 2.23.3
12:30:09.994 INFO ApplyBQSR - HTSJDK Defaults.COMPRESSION_LEVEL : 2
12:30:09.994 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
12:30:09.994 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
12:30:09.994 INFO ApplyBQSR - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
12:30:09.994 INFO ApplyBQSR - Deflater: IntelDeflater
12:30:09.994 INFO ApplyBQSR - Inflater: IntelInflater
12:30:09.994 INFO ApplyBQSR - GCS max retries/reopens: 20
12:30:09.994 INFO ApplyBQSR - Requester pays: disabled
12:30:09.994 INFO ApplyBQSR - Initializing engine
12:30:10.354 INFO ApplyBQSR - Done initializing engine
12:30:10.407 INFO ProgressMeter - Starting traversal
12:30:10.407 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute
12:30:10.619 INFO ApplyBQSR - Shutting down engine
[January 1, 2021 12:30:10 PM JST] org.broadinstitute.hellbender.tools.walkers.bqsr.ApplyBQSR done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=982515712
java.lang.IllegalStateException: The covariates table is missing ReadGroup H753JDSXY.2 in RecalTable0
at org.broadinstitute.hellbender.utils.Utils.validate(Utils.java:750)
at org.broadinstitute.hellbender.utils.recalibration.covariates.ReadGroupCovariate.keyForReadGroup(ReadGroupCovariate.java:81)
at org.broadinstitute.hellbender.utils.recalibration.covariates.ReadGroupCovariate.recordValues(ReadGroupCovariate.java:53)
at org.broadinstitute.hellbender.utils.recalibration.covariates.StandardCovariateList.recordAllValuesInStorage(StandardCovariateList.java:133)
at org.broadinstitute.hellbender.utils.recalibration.RecalUtils.computeCovariates(RecalUtils.java:546)
at org.broadinstitute.hellbender.utils.recalibration.RecalUtils.computeCovariates(RecalUtils.java:527)
at org.broadinstitute.hellbender.transformers.BQSRReadTransformer.apply(BQSRReadTransformer.java:145)
at org.broadinstitute.hellbender.transformers.BQSRReadTransformer.apply(BQSRReadTransformer.java:27)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at org.broadinstitute.hellbender.engine.ReadWalker.traverse(ReadWalker.java:94)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1049)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)

To be safe, I will attach the log of the recal.table when I first created the table as well as the command:

Command:

./gatk BaseRecalibrator -R cf3.1toplevel.fa -I Kuu_sorted_dedup.bam --known-sites bqsr_kuu_snps.vcf --known-sites bqsr_kuu_indels.vcf -O recal_kuu.table

Completion log:

12:09:57.665 INFO ProgressMeter - AAEX03025150.1:3340 121.9 524369000 4301180.3
12:10:12.594 INFO BaseRecalibrator - 43702519 read(s) filtered by: MappingQualityNotZeroReadFilter
0 read(s) filtered by: MappingQualityAvailableReadFilter
0 read(s) filtered by: MappedReadFilter
0 read(s) filtered by: NotSecondaryAlignmentReadFilter
120929799 read(s) filtered by: NotDuplicateReadFilter
0 read(s) filtered by: PassesVendorQualityCheckReadFilter
0 read(s) filtered by: WellformedReadFilter
164632318 total reads filtered
12:10:12.595 INFO ProgressMeter - AAEX03026059.1:1598 122.2 524603691 4294340.3
12:10:12.595 INFO ProgressMeter - Traversal complete. Processed 524603691 total reads in 122.2 minutes.
12:10:12.614 INFO BaseRecalibrator - Calculating quantized quality scores...
12:10:12.621 INFO BaseRecalibrator - Writing recalibration report...
12:10:13.454 INFO BaseRecalibrator - ...done!
12:10:13.454 INFO BaseRecalibrator - BaseRecalibrator was able to recalibrate 524603691 reads
12:10:13.455 INFO BaseRecalibrator - Shutting down engine
[January 1, 2021 12:10:13 PM JST] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 122.19 minutes.
Runtime.totalMemory()=693108736
Tool returned:
SUCCESS

I have had no problem with my previous ApplyBQSR with the same flow, same parameters and entry command, but somehow I am having issues with this current run. Any ideas what might be wrong, seeing I actually have 524603691 reads recalibrated?

Thanks.