question about Picard AddOrReplaceReadGroups
Answered
Hello, I have a 30X simulated bam file from running another pipeline and I was trying to run depth-of-coverage tool to get DOC but kept running into problems:
1. When I first ran depthofcoverage, it gave me an error that says my file has no readgroups, so I tried to use Picard AddOrReplaceReadGroups
2. But then after running the tool, depth-of-coverage gave me an error that said my new bam file has incorrect order of chromosomes, the exact error was
Lexicographically sorted human genome sequence detected in reads.
3. So I tried to run samtools sort on this bam, but it tells me the "EOF marker is absent and that my bam file was probably truncated". I went back to check my bam file and it looks like the size of my bam went from 100GB to 3GB after running Picard AddOrReplaceReadGroups.
So I am really confused what is happening here. I should note that when I ran Picard AddOrReplaceReadGroups, it gave me an error message that says SAM validation error: "WARNING: Read name FC:0:0:368:54711, No M or N operator between pair of D operators in CIGAR" . After I set the validation stringency to silent, it ran through.
Thank you so much!
-
Hi Davy Deng,
Could you include your AddOrReplaceReadGroups command line and entire program log? I would recommend keeping the validation stringency as STRICT so that you can fix the errors that come up.
Best,
Genevieve
-
Hello Genevieve Brandt (she/her),
Thank you so much for your response. I ran this on terras so here is the command part of the script and the docker information:
command <<< java -jar /usr/gitc/picard.jar AddOrReplaceReadGroups \ I=${inputBam} \ O=${sampleName}.readgroupadded.bam \ RGID=1 \ RGLB=lib1 \ RGPL=illumina \ RGPU=unit1 \ RGSM=${sampleName} VALIDATION_STRINGENCY=SILENT >>>runtime { docker: "broadinstitute/genomes-in-the-cloud:2.3.1-1500064817" memory: "${memoryGb} GB" cpu: "2" disks: "local-disk ${diskSpaceGb} HDD" }And here is the log file, i set the stringency to silent but the log file still recorded this error with CIGAR string:
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell_root/tmp.c04e388b [Sun Sep 26 21:53:53 UTC 2021] picard.sam.AddOrReplaceReadGroups INPUT=/cromwell_root/fc-b323d821-07e9-4d64-bde5-81d71367bcda/30X_70Purity_sorted.bam OUTPUT=1_30X_70purity_tumor.readgroupadded.bam RGID=1 RGLB=lib1 RGPL=illumina RGPU=unit1 RGSM=1_30X_70purity_tumor VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json [Sun Sep 26 21:53:53 UTC 2021] Executing as root@7202468349a3 on Linux 5.4.120+ amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2~bpo8+1-b14; Picard version: 2.8.3-SNAPSHOT INFO 2021-09-26 21:53:53 AddOrReplaceReadGroups Created read group ID=1 PL=illumina LB=lib1 SM=1_30X_70purity_tumor INFO 2021-09-26 21:54:06 AddOrReplaceReadGroups Processed 1,000,000 records. Elapsed time: 00:00:12s. Time for last 1,000,000: 12s. Last read position: chr1:3,375,986 INFO 2021-09-26 21:54:17 AddOrReplaceReadGroups Processed 2,000,000 records. Elapsed time: 00:00:24s. Time for last 1,000,000: 11s. Last read position: chr1:6,741,944 INFO 2021-09-26 21:54:30 AddOrReplaceReadGroups Processed 3,000,000 records. Elapsed time: 00:00:37s. Time for last 1,000,000: 12s. Last read position: chr1:10,101,832 INFO 2021-09-26 21:54:43 AddOrReplaceReadGroups Processed 4,000,000 records. Elapsed time: 00:00:49s. Time for last 1,000,000: 12s. Last read position: chr1:13,469,279 INFO 2021-09-26 21:54:55 AddOrReplaceReadGroups Processed 5,000,000 records. Elapsed time: 00:01:01s. Time for last 1,000,000: 11s. Last read position: chr1:16,830,201 INFO 2021-09-26 21:55:08 AddOrReplaceReadGroups Processed 6,000,000 records. Elapsed time: 00:01:15s. Time for last 1,000,000: 13s. Last read position: chr1:20,198,305 INFO 2021-09-26 21:55:20 AddOrReplaceReadGroups Processed 7,000,000 records. Elapsed time: 00:01:27s. Time for last 1,000,000: 12s. Last read position: chr1:23,558,761 INFO 2021-09-26 21:55:32 AddOrReplaceReadGroups Processed 8,000,000 records. Elapsed time: 00:01:39s. Time for last 1,000,000: 12s. Last read position: chr1:26,909,108 INFO 2021-09-26 21:55:45 AddOrReplaceReadGroups Processed 9,000,000 records. Elapsed time: 00:01:52s. Time for last 1,000,000: 13s. Last read position: chr1:30,274,430 INFO 2021-09-26 21:55:58 AddOrReplaceReadGroups Processed 10,000,000 records. Elapsed time: 00:02:04s. Time for last 1,000,000: 12s. Last read position: chr1:33,637,168 INFO 2021-09-26 21:56:10 AddOrReplaceReadGroups Processed 11,000,000 records. Elapsed time: 00:02:17s. Time for last 1,000,000: 12s. Last read position: chr1:37,007,616 INFO 2021-09-26 21:56:22 AddOrReplaceReadGroups Processed 12,000,000 records. Elapsed time: 00:02:29s. Time for last 1,000,000: 12s. Last read position: chr1:40,377,876 INFO 2021-09-26 21:56:34 AddOrReplaceReadGroups Processed 13,000,000 records. Elapsed time: 00:02:41s. Time for last 1,000,000: 12s. Last read position: chr1:43,747,264 INFO 2021-09-26 21:56:48 AddOrReplaceReadGroups Processed 14,000,000 records. Elapsed time: 00:02:55s. Time for last 1,000,000: 13s. Last read position: chr1:47,111,040 INFO 2021-09-26 21:57:00 AddOrReplaceReadGroups Processed 15,000,000 records. Elapsed time: 00:03:07s. Time for last 1,000,000: 11s. Last read position: chr1:50,484,571 INFO 2021-09-26 21:57:13 AddOrReplaceReadGroups Processed 16,000,000 records. Elapsed time: 00:03:19s. Time for last 1,000,000: 12s. Last read position: chr1:53,849,699 INFO 2021-09-26 21:57:25 AddOrReplaceReadGroups Processed 17,000,000 records. Elapsed time: 00:03:31s. Time for last 1,000,000: 12s. Last read position: chr1:57,211,176 INFO 2021-09-26 21:57:37 AddOrReplaceReadGroups Processed 18,000,000 records. Elapsed time: 00:03:43s. Time for last 1,000,000: 12s. Last read position: chr1:60,576,562 INFO 2021-09-26 21:57:51 AddOrReplaceReadGroups Processed 19,000,000 records. Elapsed time: 00:03:57s. Time for last 1,000,000: 13s. Last read position: chr1:63,939,497 INFO 2021-09-26 21:58:03 AddOrReplaceReadGroups Processed 20,000,000 records. Elapsed time: 00:04:09s. Time for last 1,000,000: 11s. Last read position: chr1:67,303,887 INFO 2021-09-26 21:58:16 AddOrReplaceReadGroups Processed 21,000,000 records. Elapsed time: 00:04:22s. Time for last 1,000,000: 13s. Last read position: chr1:70,670,289 INFO 2021-09-26 21:58:28 AddOrReplaceReadGroups Processed 22,000,000 records. Elapsed time: 00:04:34s. Time for last 1,000,000: 11s. Last read position: chr1:74,039,518 INFO 2021-09-26 21:58:40 AddOrReplaceReadGroups Processed 23,000,000 records. Elapsed time: 00:04:46s. Time for last 1,000,000: 12s. Last read position: chr1:77,408,879 INFO 2021-09-26 21:58:54 AddOrReplaceReadGroups Processed 24,000,000 records. Elapsed time: 00:05:00s. Time for last 1,000,000: 13s. Last read position: chr1:80,772,595 INFO 2021-09-26 21:59:06 AddOrReplaceReadGroups Processed 25,000,000 records. Elapsed time: 00:05:12s. Time for last 1,000,000: 11s. Last read position: chr1:84,143,101 INFO 2021-09-26 21:59:18 AddOrReplaceReadGroups Processed 26,000,000 records. Elapsed time: 00:05:25s. Time for last 1,000,000: 12s. Last read position: chr1:87,513,324 INFO 2021-09-26 21:59:31 AddOrReplaceReadGroups Processed 27,000,000 records. Elapsed time: 00:05:37s. Time for last 1,000,000: 12s. Last read position: chr1:90,882,531 INFO 2021-09-26 21:59:43 AddOrReplaceReadGroups Processed 28,000,000 records. Elapsed time: 00:05:49s. Time for last 1,000,000: 12s. Last read position: chr1:94,241,211 INFO 2021-09-26 21:59:57 AddOrReplaceReadGroups Processed 29,000,000 records. Elapsed time: 00:06:03s. Time for last 1,000,000: 13s. Last read position: chr1:97,611,281 INFO 2021-09-26 22:00:09 AddOrReplaceReadGroups Processed 30,000,000 records. Elapsed time: 00:06:15s. Time for last 1,000,000: 12s. Last read position: chr1:100,979,853 INFO 2021-09-26 22:00:21 AddOrReplaceReadGroups Processed 31,000,000 records. Elapsed time: 00:06:28s. Time for last 1,000,000: 12s. Last read position: chr1:104,339,804 INFO 2021-09-26 22:00:34 AddOrReplaceReadGroups Processed 32,000,000 records. Elapsed time: 00:06:40s. Time for last 1,000,000: 12s. Last read position: chr1:107,711,743 INFO 2021-09-26 22:00:47 AddOrReplaceReadGroups Processed 33,000,000 records. Elapsed time: 00:06:53s. Time for last 1,000,000: 12s. Last read position: chr1:111,072,736 INFO 2021-09-26 22:01:00 AddOrReplaceReadGroups Processed 34,000,000 records. Elapsed time: 00:07:07s. Time for last 1,000,000: 13s. Last read position: chr1:114,441,526 INFO 2021-09-26 22:01:12 AddOrReplaceReadGroups Processed 35,000,000 records. Elapsed time: 00:07:19s. Time for last 1,000,000: 12s. Last read position: chr1:117,807,947 INFO 2021-09-26 22:01:24 AddOrReplaceReadGroups Processed 36,000,000 records. Elapsed time: 00:07:31s. Time for last 1,000,000: 12s. Last read position: chr1:121,175,572 INFO 2021-09-26 22:01:30 AddOrReplaceReadGroups Processed 37,000,000 records. Elapsed time: 00:07:36s. Time for last 1,000,000: 5s. Last read position: chr1:124,542,412 INFO 2021-09-26 22:01:34 AddOrReplaceReadGroups Processed 38,000,000 records. Elapsed time: 00:07:41s. Time for last 1,000,000: 4s. Last read position: chr1:127,904,941 INFO 2021-09-26 22:01:39 AddOrReplaceReadGroups Processed 39,000,000 records. Elapsed time: 00:07:45s. Time for last 1,000,000: 4s. Last read position: chr1:131,272,359 INFO 2021-09-26 22:01:43 AddOrReplaceReadGroups Processed 40,000,000 records. Elapsed time: 00:07:50s. Time for last 1,000,000: 4s. Last read position: chr1:134,643,267 INFO 2021-09-26 22:01:49 AddOrReplaceReadGroups Processed 41,000,000 records. Elapsed time: 00:07:56s. Time for last 1,000,000: 5s. Last read position: chr1:135,791,721 INFO 2021-09-26 22:01:54 AddOrReplaceReadGroups Processed 42,000,000 records. Elapsed time: 00:08:01s. Time for last 1,000,000: 5s. Last read position: chr1:136,676,946 INFO 2021-09-26 22:01:59 AddOrReplaceReadGroups Processed 43,000,000 records. Elapsed time: 00:08:05s. Time for last 1,000,000: 4s. Last read position: chr1:137,563,738 INFO 2021-09-26 22:02:03 AddOrReplaceReadGroups Processed 44,000,000 records. Elapsed time: 00:08:10s. Time for last 1,000,000: 4s. Last read position: chr1:138,447,751 INFO 2021-09-26 22:02:08 AddOrReplaceReadGroups Processed 45,000,000 records. Elapsed time: 00:08:14s. Time for last 1,000,000: 4s. Last read position: chr1:139,332,695 INFO 2021-09-26 22:02:12 AddOrReplaceReadGroups Processed 46,000,000 records. Elapsed time: 00:08:19s. Time for last 1,000,000: 4s. Last read position: chr1:140,217,781 INFO 2021-09-26 22:02:17 AddOrReplaceReadGroups Processed 47,000,000 records. Elapsed time: 00:08:23s. Time for last 1,000,000: 4s. Last read position: chr1:141,103,509 INFO 2021-09-26 22:02:21 AddOrReplaceReadGroups Processed 48,000,000 records. Elapsed time: 00:08:28s. Time for last 1,000,000: 4s. Last read position: chr1:141,989,409 INFO 2021-09-26 22:02:29 AddOrReplaceReadGroups Processed 49,000,000 records. Elapsed time: 00:08:35s. Time for last 1,000,000: 7s. Last read position: chr1:142,874,702 INFO 2021-09-26 22:02:38 AddOrReplaceReadGroups Processed 50,000,000 records. Elapsed time: 00:08:45s. Time for last 1,000,000: 9s. Last read position: chr1:143,759,544 INFO 2021-09-26 22:02:48 AddOrReplaceReadGroups Processed 51,000,000 records. Elapsed time: 00:08:54s. Time for last 1,000,000: 9s. Last read position: chr1:144,645,536 INFO 2021-09-26 22:03:00 AddOrReplaceReadGroups Processed 52,000,000 records. Elapsed time: 00:09:06s. Time for last 1,000,000: 12s. Last read position: chr1:145,531,534 INFO 2021-09-26 22:03:11 AddOrReplaceReadGroups Processed 53,000,000 records. Elapsed time: 00:09:17s. Time for last 1,000,000: 10s. Last read position: chr1:146,417,530 INFO 2021-09-26 22:03:22 AddOrReplaceReadGroups Processed 54,000,000 records. Elapsed time: 00:09:29s. Time for last 1,000,000: 11s. Last read position: chr1:147,302,684 INFO 2021-09-26 22:03:33 AddOrReplaceReadGroups Processed 55,000,000 records. Elapsed time: 00:09:40s. Time for last 1,000,000: 10s. Last read position: chr1:148,190,248 INFO 2021-09-26 22:03:43 AddOrReplaceReadGroups Processed 56,000,000 records. Elapsed time: 00:09:50s. Time for last 1,000,000: 10s. Last read position: chr1:149,078,571 INFO 2021-09-26 22:03:56 AddOrReplaceReadGroups Processed 57,000,000 records. Elapsed time: 00:10:02s. Time for last 1,000,000: 12s. Last read position: chr1:149,965,098 INFO 2021-09-26 22:04:08 AddOrReplaceReadGroups Processed 58,000,000 records. Elapsed time: 00:10:14s. Time for last 1,000,000: 12s. Last read position: chr1:150,851,055 INFO 2021-09-26 22:04:20 AddOrReplaceReadGroups Processed 59,000,000 records. Elapsed time: 00:10:26s. Time for last 1,000,000: 12s. Last read position: chr1:151,735,301 INFO 2021-09-26 22:04:32 AddOrReplaceReadGroups Processed 60,000,000 records. Elapsed time: 00:10:39s. Time for last 1,000,000: 12s. Last read position: chr1:152,622,819 INFO 2021-09-26 22:04:44 AddOrReplaceReadGroups Processed 61,000,000 records. Elapsed time: 00:10:51s. Time for last 1,000,000: 12s. Last read position: chr1:153,509,263 INFO 2021-09-26 22:04:57 AddOrReplaceReadGroups Processed 62,000,000 records. Elapsed time: 00:11:04s. Time for last 1,000,000: 12s. Last read position: chr1:154,395,518 [Sun Sep 26 22:04:59 UTC 2021] picard.sam.AddOrReplaceReadGroups done. Elapsed time: 11.10 minutes. Runtime.totalMemory()=113246208 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread "main" htsjdk.samtools.SAMFormatException: SAM validation error: WARNING: Read name FC:0:0:368:54711, No M or N operator between pair of D operators in CIGAR at htsjdk.samtools.SAMUtils.processValidationErrors(SAMUtils.java:448) at htsjdk.samtools.BAMRecord.getCigar(BAMRecord.java:252) at htsjdk.samtools.SAMRecord.getAlignmentEnd(SAMRecord.java:600) at htsjdk.samtools.SAMRecord.computeIndexingBin(SAMRecord.java:1531) at htsjdk.samtools.SAMRecord.isValid(SAMRecord.java:2038) at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:664) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:650) at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:620) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:569) at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:543) at picard.sam.AddOrReplaceReadGroups.doWork(AddOrReplaceReadGroups.java:140) at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:205) at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:94) at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:104)
Thank you very much!
-
Thanks for sharing that! The AddOrReplaceReadGroups command was not successful, you have an error message you need to address:
Exception in thread "main" htsjdk.samtools.SAMFormatException: SAM validation error: WARNING: Read name FC:0:0:368:54711, No M or N operator between pair of D operators in CIGAR at htsjdk.samtools.SAMUtils.processValidationErrors(SAMUtils.java:448)
There is an error in the read named FC:0:0:368:54711, an issue with the CIGAR string. You'll want to resolve that issue before continuing. We have an article that covers the details of CIGAR strings here: https://gatk.broadinstitute.org/hc/en-us/articles/360035890791-SAM-or-BAM-or-CRAM-Mapped-sequence-data-formats
This indicates a problem when you were creating this BAM file.
Hope this helps!
Genevieve
Please sign in to leave a comment.
3 comments