LeftAlignIndels Alignments added out of order... Offending records are at [chr7:55268881] and [chr7:55268881]

Follow

James Covino

• November 30, 2023 23:44

Hello GATK Community,

We've encountered a unique bam that fails LeftAlignIndels when sorted by coordinate, but passes LeftAlignIndels when sorted by query name.  We are unable to determine why this bam is producing an error when sorted by coordinate.

Any help would be greatly appreciated. 

Best,

James

The following is a description of the issue observed.

Re-sorting the bam by coordinate with Picard-tools or samtools produces an identical error:

picard SortSam -I bedtools_intersected.bam -O bedtools_intersected.picard.sorted.bam -SO coordinate

 

gatk LeftAlignIndels -R ~/Source/ruo-analysis/archer-dependencies/hg19.fa -I bedtools_intersected.picard.sorted.bam -O broken.bam

 

java.lang.IllegalArgumentException: Alignments added out of order in SAMFileWriterImpl.addAlignment for file:///Users/jcovino/Desktop/PLAA_2390/broken.bam. Sort order is coordinate. Offending records are at [chr7:55268881] and [chr7:55268881]

 

Sorting the bam by query name resolves the error:

picard SortSam -I bedtools_intersected.bam -O bedtools_intersected.picard.sorted.queryname.bam -SO queryname

gatk LeftAlignIndels -R ~/Source/ruo-analysis/archer-dependencies/hg19.fa -I bedtools_intersected.picard.sorted.queryname.bam -O not_broken.bam

 

16:05:37.567 WARN  ReadUtils - Skipping index file creation for: not_broken.bam. Index file creation requires reads in coordinate sorted order.

16:05:37.570 WARN  IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater

16:05:37.572 INFO  ProgressMeter - Starting traversal

16:05:37.572 INFO  ProgressMeter -        Current Locus  Elapsed Minutes       Reads Processed     Reads/Minute

16:05:37.620 INFO  LeftAlignIndels - 0 read(s) filtered by: WellformedReadFilter

 

16:05:37.621 INFO  ProgressMeter -             unmapped              0.0                   521         651250.0

16:05:37.621 INFO  ProgressMeter - Traversal complete. Processed 521 total reads in 0.0 minutes.

16:05:37.676 INFO  LeftAlignIndels - Shutting down engine

[November 29, 2023 at 4:05:37 PM MST] org.broadinstitute.hellbender.tools.LeftAlignIndels done.

 

Notes:
picard ValidateSamFile produces and ERROR with the file header:

ERROR::MISSING_PLATFORM_VALUE:Read name Ion_V2_BC15_rawlib.basecaller, A platform (PL) attribute was not found for read group.

This ERROR exists on the "unbroken.bam" as well,  that was sorted by queryname and made it through LeftAlignIndels.

We used bedtools intersect (at chr7    55268881        55268881) to create a smaller bam (521 entries) with all entries at the same location.  The resulting bam (bedtools_intersected.bam) produces the aforementioned error with LeftAlignIndels when sorted by coordinate.

REQUIRED for all errors and issues:
a) GATK version used: 4.4.0.0
b) Exact command used: gatk LeftAlignIndels -R ~/Source/ruo-analysis/archer-dependencies/hg19.fa -I bedtools_intersected.bam -O broken.bam
c) Entire program log:

Using GATK jar /usr/local/bin/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar

Running:

    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /usr/local/bin/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar LeftAlignIndels -R /Users/jcovino/Source/ruo-analysis/archer-dependencies/hg19.fa -I bedtools_intersected.bam -O broken.bam

15:04:40.220 INFO  NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/usr/local/bin/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib

15:04:40.242 WARN  NativeLibraryLoader - Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression5008107439776480310.dylib: dlopen(/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression5008107439776480310.dylib, 0x0001): tried: '/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression5008107439776480310.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression5008107439776480310.dylib' (no such file), '/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression5008107439776480310.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))

15:04:40.245 INFO  NativeLibraryLoader - Loading libgkl_compression.dylib from jar:file:/usr/local/bin/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.dylib

15:04:40.247 WARN  NativeLibraryLoader - Unable to load libgkl_compression.dylib from native/libgkl_compression.dylib (/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression11989923337171710117.dylib: dlopen(/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression11989923337171710117.dylib, 0x0001): tried: '/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression11989923337171710117.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression11989923337171710117.dylib' (no such file), '/private/var/folders/k1/_lybfxrx0fb2t_krcc1kx3880000gq/T/libgkl_compression11989923337171710117.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')))

15:04:40.249 INFO  LeftAlignIndels - ------------------------------------------------------------

15:04:40.250 INFO  LeftAlignIndels - The Genome Analysis Toolkit (GATK) v4.4.0.0

15:04:40.251 INFO  LeftAlignIndels - For support and documentation go to https://software.broadinstitute.org/gatk/

15:04:40.251 INFO  LeftAlignIndels - Executing as jcovino@mac-jcovino on Mac OS X v14.0 aarch64

15:04:40.251 INFO  LeftAlignIndels - Java runtime: OpenJDK 64-Bit Server VM v17.0.9+9

15:04:40.251 INFO  LeftAlignIndels - Start Date/Time: November 30, 2023 at 3:04:40 PM MST

15:04:40.251 INFO  LeftAlignIndels - ------------------------------------------------------------

15:04:40.251 INFO  LeftAlignIndels - ------------------------------------------------------------

15:04:40.251 INFO  LeftAlignIndels - HTSJDK Version: 3.0.5

15:04:40.251 INFO  LeftAlignIndels - Picard Version: 3.0.0

15:04:40.251 INFO  LeftAlignIndels - Built for Spark Version: 3.3.1

15:04:40.251 INFO  LeftAlignIndels - HTSJDK Defaults.COMPRESSION_LEVEL : 2

15:04:40.252 INFO  LeftAlignIndels - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

15:04:40.252 INFO  LeftAlignIndels - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

15:04:40.252 INFO  LeftAlignIndels - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

15:04:40.252 INFO  LeftAlignIndels - Deflater: JdkDeflater

15:04:40.252 INFO  LeftAlignIndels - Inflater: JdkInflater

15:04:40.252 INFO  LeftAlignIndels - GCS max retries/reopens: 20

15:04:40.252 INFO  LeftAlignIndels - Requester pays: disabled

15:04:40.252 INFO  LeftAlignIndels - Initializing engine

15:04:40.275 WARN  IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater

15:04:40.276 WARN  IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater

15:04:40.329 INFO  LeftAlignIndels - Done initializing engine

15:04:40.334 WARN  IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater

15:04:40.338 INFO  ProgressMeter - Starting traversal

15:04:40.338 INFO  ProgressMeter -        Current Locus  Elapsed Minutes       Reads Processed     Reads/Minute

15:04:40.380 INFO  LeftAlignIndels - Shutting down engine

[November 30, 2023 at 3:04:40 PM MST] org.broadinstitute.hellbender.tools.LeftAlignIndels done. Elapsed time: 0.00 minutes.

Runtime.totalMemory()=327155712

java.lang.IllegalArgumentException: Alignments added out of order in SAMFileWriterImpl.addAlignment for file:///Users/jcovino/Desktop/PLAA_2390/broken.bam. Sort order is coordinate. Offending records are at [chr7:55268881] and [chr7:55268881]

at htsjdk.samtools.SAMFileWriterImpl.assertPresorted(SAMFileWriterImpl.java:212)

at htsjdk.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:199)

at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:36)

at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:16)

at htsjdk.samtools.util.AbstractAsyncWriter$WriterRunnable.run(AbstractAsyncWriter.java:123)

at java.base/java.lang.Thread.run(Thread.java:840)

 

0

• 

  SkyWarrior

  • February 15, 2024 21:55
  • Edited

  Hi James Covino

  I am not sure if you could solve your issue but let me ask. Have you observed any weird reads within bedtools intersect output via IGV or just samtools view command ?

  Would you be able to share any images of the region? 

  Also I noticed that you are trying to run these tools on a Apple Silicon Mac therefore you are receiving many error messages with response to library incompatibilities. Can you try running these in a compatible x86_64 compute environment and see if the issue still persists?

  Regards 

  0

  Comment actions Permalink
• 

  James Covino

  • March 21, 2024 21:42
  • Edited

  Hi @SkyWarrior,  
   
  Thanks for the reply.  
   
  I ran this on my personal Mac (Silicon) but we saw and duplicated this error on our deployed instance (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz).   
   
  I'm attaching a sam with two alignments that result in the error from LeftAlignIndels. It appears that the first alignment is simplified into 56S74M, but with a starting coordinate of 55268882. When the next alignment is processed and written to the output, the coordinate-sorted order is no longer preserved and the processing crashes with the aforementioned error.  LeftAlignIndels assumes that if the input is coordinate-sorted, the output will be as well. 
   
   
  @HD  VN:1.6  SO:coordinate  
  @SQ  SN:chr7  LN:159138663  
  @RG  ID:Ion_V2_BC15_rawlib.basecaller  SM:Ion_V2_BC15_rawlib.basecaller  
  CCAAATAACATGTCTTCT_molbar_1  2064  chr7  55268881  60  56S2M4D2M3I67M  *  0  0  TGATCATCGAATTCTCCAAAATGGCCCGAGACCCCCAGCGCTACCTTGTCATTTAGGGGATGAAAGAATGCATTTGCCAAGTCCTACAGACTCCAACTTCTACCGTGCCCTGATGGATGAAGAAGACATG  <<>>>==>>9?7///)/);;;<7=8>==>>>1>>B>>>;<<<<7<6==<<<7@<8*9988=7+780399///19997=7===8<=<;;=<<<8<9==9??==9====8A===@=9=?@B??<7=;;:;<=  SA:Z:chr7,55268051,-,56M74S,60,*;  AS:i:73  YX:i:41  RG:Z:Ion_V2_BC15_rawlib.basecaller  NM:i:7  MD:Z:2^GGAT69  
  CCAACAGCTGGCAAATGC_molbar_1  2064  chr7  55268881  60  64S24M  *  0  0  CCGTGAGTTGATCATCGAATTCTCCAAAATGGCCCGAGACCCCCAGCGCTACCTTGTCATTCAGGGGGATGAAAGAATGCATTTGCCA  8=====<7<=>>===<=9<7<<<68)77777<6=>===<0;;;;;====>=9=9==>=>9>=<1<<<<<=<7=<<9>>;?89@@<9=9  SA:Z:chr7,55268043,-,64M24S,60,*;  HI:i:1  AS:i:88  RG:Z:Ion_V2_BC15_rawlib.basecaller  NM:i:0  MD:Z:24  
   
  We are currently using STAR 2.7.9a to generate this alignment, and perhaps a better alignment will be generated with a more recent version of STAR, which we will try next. 

  0

  Comment actions Permalink
• 

  Gökalp Çelik

  • March 22, 2024 03:34
  • Edited

  Hi James Covino

  We were able to replicate the issue and it looks like there is a conflict within the conditional statements within the actual method (Looks like this is a previously uncaught edge case) that checks the read sorting order when reads are fed with coordinate sorted order. Issue does not happen when reads are fed with queryname order therefore the only solution until a fix arrives will be to use queryname sorted bam files to left align indels. Until we have a PR ready we recommend this quick solution for your use case. 

  I hope this helps. 

  1

  Comment actions Permalink
• 

  James Covino

  • August 21, 2024 17:26

  Hi GATK team and Gökalp Çelik ,

  We are interested in the fix to this leftalignindels edge case .
  Do you have a github PR or issue that we can follow?

   

  Best,

  James

  0

  Comment actions Permalink
• 

  Gökalp Çelik

  • September 09, 2024 07:34

  Hi James Covino

  Sorry for the delay on this issue. We started an issue ticket in the github however the current solution to this problem is feeding reads in queryname sorted way. Our team will look into this matter to see if it is an easy fix or requires more work to get it done. We cannot promise an easy solution to this problem immediately and it may definitely be in the backlog for sometime. If you have a solution to fix it you are welcome to suggest a PR with proper tests and we can certainly review it. 

  https://github.com/broadinstitute/gatk/issues/8975 

  Regards. 

  0

  Comment actions Permalink

Please sign in to leave a comment.