Multithreaded Processing on VariantContexts with VCFFileReader
Dear GATK,
I'm using HTSJDK version 2.24.1. And I'm using the VCFFileReader to iterate over the VariantContexts from a VCF file. I have multiple threads processing the VariantContexts. And it appears that the Genotypes are sometimes changed while a thread is using it. Makes me think that HTSJDK is reusing objects. Below is some logging from a single method showing the change in Genotypes. I can prevent this problem by single threading my application. Do you have any suggestions?
3_1 [VC Unknown @ 1:1-3000 Q. of type=SYMBOLIC alleles=[T*, <11>, <21>] attr={END=3000} GT=GT 2/2 1/1 filters=
3_2 [VC Unknown @ 1:1-3000 Q. of type=SYMBOLIC alleles=[T*, <11>, <21>] attr={END=3000} GT=[[LineA_Assembly <21>/<21>],[LineB_Assembly <12>/<12>]] filters=
Thank you,
Terry
-
Hi Terry Casstevens,
Do you have one reader that is being used by multiple threads? This will not work and will lead to issues.
One workaround you could try would be having one reader that records the reads in a blocking cue. The threads can interact with that threadsafe data structure.
Hope this helps!
Genevieve
-
Hi Genevieve,
Thank you for the reply. I only have one thread using the reader. Although I have multiple threads using the VariantContext objects from the reader.
When you say "records the reads", what exactly do you mean? Getting the string information out of the VariantContext to put on the cue? If you mean putting the VariantContext on a cue, that's basically what I'm doing. I'm using a Channel in Kotlin as the cue.
Thank you again,
Terry
-
Hi Terry,
A few questions. What are you seeing change exactly? Is it the result of calling toString() on the genotype context? There's a weird quirk of genotype contexts where they lazily parse some of the data in themselves only when needed. The toString method won't force parsing of the genotypes so it's output could change depending on the other operations that have occurred on the VariantContext object. I think it's possible that you're seeing an issue related to lazy parsing. In general it's best not to rely on toString() for anything other than human readable log messages for exactly this sort of reason. To test if that's the issue you could use `VariantContext.toStringDecodeGenotypes()` instead of `toString()` in your log messages, this will force a decode and should be consistent over time.
If it's not just a confusion with the log messages and the underlying data you get back from calls like VariantContext.getGenotypes() is actually changing, than there must be some multithreading issue. It sounds like you have a single reader putting VariantContext objects into a kotlin channel, and then multiple threads pulling objects out of the channel? I would have expected that to work. Is it possible you could share your code? That would help clarify if the issue is on your end or a bug in the library.
Thank you,
Louis
-
Hi Louis,
I do see the lazily parsing that you mention. But it is more than that. In my original post, the second logging line shows a genotype of <12> which doesn't exist in the list of alleles. That <12> is from another position.
Yes, my reader is putting VariantContext objects into a Kotlin channel and multiple threads are pulling the objects out of the channel. The code is at this link, although I have commented out the multi-threading, so that it will work correctly.
Thank you,
Terry
Please sign in to leave a comment.
4 comments