Lifts over a VCF file from one reference build to another.
Summary
Tool for "lifting over" a VCF from one genome build to another, producing a properly headered, sorted and indexed VCF in one go.Details
This tool adjusts the coordinates of variants within a VCF file to match a new reference. The output file will be sorted and indexed using the target reference build. To be clear, REFERENCE_SEQUENCE should be the target reference build (that is, the "new" one). The tool is based on the UCSC LiftOver tool (see http://genome.ucsc.edu/cgi-bin/hgLiftOver) and uses a UCSC chain file to guide its operation. For each variant, the tool will look for the target coordinate, reverse-complement and left-align the variant if needed, and, in the case that the reference and alternate alleles of a SNP have been swapped in the new genome build, it will adjust the SNP, and correct AF-like INFO fields and the relevant genotypes.Example
java -jar picard.jar LiftoverVcf \ I=input.vcf \ O=lifted_over.vcf \ CHAIN=b37tohg38.chain \ REJECT=rejected_variants.vcf \ R=reference_sequence.fastaCaveats
Rejected Records
Records may be rejected because they cannot be lifted over or because of sequence incompatibilities between the source and target reference genomes. Rejected records will be emitted to the REJECT file using the source genome build coordinates. The reason for the rejection will be stated in the FILTER field, and more detail may be placed in the INFO field.Memory Use
LiftOverVcf sorts the output using a "SortingCollection" which relies on MAX_RECORDS_IN_RAM to specify how many (vcf) records to hold in memory before "spilling" to disk. The default value is reasonable when sorting SAM files, but not for VCFs as there is no good default due to the dependence on the number of samples and amount of information in the INFO and FORMAT fields. Consider lowering to 100,000 or even less if you have many genotypes.Category Variant Manipulation
Overview
Summary
Tool for "lifting over" a VCF from one genome build to another, producing a properly headered, sorted and indexed VCF in one go.Details
This tool adjusts the coordinates of variants within a VCF file to match a new reference. The output file will be sorted and indexed using the target reference build. To be clear, #REFERENCE_SEQUENCE should be the target reference build (that is, the "new" one). The tool is based on the UCSC LiftOver tool and uses a UCSC chain file to guide its operation.For each variant, the tool will look for the target coordinate, reverse-complement and left-align the variant if needed, and, in the case that the reference and alternate alleles of a SNP have been swapped in the new genome build, it will adjust the SNP, and correct AF-like INFO fields and the relevant genotypes.
Example
java -jar picard.jar LiftoverVcf \\
I=input.vcf \\
O=lifted_over.vcf \\
CHAIN=b37tohg38.chain \\
REJECT=rejected_variants.vcf \\
R=reference_sequence.fasta
Caveats
Rejected Records
Records may be rejected because they cannot be lifted over or because of sequence incompatibilities between the source and target reference genomes. Rejected records will be emitted to the #REJECT file using the source genome build coordinates. The reason for the rejection will be stated in the FILTER field, and more detail may be placed in the INFO field.Memory Use
LiftOverVcf sorts the output using a htsjdk.samtools.util.SortingCollection which relies on #MAX_RECORDS_IN_RAM to specify how many (vcf) records to hold in memory before "spilling" to disk. The default value is reasonable when sorting SAM files, but not for VCFs as there is no good default due to the dependence on the number of samples and amount of information in the INFO and FORMAT fields. Consider lowering to 100,000 or even less if you have many genotypes.LiftoverVcf (Picard) specific arguments
This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.
| Argument name(s) | Default value | Summary | |
|---|---|---|---|
| Required Arguments | |||
|
--CHAIN -C |
null | The liftover chain file. See https://genome.ucsc.edu/goldenPath/help/chain.html for a description of chain files. See http://hgdownload.soe.ucsc.edu/downloads.html#terms for where to download chain files. | |
|
--INPUT -I |
null | The input VCF/BCF file to be lifted over. | |
|
--OUTPUT -O |
null | The output location for the lifted over VCF/BCF. | |
|
--REFERENCE_SEQUENCE -R |
null | The reference sequence (fasta) for the TARGET genome build (i.e., the new one. The fasta file must have an accompanying sequence dictionary (.dict file). | |
|
--REJECT |
null | File to which to write rejected records. | |
| Optional Tool Arguments | |||
|
--ALLOW_MISSING_FIELDS_IN_HEADER |
false | Allow INFO and FORMAT in the records that are not found in the header | |
|
--arguments_file |
[] | read one or more arguments files and add them to the command line | |
|
--help -h |
false | display the help message | |
|
--LIFTOVER_MIN_MATCH |
1.0 | The minimum percent match required for a variant to be lifted. | |
|
--TAGS_TO_DROP |
[MAX_AF] | INFO field annotations that should be deleted when swapping reference with variant alleles. | |
|
--TAGS_TO_REVERSE |
[AF] | INFO field annotations that behave like an Allele Frequency and should be transformed with x->1-x when swapping reference with variant alleles. | |
|
--version |
false | display the version number for this tool | |
|
--WARN_ON_MISSING_CONTIG -WMC |
false | Warn on missing contig. | |
|
--WRITE_ORIGINAL_POSITION |
false | Write the original contig/position for lifted variants to the INFO field. | |
| Optional Common Arguments | |||
|
--COMPRESSION_LEVEL |
5 | Compression level for all compressed files created (e.g. BAM and VCF). | |
|
--CREATE_INDEX |
false | Whether to create a BAM index when writing a coordinate-sorted BAM file. | |
|
--CREATE_MD5_FILE |
false | Whether to create an MD5 digest for any BAM or FASTQ files created. | |
|
--GA4GH_CLIENT_SECRETS |
client_secrets.json | Google Genomics API client_secrets.json file path. | |
|
--MAX_RECORDS_IN_RAM |
500000 | When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed. | |
|
--QUIET |
false | Whether to suppress job-summary info on System.err. | |
|
--TMP_DIR |
[] | One or more directories with space available to be used by this program for temporary storage of working files | |
|
--USE_JDK_DEFLATER -use_jdk_deflater |
false | Use the JDK Deflater instead of the Intel Deflater for writing compressed output | |
|
--USE_JDK_INFLATER -use_jdk_inflater |
false | Use the JDK Inflater instead of the Intel Inflater for reading compressed input | |
|
--VALIDATION_STRINGENCY |
STRICT | Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded. | |
|
--VERBOSITY |
INFO | Control verbosity of logging. | |
| Advanced Arguments | |||
|
--showHidden |
false | display hidden arguments | |
Argument details
Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.
--ALLOW_MISSING_FIELDS_IN_HEADER / NA
Allow INFO and FORMAT in the records that are not found in the header
boolean false
--arguments_file / NA
read one or more arguments files and add them to the command line
List[File] []
--CHAIN / -C
The liftover chain file. See https://genome.ucsc.edu/goldenPath/help/chain.html for a description of chain files. See http://hgdownload.soe.ucsc.edu/downloads.html#terms for where to download chain files.
R File null
--COMPRESSION_LEVEL / NA
Compression level for all compressed files created (e.g. BAM and VCF).
int 5 [ [ -∞ ∞ ] ]
--CREATE_INDEX / NA
Whether to create a BAM index when writing a coordinate-sorted BAM file.
Boolean false
--CREATE_MD5_FILE / NA
Whether to create an MD5 digest for any BAM or FASTQ files created.
boolean false
--GA4GH_CLIENT_SECRETS / NA
Google Genomics API client_secrets.json file path.
String client_secrets.json
--help / -h
display the help message
boolean false
--INPUT / -I
The input VCF/BCF file to be lifted over.
R File null
--LIFTOVER_MIN_MATCH / NA
The minimum percent match required for a variant to be lifted.
double 1.0 [ [ -∞ ∞ ] ]
--MAX_RECORDS_IN_RAM / NA
When writing files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort the file, and increases the amount of RAM needed.
Integer 500000 [ [ -∞ ∞ ] ]
--OUTPUT / -O
The output location for the lifted over VCF/BCF.
R File null
--QUIET / NA
Whether to suppress job-summary info on System.err.
Boolean false
--REFERENCE_SEQUENCE / -R
The reference sequence (fasta) for the TARGET genome build (i.e., the new one. The fasta file must have an accompanying sequence dictionary (.dict file).
R File null
--REJECT / NA
File to which to write rejected records.
R File null
--showHidden / -showHidden
display hidden arguments
boolean false
--TAGS_TO_DROP / NA
INFO field annotations that should be deleted when swapping reference with variant alleles.
Collection[String] [MAX_AF]
--TAGS_TO_REVERSE / NA
INFO field annotations that behave like an Allele Frequency and should be transformed with x->1-x when swapping reference with variant alleles.
Collection[String] [AF]
--TMP_DIR / NA
One or more directories with space available to be used by this program for temporary storage of working files
List[File] []
--USE_JDK_DEFLATER / -use_jdk_deflater
Use the JDK Deflater instead of the Intel Deflater for writing compressed output
Boolean false
--USE_JDK_INFLATER / -use_jdk_inflater
Use the JDK Inflater instead of the Intel Inflater for reading compressed input
Boolean false
--VALIDATION_STRINGENCY / NA
Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
The --VALIDATION_STRINGENCY argument is an enumerated type (ValidationStringency), which can have one of the following values:
- STRICT
- LENIENT
- SILENT
ValidationStringency STRICT
--VERBOSITY / NA
Control verbosity of logging.
The --VERBOSITY argument is an enumerated type (LogLevel), which can have one of the following values:
- ERROR
- WARNING
- INFO
- DEBUG
LogLevel INFO
--version / NA
display the version number for this tool
boolean false
--WARN_ON_MISSING_CONTIG / -WMC
Warn on missing contig.
boolean false
--WRITE_ORIGINAL_POSITION / NA
Write the original contig/position for lifted variants to the INFO field.
boolean false
GATK version 4.0.2.0 built at 02-13-2019 02:13:49.
6 comments
Hello - I am liftover from a VCF in UCSC hg19 coordinates (with "chr" prefixes) to b37 coordinates. My understanding is that in order to successfully use this tool, I would need a chain file that looks something like this hg19tob37.chain.
Is this correct? Does such a file exist, and if so, where can I find it? It is not at the UCSC links listed above.
Thanks,
Riaz
Some of the legacy forums suggest it should be here: ftp://ftp.broadinstitute.org/Liftover_Chain_Files
However, this does not seem accessible.
Thanks,
Riaz
Riaz,
You should be able to find the chain files to liftOver from hg19 to other versions here: https://hgdownload.soe.ucsc.edu/goldenPath/hg19/liftOver/
But I guess that one is not there.
https://gatk.broadinstitute.org/hc/en-us/articles/360035890711-GRCh37-hg19-b37-humanG1Kv37-Human-Reference-Discrepancies
Jose
The provided VCF file is malformed at approximately line number 3460: 0/0:.:.:.:.:.:. is not a valid start position in the VCF format:
I have been using LiftOverVcf with the below command:
bcftools view variants.bcf | picard LiftoverVcf I=/dev/stdin O=lifted_over.vcf CHAIN=canFam3ToCanFam4.over.chain.gz REJECT=rejected_variants.vcf R=target.fasta MAX_RECORDS_IN_RAM=100000
It gives the error:
The input bcf file is a merged multi-sample file and is piped by bcftools. The file has missing genotypes for some samples as `0/0:.:.:.:.:.:` and as `./.:0,0:0:.:0,0,0` for few other samples in the first record itself and picard doesn't seem to like this kind of genotypes.
Could someone help how to run liftover in such cases?
I'm currently working with a VCF for a group a 292 plant samples. The current SNPs were created by mapping to a reference assembly with 1542 contigs, but I want to lift those over to the new Hi-C scaffolded chromosome-level assembly. The biggest problem that I'm having is the requirement of a .chain file. Can this file be generated in some way? I'm having a hard time finding a tool to do this. Thanks for any help you can provide
EDIT: I was able to find a way through transanno
https://github.com/informationsea/transanno
Does this LiftoverVCF tool retain phase information in a phased VCF file? In other words, if I lift over a phased VCF file from hg38 to hg19, is the phase information still accurate in the lifted over file?
Please sign in to leave a comment.