Genome Analysis Toolkit

Variant Discovery in High-Throughput Sequencing Data

GATK process banner

Need Help?

Search our documentation

Community Forum

Hi, How can we help?

Developed in the Data Sciences Platform at the Broad Institute, the toolkit offers a wide variety of tools with a primary focus on variant discovery and genotyping. Its powerful processing engine and high-performance computing features make it capable of taking on projects of any size. Learn more

MarkDuplicatesGATK (EXPERIMENTAL) Follow

  • Metagenomics
  • PathSeqBuildKmers
  • PathSeqBuildReferenceTaxonomy
  • PathSeqBwaSpark
  • PathSeqFilterSpark
  • PathSeqPipelineSpark
  • PathSeqScoreSpark
  • Other
  • CreateHadoopBamSplittingIndex
  • FifoBuffer (Picard)
  • FixCallSetSampleOrdering
  • GatherBQSRReports
  • GatherTranches
  • IndexFeatureFile
  • ParallelCopyGCSDirectoryIntoHDFSSpark
  • Read Data Manipulation
  • AddCommentsToBam (Picard)
  • AddOrReplaceReadGroups (Picard)
  • ApplyBQSR
  • ApplyBQSRSpark
  • BQSRPipelineSpark
  • BamToBfq (Picard)
  • BaseRecalibrator
  • BaseRecalibratorSpark
  • BaseRecalibratorSparkSharded
  • BuildBamIndex (Picard)
  • BwaAndMarkDuplicatesPipelineSpark
  • BwaSpark
  • CleanSam (Picard)
  • ClipReads
  • ConvertHeaderlessHadoopBamShardToBam
  • DownsampleSam (Picard)
  • ExtractOriginalAlignmentRecordsByNameSpark
  • FastqToSam (Picard)
  • FilterSamReads (Picard)
  • FixMateInformation (Picard)
  • FixMisencodedBaseQualityReads
  • GatherBamFiles (Picard)
  • LeftAlignIndels
  • MarkDuplicates (Picard)
  • MarkDuplicatesGATK
  • MarkDuplicatesSpark
  • MarkDuplicatesWithMateCigar (Picard)
  • MergeBamAlignment (Picard)
  • MergeSamFiles (Picard)
  • PositionBasedDownsampleSam (Picard)
  • PrintReads
  • PrintReadsSpark
  • ReorderSam (Picard)
  • ReplaceSamHeader (Picard)
  • RevertBaseQualityScores
  • RevertOriginalBaseQualitiesAndAddMateCigar (Picard)
  • RevertSam (Picard)
  • SamFormatConverter (Picard)
  • SamToFastq (Picard)
  • SetNmAndUqTags (Picard)
  • SetNmMdAndUqTags (Picard)
  • SimpleMarkDuplicatesWithMateCigar (Picard)
  • SortSam (Picard)
  • SortSamSpark
  • SplitNCigarReads
  • SplitReads
  • SplitSamByLibrary (Picard)
  • SplitSamByNumberOfReads (Picard)
  • UmiAwareMarkDuplicatesWithMateCigar (Picard)
  • UnmarkDuplicates
  • Reference
  • BaitDesigner (Picard)
  • BwaMemIndexImageCreator
  • CreateSequenceDictionary (Picard)
  • ExtractSequences (Picard)
  • FindBadGenomicKmersSpark
  • NonNFastaSize (Picard)
  • NormalizeFasta (Picard)
  • ScatterIntervalsByNs (Picard)
  • Short Variant Discovery
  • CombineGVCFs
  • GenomicsDBImport
  • GenotypeGVCFs
  • HaplotypeCaller
  • HaplotypeCallerSpark
  • Mutect2
  • ReadsPipelineSpark
  • Structural Variant Discovery
  • DiscoverVariantsFromContigAlignmentsSAMSpark
  • ExtractSVEvidenceSpark
  • FindBreakpointEvidenceSpark
  • StructuralVariationDiscoveryPipelineSpark
  • SvDiscoverFromLocalAssemblyContigAlignmentsSpark
  • Variant Evaluation and Refinement
  • AnnotatePairOrientation
  • AnnotateVcfWithBamDepth
  • AnnotateVcfWithExpectedAlleleFraction
  • CalculateGenotypePosteriors
  • CalculateMixingFractions
  • Concordance
  • CountFalsePositives
  • CountVariants
  • CountVariantsSpark
  • FindMendelianViolations (Picard)
  • Funcotator
  • GenotypeConcordance (Picard)
  • ValidateBasicSomaticShortMutations
  • ValidateVariants
  • VariantsToTable
  • Variant Filtering
  • ApplyVQSR
  • CreateSomaticPanelOfNormals
  • FilterByOrientationBias
  • FilterMutectCalls
  • FilterVcf (Picard)
  • VariantFiltration
  • VariantRecalibrator
  • Variant Manipulation
  • FixVcfHeader (Picard)
  • GatherVcfs (Picard)
  • GatherVcfsCloud
  • LiftoverVcf (Picard)
  • MakeSitesOnlyVcf (Picard)
  • MergeVcfs (Picard)
  • PrintVariantsSpark
  • RemoveNearbyIndels
  • RenameSampleInVcf (Picard)
  • SelectVariants
  • SortVcf (Picard)
  • SplitVcfs (Picard)
  • UpdateVCFSequenceDictionary
  • UpdateVcfSequenceDictionary (Picard)
  • VcfFormatConverter (Picard)
  • VcfToIntervalList (Picard)

  • Base Calling
  • CheckIlluminaDirectory (Picard)
  • CollectIlluminaBasecallingMetrics (Picard)
  • CollectIlluminaLaneMetrics (Picard)
  • ExtractIlluminaBarcodes (Picard)
  • IlluminaBasecallsToFastq (Picard)
  • IlluminaBasecallsToSam (Picard)
  • MarkIlluminaAdapters (Picard)

  • Read Filters
  • AlignmentAgreesWithHeaderReadFilter
  • AllowAllReadsReadFilter
  • AmbiguousBaseReadFilter
  • CigarContainsNoNOperator
  • FirstOfPairReadFilter
  • FragmentLengthReadFilter
  • GoodCigarReadFilter
  • HasReadGroupReadFilter
  • LibraryReadFilter
  • MappedReadFilter
  • MappingQualityAvailableReadFilter
  • MappingQualityNotZeroReadFilter
  • MappingQualityReadFilter
  • MatchingBasesAndQualsReadFilter
  • MateDifferentStrandReadFilter
  • MateOnSameContigOrNoMappedMateReadFilter
  • MetricsReadFilter
  • NonZeroFragmentLengthReadFilter
  • NonZeroReferenceLengthAlignmentReadFilter
  • NotDuplicateReadFilter
  • NotSecondaryAlignmentReadFilter
  • NotSupplementaryAlignmentReadFilter
  • OverclippedReadFilter
  • PairedReadFilter
  • PassesVendorQualityCheckReadFilter
  • PlatformReadFilter
  • PlatformUnitReadFilter
  • PrimaryLineReadFilter
  • ProperlyPairedReadFilter
  • ReadGroupBlackListReadFilter
  • ReadGroupReadFilter
  • ReadLengthEqualsCigarLengthReadFilter
  • ReadLengthReadFilter
  • ReadNameReadFilter
  • ReadStrandFilter
  • SampleReadFilter
  • SecondOfPairReadFilter
  • SeqIsStoredReadFilter
  • ValidAlignmentEndReadFilter
  • ValidAlignmentStartReadFilter
  • WellformedReadFilter
  • Variant Annotations
  • AS_BaseQualityRankSumTest
  • AS_FisherStrand
  • AS_InbreedingCoeff
  • AS_MappingQualityRankSumTest
  • AS_QualByDepth
  • AS_RMSMappingQuality
  • AS_ReadPosRankSumTest
  • AS_StrandOddsRatio
  • BaseQuality
  • BaseQualityRankSumTest
  • ChromosomeCounts
  • ClippingRankSumTest
  • Coverage
  • DepthPerAlleleBySample
  • DepthPerSampleHC
  • ExcessHet
  • FisherStrand
  • FragmentLength
  • GenotypeSummaries
  • InbreedingCoeff
  • LikelihoodRankSumTest
  • MappingQuality
  • MappingQualityRankSumTest
  • MappingQualityZero
  • OxoGReadCounts
  • PossibleDeNovo
  • QualByDepth
  • RMSMappingQuality
  • ReadPosRankSumTest
  • ReadPosition
  • ReferenceBases
  • SampleList
  • StrandArtifact
  • StrandBiasBySample
  • StrandOddsRatio
  • TandemRepeat
  • UniqueAltReadCount
  • Examines aligned records in the supplied SAM/BAM/CRAM file to locate duplicate molecules.

    Category Read Data Manipulation


    Overview

    A better duplication marking algorithm that handles all cases including clipped and gapped alignments.

    MarkDuplicatesGATK specific arguments

    This table summarizes the command-line arguments that are specific to this tool. For more details on each argument, see the list further down below the table or click on an argument name to jump directly to that entry in the list.

    Argument name(s) Default value Summary
    Required Arguments
    --input
     -I
    [] One or more input SAM/BAM/CRAM files to analyze. Must be coordinate sorted.
    --METRICS_FILE
     -M
    null File to write duplication metrics to
    --output
     -O
    null The output file to write marked records to
    Optional Tool Arguments
    --arguments_file
    [] read one or more arguments files and add them to the command line
    --ASSUME_SORTED
     -AS
    false If true, assume that the input file is coordinate sorted even if the header says otherwise.
    --COMMENT
     -CO
    [] Comment(s) to include in the output file's header.
    --DUPLICATE_SCORING_STRATEGY
     -DS
    SUM_OF_BASE_QUALITIES The scoring strategy for choosing the non-duplicate among candidates.
    --gcs-max-retries
     -gcs-retries
    20 If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection
    --help
     -h
    false display the help message
    --MAX_FILE_HANDLES_FOR_READ_ENDS_MAP
     -MAX_FILE_HANDLES
    8000 Maximum number of file handles to keep open when spilling read ends to disk. Set this number a little lower than the per-process maximum number of file that may be open. This number can be found by executing the 'ulimit -n' command on a Unix system.
    --MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP
     -MAX_SEQS
    50000 This option is obsolete. ReadEnds will always be spilled to disk.
    --OPTICAL_DUPLICATE_PIXEL_DISTANCE
    100 The maximum offset between two duplicate clusters in order to consider them optical duplicates. This should usually be set to some fairly small number (e.g. 5-10 pixels) unless using later versions of the Illumina pipeline that multiply pixel values by 10, in which case 50-100 is more normal.
    --PROGRAM_GROUP_COMMAND_LINE
     -PG_COMMAND
    null Value of CL tag of PG record to be created. If not supplied the command line will be detected automatically.
    --PROGRAM_GROUP_NAME
     -PG_NAME
    MarkDuplicatesGATK Value of PN tag of PG record to be created.
    --PROGRAM_GROUP_VERSION
     -PG_VERSION
    null Value of VN tag of PG record to be created. If not specified, the version will be detected automatically.
    --PROGRAM_RECORD_ID
     -PG
    MarkDuplicatesGATK The program record ID for the @PG record(s) created by this program. Set to null to disable PG record creation. This string may have a suffix appended to avoid collision with other program record IDs.
    --READ_NAME_REGEX
    [a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* Regular expression that can be used to parse read names in the incoming SAM file. Read names are parsed to extract three variables: tile/region, x coordinate and y coordinate. These values are used to estimate the rate of optical duplication in order to give a more accurate estimated library size. Set this option to null to disable optical duplicate detection. The regular expression should contain three capture groups for the three variables, in order. It must match the entire read name. Note that if the default regex is specified, a regex match is not actually done, but instead the read name is split on colon character. For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.
    --REMOVE_DUPLICATES
    false If true do not write duplicates to the output file instead of writing them with appropriate flags set.
    --reportMemoryStats
    false Report Memory Stats at various times during the run
    --SORTING_COLLECTION_SIZE_RATIO
    0.25 This number, plus the maximum RAM available to the JVM, determine the memory footprint used by some of the sorting collections. If you are running out of memory, try reducing this number.
    --version
    false display the version number for this tool
    Optional Common Arguments
    --COMPRESSION_LEVEL
    5 Compression level for all compressed files created (e.g. BAM and GELI).
    --CREATE_INDEX
    false Whether to create a BAM index when writing a coordinate-sorted BAM file.
    --CREATE_MD5_FILE
    false Whether to create an MD5 digest for any BAM or FASTQ files created.
    --gatk-config-file
    null A configuration file to use with the GATK.
    --MAX_RECORDS_IN_RAM
    500000 When writing SAM files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort a SAM file, and increases the amount of RAM needed.
    --QUIET
    false Whether to suppress job-summary info on System.err.
    --reference
     -R
    null Reference sequence file.
    --TMP_DIR
    [] Undocumented option
    --use-jdk-deflater
     -jdk-deflater
    false Whether to use the JdkDeflater (as opposed to IntelDeflater)
    --use-jdk-inflater
     -jdk-inflater
    false Whether to use the JdkInflater (as opposed to IntelInflater)
    --VALIDATION_STRINGENCY
    STRICT Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.
    --verbosity
    INFO Control verbosity of logging.
    Advanced Arguments
    --showHidden
    false display hidden arguments

    Argument details

    Arguments in this list are specific to this tool. Keep in mind that other arguments are available that are shared with other tools (e.g. command-line GATK arguments); see Inherited arguments above.


    --arguments_file / NA

    read one or more arguments files and add them to the command line

    List[File]  []


    --ASSUME_SORTED / -AS

    If true, assume that the input file is coordinate sorted even if the header says otherwise.

    boolean  false


    --COMMENT / -CO

    Comment(s) to include in the output file's header.

    List[String]  []


    --COMPRESSION_LEVEL / NA

    Compression level for all compressed files created (e.g. BAM and GELI).

    int  5  [ [ -∞  ∞ ] ]


    --CREATE_INDEX / NA

    Whether to create a BAM index when writing a coordinate-sorted BAM file.

    Boolean  false


    --CREATE_MD5_FILE / NA

    Whether to create an MD5 digest for any BAM or FASTQ files created.

    boolean  false


    --DUPLICATE_SCORING_STRATEGY / -DS

    The scoring strategy for choosing the non-duplicate among candidates.

    The --DUPLICATE_SCORING_STRATEGY argument is an enumerated type (ScoringStrategy), which can have one of the following values:

    SUM_OF_BASE_QUALITIES
    TOTAL_MAPPED_REFERENCE_LENGTH
    RANDOM

    ScoringStrategy  SUM_OF_BASE_QUALITIES


    --gatk-config-file / NA

    A configuration file to use with the GATK.

    String  null


    --gcs-max-retries / -gcs-retries

    If the GCS bucket channel errors out, how many times it will attempt to re-initiate the connection

    int  20  [ [ -∞  ∞ ] ]


    --help / -h

    display the help message

    boolean  false


    --input / -I

    One or more input SAM/BAM/CRAM files to analyze. Must be coordinate sorted.

    R List[File]  []


    --MAX_FILE_HANDLES_FOR_READ_ENDS_MAP / -MAX_FILE_HANDLES

    Maximum number of file handles to keep open when spilling read ends to disk. Set this number a little lower than the per-process maximum number of file that may be open. This number can be found by executing the 'ulimit -n' command on a Unix system.

    int  8000  [ [ -∞  ∞ ] ]


    --MAX_RECORDS_IN_RAM / NA

    When writing SAM files that need to be sorted, this will specify the number of records stored in RAM before spilling to disk. Increasing this number reduces the number of file handles needed to sort a SAM file, and increases the amount of RAM needed.

    Integer  500000  [ [ -∞  ∞ ] ]


    --MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP / -MAX_SEQS

    This option is obsolete. ReadEnds will always be spilled to disk.
    If more than this many sequences in SAM file, don't spill to disk because there will not be enough file handles.

    int  50000  [ [ -∞  ∞ ] ]


    --METRICS_FILE / -M

    File to write duplication metrics to

    R File  null


    --OPTICAL_DUPLICATE_PIXEL_DISTANCE / NA

    The maximum offset between two duplicate clusters in order to consider them optical duplicates. This should usually be set to some fairly small number (e.g. 5-10 pixels) unless using later versions of the Illumina pipeline that multiply pixel values by 10, in which case 50-100 is more normal.

    int  100  [ [ -∞  ∞ ] ]


    --output / -O

    The output file to write marked records to

    R File  null


    --PROGRAM_GROUP_COMMAND_LINE / -PG_COMMAND

    Value of CL tag of PG record to be created. If not supplied the command line will be detected automatically.

    String  null


    --PROGRAM_GROUP_NAME / -PG_NAME

    Value of PN tag of PG record to be created.

    String  MarkDuplicatesGATK


    --PROGRAM_GROUP_VERSION / -PG_VERSION

    Value of VN tag of PG record to be created. If not specified, the version will be detected automatically.

    String  null


    --PROGRAM_RECORD_ID / -PG

    The program record ID for the @PG record(s) created by this program. Set to null to disable PG record creation. This string may have a suffix appended to avoid collision with other program record IDs.

    String  MarkDuplicatesGATK


    --QUIET / NA

    Whether to suppress job-summary info on System.err.

    Boolean  false


    --READ_NAME_REGEX / NA

    Regular expression that can be used to parse read names in the incoming SAM file. Read names are parsed to extract three variables: tile/region, x coordinate and y coordinate. These values are used to estimate the rate of optical duplication in order to give a more accurate estimated library size. Set this option to null to disable optical duplicate detection. The regular expression should contain three capture groups for the three variables, in order. It must match the entire read name. Note that if the default regex is specified, a regex match is not actually done, but instead the read name is split on colon character. For 5 element names, the 3rd, 4th and 5th elements are assumed to be tile, x and y values. For 7 element names (CASAVA 1.8), the 5th, 6th, and 7th elements are assumed to be tile, x and y values.

    String  [a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).*


    --reference / -R

    Reference sequence file.

    File  null


    --REMOVE_DUPLICATES / NA

    If true do not write duplicates to the output file instead of writing them with appropriate flags set.

    boolean  false


    --reportMemoryStats / NA

    Report Memory Stats at various times during the run

    boolean  false


    --showHidden / -showHidden

    display hidden arguments

    boolean  false


    --SORTING_COLLECTION_SIZE_RATIO / NA

    This number, plus the maximum RAM available to the JVM, determine the memory footprint used by some of the sorting collections. If you are running out of memory, try reducing this number.

    double  0.25  [ [ -∞  ∞ ] ]


    --TMP_DIR / NA

    Undocumented option

    List[File]  []


    --use-jdk-deflater / -jdk-deflater

    Whether to use the JdkDeflater (as opposed to IntelDeflater)

    boolean  false


    --use-jdk-inflater / -jdk-inflater

    Whether to use the JdkInflater (as opposed to IntelInflater)

    boolean  false


    --VALIDATION_STRINGENCY / NA

    Validation stringency for all SAM files read by this program. Setting stringency to SILENT can improve performance when processing a BAM file in which variable-length data (read, qualities, tags) do not otherwise need to be decoded.

    The --VALIDATION_STRINGENCY argument is an enumerated type (ValidationStringency), which can have one of the following values:

    STRICT
    LENIENT
    SILENT

    ValidationStringency  STRICT


    --verbosity / -verbosity

    Control verbosity of logging.

    The --verbosity argument is an enumerated type (LogLevel), which can have one of the following values:

    ERROR
    WARNING
    INFO
    DEBUG

    LogLevel  INFO


    --version / NA

    display the version number for this tool

    boolean  false


    Return to top


    GATK version 4.0.2.0 built at 25-48-2019 02:48:40.

    0 comments

    Please sign in to leave a comment.

    Powered by Zendesk