Dear GATK!

my aim is to transform an interval file (Exome-Agilent_V6_UTR.bed with 296,037 lines) to have <30,000 lines to be able to run GATK-haplotype-calller with germline samples on Terra (formerly firecloud). I'm finding germline muatations on a cancer cohort (of which I have Tumor/Normal Whole Exome Seq data)

Terra support says my problem is gatk specific so i should open an issue in GATK forum, and suggesetd i explore step one in gatk blog
https://gatk.broadinstitute.org/hc/en-us/articles/360037226712-PreprocessIntervals
https://gatk.broadinstitute.org/hc/en-us/articles/360035531152--How-to-Call-common-and-rare-germline-copy-number-variants

workflow being used; https://portal.firecloud.org/?return=terra#methods/gatk/haplotypecaller-gvcf-gatk4/17
my original interval list (Exome-Agilent_V6_UTR.bed with 296,037 lines) found here; https://drive.google.com/drive/folders/1qiVXOd2Mqhtqyxt_kaxd-OvewWtcytDI?usp=sharing

• Question- How can I transform my interval list with 297,694 lines to <30,000 lines?

At this point I've exhaustesd all my ideas and would appreciate your insight.

Below, I have descibed all the approaches i have used so far which didn't work

• Approach one; gatk Picard PreprocessIntervals

1. using `gatk Picard PreprocessIntervals` and options `--padding 250` `--bin-length 0` and `-imr OVERLAPPING_ONLY` i added 250bp to both sides of cordinates. Reduced to 296,037 lines but still >35,000 lines i neeeded.
2. repeated step 1 separately with --padding of 500 and 1000 but all with `--bin-length 0` and `-imr OVERLAPPING_ONLY` but they all ended up giving the same 296,037 lines

• Approach two; AnnotateIntervals and FilterIntervals based on gc

1. I annotated intervals with gc values using `gatk AnnotateIntervals` and filtered based on gc. resulted in 296,032 lines
2. filtered the annotated intervals with `--maximum-gc-content 0.7` `gatk FilterIntervals`
had 287,386 lines. i guess going less than `--maximum-gc-content 0.7` will get me close to 30,000 lines I'm looking for but might end up throwing away regions that might be important downstream

• Approach 3; `gatk Picard PreprocessIntervals` `--padding 250` and merge intervals with bedtools

1. using `gatk Picard PreprocessIntervals` I added `--padding 250` to interval list, converted to .bed file
then i sorted and merged interval list (.bed format) using bedtools (and Maximum distance between features allowed for features to be merged option -d 10500)
https://bedtools.readthedocs.io/en/latest/content/tools/merge.html
this gave me final interval list with 34,082 lines, exactly what in needed but fails when i use it to run gatk-haplotype-caller

2. took the original interval list `Exome-Agilent_V6_UTR.bed`, skipped the padding step, sorted and merged intervals with bedtools (with option -d 10500)
but threw an error when running gatk haplotype caller
HaplotypeCaller; 23933; Could not build the path "chr13 45337173 45341624". It may refer to a filesystem not supported by this instance of Cromwell. Supported filesystems are: Google Cloud Storage, DRS. Failures: Google Cloud Storage: Path "chr13 45337173 45341624" does not have a gcs scheme (IllegalArgumentException) DRS: chr13 45337173 45341624 does not have a drs scheme. (IllegalArgumentException) Please refer to the documentation for more information on how to configure filesystems: http://cromwell.readthedocs.io/en/develop/backends/HPC/#filesystems

3. bedtools intersect interval_list in .bed format in previous steps with hg38 chromosome cordinates, used in running gatk-haplotype-caller but failed when i run gatk-haplotype caller on terra
HaplotypeCaller 20246 Could not build the path "chr3 177539325 177539660". It may refer to a filesystem not supported by this instance of Cromwell. Supported filesystems are: Google Cloud Storage, DRS. Failures: Google Cloud Storage: Path "chr3 177539325 177539660" does not have a gcs scheme (IllegalArgumentException) DRS: chr3 177539325 177539660 does not have a drs scheme. (IllegalArgumentException) Please refer to the documentation for more information on how to configure filesystems: http://cromwell.readthedocs.io/en/develop/backends/HPC/#filesystems

HaplotypeCaller 29366 Could not build the path "chr9 34048780 34049001". It may refer to a filesystem not supported by this instance of Cromwell. Supported filesystems are: Google Cloud Storage, DRS. Failures: Google Cloud Storage: Path "chr9 34048780 34049001" does not have a gcs scheme (IllegalArgumentException) DRS: chr9 34048780 34049001 does not have a drs scheme. (IllegalArgumentException) Please refer to the documentation for more information on how to configure filesystems: http://cromwell.readthedocs.io/en/develop/backends/HPC/#filesystems

• Approach four; split interval list to generate 50 cattered_calling_intervals, but failed when i run gatk-haplotype-caller

ScatterNode Workflow has scatter width 299404, which is more than the max scatter width 35000 allowed per scatter!

`

########code example#############
#######################################################
############# Aproach one #####################################################################
########################################################

####runninging the actual command
gatk PreprocessIntervals -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -L Exome-Agilent_V6_UTR.bed --bin-length 0 -imr OVERLAPPING_ONLY --padding 250 -O colonPR.hg38.preprocessed_intervals.interval_list

[ahunos01@lc02a29 intervals_preprocessing]$ gatk PreprocessIntervals -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -L Exome-Agilent_V6_UTR.bed --bin-length 0 -imr OVERLAPPING_ONLY --padding 250 -O colonPR.hg38.preprocessed_intervals.interval_list
Using GATK jar /hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar PreprocessIntervals -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -L Exome-Agilent_V6_UTR.bed --bin-length 0 -imr OVERLAPPING_ONLY --padding 250 -O colonPR.hg38.preprocessed_intervals.interval_list
21:51:01.947 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
21:51:02.098 INFO PreprocessIntervals - ------------------------------------------------------------
21:51:02.098 INFO PreprocessIntervals - The Genome Analysis Toolkit (GATK) v4.1.3.0
21:51:02.098 INFO PreprocessIntervals - For support and documentation go to https://software.broadinstitute.org/gatk/
21:51:02.099 INFO PreprocessIntervals - Executing as ahunos01@lc02a29 on Linux v3.10.0-957.el7.x86_64 amd64
21:51:02.099 INFO PreprocessIntervals - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_211-b12
21:51:02.099 INFO PreprocessIntervals - Start Date/Time: March 9, 2020 9:51:01 PM EDT
21:51:02.099 INFO PreprocessIntervals - ------------------------------------------------------------
21:51:02.099 INFO PreprocessIntervals - ------------------------------------------------------------
21:51:02.099 INFO PreprocessIntervals - HTSJDK Version: 2.20.1
21:51:02.099 INFO PreprocessIntervals - Picard Version: 2.20.5
21:51:02.099 INFO PreprocessIntervals - HTSJDK Defaults.COMPRESSION_LEVEL : 2
21:51:02.099 INFO PreprocessIntervals - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
21:51:02.099 INFO PreprocessIntervals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
21:51:02.099 INFO PreprocessIntervals - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
21:51:02.100 INFO PreprocessIntervals - Deflater: IntelDeflater
21:51:02.100 INFO PreprocessIntervals - Inflater: IntelInflater
21:51:02.100 INFO PreprocessIntervals - GCS max retries/reopens: 20
21:51:02.100 INFO PreprocessIntervals - Requester pays: disabled
21:51:02.100 INFO PreprocessIntervals - Initializing engine
21:51:03.982 INFO FeatureManager - Using codec BEDCodec to read file file:///sc/hydra/projects/canan/colon_cancer10312019/intervals_preprocessing/Exome-Agilent_V6_UTR.bed
21:51:04.407 INFO IntervalArgumentCollection - Processing 90499549 bp from intervals
21:51:04.446 INFO PreprocessIntervals - Done initializing engine
21:51:04.446 INFO ProgressMeter - Starting traversal
21:51:04.446 INFO ProgressMeter - Current Locus Elapsed Minutes Records Processed Records/Minute
21:51:04.447 INFO ProgressMeter - unmapped 0.0 0 NaN
21:51:04.447 INFO ProgressMeter - Traversal complete. Processed 0 total records in 0.0 minutes.
21:51:04.447 INFO PreprocessIntervals - Padding intervals...
21:51:04.512 INFO PreprocessIntervals - Generating bins...
21:51:04.544 INFO PreprocessIntervals - Filtering bins containing only Ns...
22:31:06.519 INFO PreprocessIntervals - Writing bins to /sc/hydra/projects/canan/colon_cancer10312019/intervals_preprocessing/colonPR.hg38.preprocessed_intervals.interval_list...
22:31:06.660 INFO PreprocessIntervals - PreprocessIntervals complete.
22:31:06.660 INFO PreprocessIntervals - Shutting down engine
[March 9, 2020 10:31:06 PM EDT] org.broadinstitute.hellbender.tools.copynumber.PreprocessIntervals done. Elapsed time: 40.08 minutes.
Runtime.totalMemory()=514850816
[ahunos01@lc02a29 intervals_preprocessing]$

###testing out new interval list
[ahunos01@lc02a29 intervals_preprocessing]$ cat colonPR.hg38.preprocessed_intervals.interval_list | grep -v "^@" | wc -l
296037 ###worked, but greater than than 30k

$ gatk PreprocessIntervals -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -L Exome-Agilent_V6_UTR.bed --bin-length 0 -imr OVERLAPPING_ONLY --padding 500 -O colonPR.hg38.preprocessed_intervals.500padding.interval_list
$ cat colonPR.hg38.preprocessed_intervals.500padding.interval_list | grep -v "^@"| wc -l
296037 

$ gatk PreprocessIntervals -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta -L Exome-Agilent_V6_UTR.bed --bin-length 0 -imr OVERLAPPING_ONLY --padding 1000 -O colonPR.hg38.preprocessed_intervals.1000padding.interval_list
$ cat colonPR.hg38.preprocessed_intervals.1000padding.interval_list |grep -v "^@" | wc -l
296037

#######################################################
############# Aproach two #####################################################################
########################################################

####Annotate interval list
gatk AnnotateIntervals \
-L colonPR.hg38.preprocessed_intervals.local.padding250.interval_list \
-R Homo_sapiens_assembly38.fasta \
-imr OVERLAPPING_ONLY \
-O colonPR.hg38.annotated.tsv

####filteringintervals
gatk FilterIntervals \
-L colonPR.hg38.preprocessed_intervals.local.padding250.interval_list \
--annotated-intervals colonPR.hg38.annotated.tsv \
-imr OVERLAPPING_ONLY \
-O colonPR.hg38.cohort.gc.filtered.interval_list

#####filtering intervals with gc max gc of 0.7
gatk FilterIntervals \
-L colonPR.hg38.preprocessed_intervals.local.padding250.interval_list \
--annotated-intervals colonPR.hg38.annotated.tsv \
--maximum-gc-content 0.7 \
-imr OVERLAPPING_ONLY \
-O colonPR.hg38.cohort.maxGC0.7.filtered.interval_list

###########################################################################################
#######################################################
############# Aproach three #################################################
########################################################

(base) [ahunos01@li03c04 intervals_preprocessing]$ gatk IntervalListToBed --INPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list --OUTPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list.bed
Using GATK jar /hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar IntervalListToBed --INPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list --OUTPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list.bed
23:09:26.364 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/hpc/packages/minerva-centos7/gatk/4.1.3.0/gatk-4.1.3.0/gatk-package-4.1.3.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
[Fri Mar 13 23:09:26 EDT 2020] IntervalListToBed --INPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list --OUTPUT colonPR.hg38.preprocessed_intervals.padding250.interval_list.bed --SCORE 500 --SORT true --VERBOSITY INFO --QUIET false --VALIDATION_STRINGENCY STRICT --COMPRESSION_LEVEL 2 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[Fri Mar 13 23:09:26 EDT 2020] Executing as ahunos01@li03c04 on Linux 3.10.0-957.10.1.el7.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_211-b12; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: Version:4.1.3.0
[Fri Mar 13 23:09:28 EDT 2020] picard.util.IntervalListToBed done. Elapsed time: 0.04 minutes.
Runtime.totalMemory()=2565341184
Tool returned:
0
(base) [ahunos01@li03c04 intervals_preprocessing]$

###sort bed file such that chromosome cordinates of rference file will be same as in .bed file
(base) [ahunos01@li03c04 intervals_preprocessing]$ bedtools sort -faidx Homo_sapiens_assembly38.fasta.fai.BedtoolsSort.txt -i colonPR.hg38.preprocessed_intervals.padding250.interval_list.bed > colonPR.hg38.preprocessed_intervals.padding250.interval_list.faidxSorted.bed

##bedtools merge
$ bedtools merge -i colonPR.hg38.preprocessed_intervals.padding250.interval_list.faidxSorted.bed -d 10500 > colonPR.hg38.preprocessed_intervals.padding250.interval_list.faidxSorted.d10500merged.bed
###Chromosome ordering is correct

#### using UCSC Kent Utilities kit and bedops
$ fetchChromSizes hg38 | awk -v FS="\t" -v OFS="\t" '{ print $1, "0", $2; }' | grep -v ".*_.*" | sort-bed - > hg38.nuclearOnly.bed

$ sort -k1,1V -k2,2n hg38.nuclearOnly.bed > hg38.nuclearOnly.sorted.bed

###intersect the interval list bed file with hg38 human reference genome cordinates
$ bedtools intersect -a colonPR.hg38.preprocessed_intervals.padding250.interval_list.faidxSorted.d10500merged.bed -b hg38.nuclearOnly.sorted.bed > colonPR.hg38.preprocessed_intervals.padding250.interval_list.faidxSorted.d10500merged.intersected.bed

$ bedtools intersect -a Exome-Agilent_V6_UTR.sorted.d12000merged.bed -b hg38.nuclearOnly.sorted.bed > Exome-Agilent_V6_UTR.sorted.d12000merged.intersected.bed

###########################################################################################
#######################################################
############# Aproach four #################################################
########################################################
####split interval lkist
gatk SplitIntervals \
-R Homo_sapiens_assembly38.fasta \
-L colonPR.hg38.preprocessed_intervals.interval_list \
--scatter-count 50 \
-O scattered_calling_intervals

(base) [ahunos01@li03c03 intervals_preprocessing]$ gsutil cp scattered_calling_intervals/ gs://fc-1f2cec15-4c83-4842-8fde-c79a1131b2dd/

Updates are available for some Cloud SDK components. To install them,
please run:
$ gcloud components update

Omitting directory "file://scattered_calling_intervals/". (Did you mean to do cp -r?)
CommandException: No URLs matched
(base) [ahunos01@li03c03 intervals_preprocessing]$ gsutil cp -r scattered_calling_intervals/ gs://fc-1f2cec15-4c83-4842-8fde-c79a1131b2dd/
Copying file://scattered_calling_intervals/0034-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0031-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0026-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0038-scattered.interval_list [Content-Type=application/octet-stream]...
\ [4 files][ 2.6 MiB/ 2.6 MiB]
==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m cp ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.

Copying file://scattered_calling_intervals/0018-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0004-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0001-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0048-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0005-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0029-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0043-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0008-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0036-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0049-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0009-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0010-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0011-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0013-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0006-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0020-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0016-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0014-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0019-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0035-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0032-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0033-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0003-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0028-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0037-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0017-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0046-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0030-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0047-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0042-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0045-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0023-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0015-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0027-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0025-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0012-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0007-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0021-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0024-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0039-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0002-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0000-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0040-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0022-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0041-scattered.interval_list [Content-Type=application/octet-stream]...
Copying file://scattered_calling_intervals/0044-scattered.interval_list [Content-Type=application/octet-stream]...
/ [50 files][ 32.2 MiB/ 32.2 MiB] 1.7 MiB/s
Operation completed over 50 objects/32.2 MiB.
(base) [ahunos01@li03c03 intervals_preprocessing]$ pwd
/sc/hydra/projects/canan/colon_cancer10312019/intervals_preprocessing
(base) [ahunos01@li03c03 intervals_preprocessing]$cd

###move to google console
ekwame001@cs*:~$ gsutil ls gs://fc-1f2cec15-4c83-4842-8fde-c79a1131b2dd/scattered_calling_intervals/ > ColonC_PR_scatter
ed_calling_intervals_list

`