Funcotator Problem docker image code:400 message: Bucket is a requester pays bucket but no user project provided.

Follow

Robert Bremel

• May 08, 2025 15:30

REQUIRED for all errors and issues:
a) GATK version used:
b) Exact command used:
c) Entire program log:

It has been down with this problem all week.  It had the same problem in February and was fixed, but has happened again?

 

 

docker run -d --rm  --name GBM_00188a1_SavyKomodo_funky_gatk   -v  G:/gatk_dock:/gatk/mydata   broadinstitute/gatk:latest  sleep infinity  7fa94104db22f826018159556875043e97ef5ddb5af184bbf6f9696c920b8ed6

 

"Running."

Docker RUN mode 

Docker Command:

 

docker exec GBM_00188a1_SavyKomodo_funky_gatk   ./gatk Funcotator   -R                  mydata/refs/Homo_sapiens_assembly38.fasta -V                  mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_filtered_hg38.vcf -O                  mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_funcotator_hg38.maf --output-file-format MAF  --data-sources-path mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/ --verbosity DEBUG  --ref-version hg38 

Using GATK jar /gatk/gatk-package-4.6.1.0-local.jar

Running:

    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.6.1.0-local.jar Funcotator -R mydata/refs/Homo_sapiens_assembly38.fasta -V mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_filtered_hg38.vcf -O mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_funcotator_hg38.maf --output-file-format MAF --data-sources-path mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/ --verbosity DEBUG --ref-version hg38

15:16:15.527 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.6.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so

15:16:15.563 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /tmp/libgkl_compression9473774619665196619.so

SLF4J(W): Class path contains multiple SLF4J providers.

SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@5b8853]

SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@1b8aaeab]

SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@5b8853]

15:16:15.946 INFO  Funcotator - ------------------------------------------------------------

15:16:15.951 INFO  Funcotator - The Genome Analysis Toolkit (GATK) v4.6.1.0

15:16:15.951 INFO  Funcotator - For support and documentation go to https://software.broadinstitute.org/gatk/

15:16:15.951 INFO  Funcotator - Executing as root@7fa94104db22 on Linux v5.15.167.4-microsoft-standard-WSL2 amd64

15:16:15.952 INFO  Funcotator - Java runtime: OpenJDK 64-Bit Server VM v17.0.12+7-Ubuntu-1ubuntu222.04

15:16:15.952 INFO  Funcotator - Start Date/Time: May 8, 2025 at 3:16:15 PM GMT

15:16:15.952 INFO  Funcotator - ------------------------------------------------------------

15:16:15.953 INFO  Funcotator - ------------------------------------------------------------

15:16:15.954 INFO  Funcotator - HTSJDK Version: 4.1.3

15:16:15.954 INFO  Funcotator - Picard Version: 3.3.0

15:16:15.955 INFO  Funcotator - Built for Spark Version: 3.5.0

15:16:15.957 INFO  Funcotator - HTSJDK Defaults.BUFFER_SIZE : 131072

15:16:15.957 INFO  Funcotator - HTSJDK Defaults.COMPRESSION_LEVEL : 2

15:16:15.957 INFO  Funcotator - HTSJDK Defaults.CREATE_INDEX : false

15:16:15.957 INFO  Funcotator - HTSJDK Defaults.CREATE_MD5 : false

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.CUSTOM_READER_FACTORY : 

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.REFERENCE_FASTA : null

15:16:15.958 INFO  Funcotator - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL

15:16:15.959 INFO  Funcotator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

15:16:15.959 INFO  Funcotator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

15:16:15.959 INFO  Funcotator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

15:16:15.959 INFO  Funcotator - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false

15:16:15.959 DEBUG ConfigFactory - Configuration file values: 

15:16:15.964 DEBUG ConfigFactory - gcsMaxRetries = 20

15:16:15.964 DEBUG ConfigFactory - gcsProjectForRequesterPays = 

15:16:15.964 DEBUG ConfigFactory - gatk_stacktrace_on_user_exception = false

15:16:15.964 DEBUG ConfigFactory - samjdk.use_async_io_read_samtools = false

15:16:15.964 DEBUG ConfigFactory - samjdk.use_async_io_write_samtools = true

15:16:15.965 DEBUG ConfigFactory - samjdk.use_async_io_write_tribble = false

15:16:15.965 DEBUG ConfigFactory - samjdk.compression_level = 2

15:16:15.965 DEBUG ConfigFactory - spark.kryoserializer.buffer.max = 512m

15:16:15.965 DEBUG ConfigFactory - spark.driver.maxResultSize = 0

15:16:15.965 DEBUG ConfigFactory - spark.driver.userClassPathFirst = true

15:16:15.965 DEBUG ConfigFactory - spark.io.compression.codec = lzf

15:16:15.965 DEBUG ConfigFactory - spark.executor.memoryOverhead = 600

15:16:15.966 DEBUG ConfigFactory - spark.driver.extraJavaOptions = 

15:16:15.966 DEBUG ConfigFactory - spark.executor.extraJavaOptions = 

15:16:15.966 DEBUG ConfigFactory - codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]

15:16:15.966 DEBUG ConfigFactory - read_filter_packages = [org.broadinstitute.hellbender.engine.filters]

15:16:15.967 DEBUG ConfigFactory - annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]

15:16:15.967 DEBUG ConfigFactory - cloudPrefetchBuffer = 40

15:16:15.967 DEBUG ConfigFactory - cloudIndexPrefetchBuffer = -1

15:16:15.967 DEBUG ConfigFactory - createOutputBamIndex = true

15:16:15.967 INFO  Funcotator - Deflater: IntelDeflater

15:16:15.967 INFO  Funcotator - Inflater: IntelInflater

15:16:15.973 INFO  Funcotator - GCS max retries/reopens: 20

15:16:15.974 INFO  Funcotator - Requester pays: disabled

15:16:15.975 INFO  Funcotator - Initializing engine

15:16:16.424 INFO  FeatureManager - Using codec VCFCodec to read file file:///gatk/mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_filtered_hg38.vcf

15:16:16.729 INFO  Funcotator - Done initializing engine

15:16:16.730 INFO  Funcotator - Validating sequence dictionaries...

15:16:16.767 INFO  Funcotator - Processing user transcripts/defaults/overrides...

15:16:16.769 INFO  Funcotator - Initializing data sources...

15:16:16.773 INFO  DataSourceUtils - Initializing data sources from directory: mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/

15:16:16.790 WARN  DataSourceUtils - README file has improperly formatted version string: Version:          1.7.20200429s

15:16:16.792 WARN  DataSourceUtils - Unable to read version information from data sources info/readme file: file:///gatk/mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/MANIFEST.txt

15:16:16.792 INFO  DataSourceUtils - Data sources source: ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/funcotator/funcotator_dataSources.v1.7.20200429s.tar.gz

15:16:16.792 INFO  DataSourceUtils - Data sources alternate source: gs://broad-public-datasets/funcotator/funcotator_dataSources.v1.7.20200429s.tar.gz

15:16:16.873 INFO  DataSourceUtils - Resolved data source file path: file:///gatk/gencode_xhgnc_v90_38.hg38.tsv -> file:///gatk/mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/gencode_xhgnc/hg38/gencode_xhgnc_v90_38.hg38.tsv

15:16:16.892 INFO  DataSourceUtils - Resolved data source file path: file:///gatk/oreganno.tsv -> file:///gatk/mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/oreganno/hg38/oreganno.tsv

15:16:16.910 INFO  DataSourceUtils - Resolved data source file path: file:///gatk/simple_uniprot_Dec012014.tsv -> file:///gatk/mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/simple_uniprot/hg38/simple_uniprot_Dec012014.tsv

15:16:16.928 INFO  DataSourceUtils - Resolved data source file path: file:///gatk/Cosmic.db -> file:///gatk/mydata/dataSourcesFolder/funcotator_dataSources.v1.7.20200521s/cosmic/hg38/Cosmic.db

15:16:18.283 INFO  Funcotator - Shutting down engine

15:16:18.284 DEBUG FeatureDataSource - Cache statistics for FeatureInput drivingVariantFile:/gatk/mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_filtered_hg38.vcf:

15:16:18.285 DEBUG FeatureCache - Cache hit rate  was 0.00% (0 out of 0 total queries)

15:16:18.286 DEBUG FeatureDataSource - Cache statistics for FeatureInput drivingVariantFile:/gatk/mydata/GBM_00188a1_SavyKomodo/analysis/GBM_00188a1-85919_DT_SavyKomodo_mutect2_filtered_hg38.vcf:

15:16:18.286 DEBUG FeatureCache - Cache hit rate  was 0.00% (0 out of 0 total queries)

[May 8, 2025 at 3:16:18 PM GMT] org.broadinstitute.hellbender.tools.funcotator.Funcotator done. Elapsed time: 0.05 minutes.

Runtime.totalMemory()=285212672

code:      400

message:   Bucket is a requester pays bucket but no user project provided.

reason:    required

location:  null

retryable: false

com.google.cloud.storage.StorageException: Bucket is a requester pays bucket but no user project provided.

at com.google.cloud.storage.StorageException.translate(StorageException.java:170)

at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:329)

at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:554)

at com.google.cloud.storage.StorageImpl.lambda$internalGetBlob$63(StorageImpl.java:1585)

at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)

at com.google.cloud.RetryHelper.run(RetryHelper.java:76)

at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)

at com.google.cloud.storage.Retrying.run(Retrying.java:65)

at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1524)

at com.google.cloud.storage.StorageImpl.internalGetBlob(StorageImpl.java:1583)

at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:328)

at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.checkAccess(CloudStorageFileSystemProvider.java:768)

at java.base/java.nio.file.Files.exists(Files.java:2527)

at org.broadinstitute.hellbender.tools.funcotator.dataSources.DataSourceUtils.assertPathFilePropertiesField(DataSourceUtils.java:972)

at org.broadinstitute.hellbender.tools.funcotator.dataSources.DataSourceUtils.assertConfigFilePropertiesAreValid(DataSourceUtils.java:875)

at org.broadinstitute.hellbender.tools.funcotator.dataSources.DataSourceUtils.getAndValidateDataSourcesFromPaths(DataSourceUtils.java:239)

at org.broadinstitute.hellbender.tools.funcotator.Funcotator.onTraversalStart(Funcotator.java:781)

at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1117)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:150)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:203)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:222)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:166)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:209)

at org.broadinstitute.hellbender.Main.main(Main.java:306)

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request

GET https://storage.googleapis.com/storage/v1/b/broad-public-datasets/o/funcotator%2FgnomAD_2.1_VCF_INFO_AF_Only%2Fhg38%2Fgnomad.exomes.r2.1.sites.liftoverToHg38.INFO_ANNOTATIONS_FIXED.vcf.gz?fields=bucket,name,id&projection=full

{

  "code" : 400,

  "errors" : [ {

    "domain" : "global",

    "message" : "Bucket is a requester pays bucket but no user project provided.",

    "reason" : "required"

  } ],

  "message" : "Bucket is a requester pays bucket but no user project provided."

}

at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)

at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)

at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)

at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:551)

... 21 more

 

0

• 

  Faezeh Darbaniyan

  • May 12, 2025 19:05

  Hi,

  I have the same issue and wonder if this is going to be fixed. Please let us know how long it may take.

  Thanks a lot for generating this awesome tool.

  Faezeh

  0

  Comment actions Permalink
• 

  A. Brink

  • May 15, 2025 13:25
  • Edited

  Same problem here with Funcotator (GATK 4.6.1.0) using the non-Docker version.

  "Bucket is a requester pays bucket but no user project provided."

  Hope that there will be a fix soon!

  Addition: same problem with GATK 4.6.2.0, although I hoped that one of the highlights of this update ("Funcotator data location moved") would have fixed the problem. Apparently not.

   

   

   

  0

  Comment actions Permalink
• 

  Robert Bremel

  • May 15, 2025 15:09

  This seems to be caused by the gnomAD buckets.  Using the v1.8 funcotator_datasources all the other annotations except gnomAD appear to run without error in Docker.  Including the genomic or somatic gnomAD datasources in the Funcotator runtime argument generates the error

  --data-sources-path mydata/dataSourcesFolder/funcotator_dataSources.v1.8.hg38.20230908s

  or 

  --data-sources-path mydata/dataSourcesFolder/funcotator_dataSources.v1.8.hg38.20230908g

   

   

  0

  Comment actions Permalink
• 

  Viraja Palwai

  • May 21, 2025 17:26
  • Edited

  Hi!

  I have a similar problem with Funcotator (GATK 4.6.2.0) non-Docker version. I Tried downloading manually using gsutil and it throws the error, "This bucket has Requester Pays enabled".

  Error with Funcotator:

  {

    "code" : 400,

    "errors" : [ {

      "domain" : "global",

      "message" : "Bucket is a requester pays bucket but no user project provided.",

      "reason" : "required"

    } ],

    "message" : "Bucket is a requester pays bucket but no user project provided."

  }

  0

  Comment actions Permalink
• 

  Viraja Palwai

  • May 22, 2025 23:41
  • Edited

  Hi!

  I just want share what worked for me.

  First you need to create a google cloud account, then create a project and activate the billing.   When you create a project, you will get a project ID which needs to be entered with --gcs-project-for-requester-pays < project ID> (Apart from other required fields) for FuncotatorDataSourceDownloader.

  Example: gatk FuncotatorDataSourceDownloader --germline --hg38 --validate-integrity --extract-after-download --gcs-project-for-requester-pays <project ID>

  Before running the pre packed downloader, login into your cloud account so that it has your account details and project ID. Then, go to your cloud account > IAM  > select you account (email in my case) > grant access > select role and look for "storage object viewer" (or just type) and save.

  Then run FuncotatorDataSourceDownloader and tar gnomAD gz files, and then run Funcotator.

  Hope this helps!

   

   

  0

  Comment actions Permalink
• 

  A. Brink

  • May 26, 2025 13:28
  • Edited

  Hi, I'm glad I found a fix today that does not need a google account nor billing. After reading the highlights of the most recent GATK version (https://github.com/broadinstitute/gatk/releases) in which the "Funcotator Data Location Moved" was announced, it occured to me that maybe this new location has to be known for my downloaded Funcotator as well. It turned out that indeed gnomAD (thanks Robert Bremel) was the one causing problems. After fixing the gnomAD_exome.config and gnomAD_genome.config files in the downloaded gnomAD folders of my funcotator_dataSources.v1.8.hg19.20230908s database (change src_file=gs://broad-public-datasets to src_file=gs://gcp-public-data--broad-references) Funcotator was running smoothly again.

  1

  Comment actions Permalink

Please sign in to leave a comment.