FuncotatorDataSourceDownloader problem at google
Had a problem with Funcotator yesterday thought maybe the datasources had been corrupted so tried to download new set. Got same error
docker version 4.6.1.0 = docker "latest"
code: 400
message: Bucket is a requester pays bucket but no user project provided.
reason: required
location: null
retryable: false
com.google.cloud.storage.StorageException: Bucket is a requester pays bucket but no user project provided.
(gatk) root@3996f55bb7a7:/gatk# ./gatk FuncotatorDataSourceDownloader --somatic --hg38 --validate-integrity --extract-after-download --output mydata/tmp
Using GATK jar /gatk/gatk-package-4.6.1.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.6.1.0-local.jar FuncotatorDataSourceDownloader --somatic --hg38 --validate-integrity --extract-after-download --output mydata/tmp
14:51:31.608 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.6.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@37606fee]
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@33d28f0a]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@37606fee]
14:51:31.711 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------
14:51:31.715 INFO FuncotatorDataSourceDownloader - The Genome Analysis Toolkit (GATK) v4.6.1.0
14:51:31.715 INFO FuncotatorDataSourceDownloader - For support and documentation go to https://software.broadinstitute.org/gatk/
14:51:31.716 INFO FuncotatorDataSourceDownloader - Executing as root@3996f55bb7a7 on Linux v5.15.167.4-microsoft-standard-WSL2 amd64
14:51:31.716 INFO FuncotatorDataSourceDownloader - Java runtime: OpenJDK 64-Bit Server VM v17.0.12+7-Ubuntu-1ubuntu222.04
14:51:31.716 INFO FuncotatorDataSourceDownloader - Start Date/Time: February 22, 2025 at 2:51:31 PM GMT
14:51:31.716 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------
14:51:31.716 INFO FuncotatorDataSourceDownloader - ------------------------------------------------------------
14:51:31.718 INFO FuncotatorDataSourceDownloader - HTSJDK Version: 4.1.3
14:51:31.721 INFO FuncotatorDataSourceDownloader - Picard Version: 3.3.0
14:51:31.723 INFO FuncotatorDataSourceDownloader - Built for Spark Version: 3.5.0
14:51:31.733 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.COMPRESSION_LEVEL : 2
14:51:31.734 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
14:51:31.734 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
14:51:31.735 INFO FuncotatorDataSourceDownloader - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
14:51:31.735 INFO FuncotatorDataSourceDownloader - Deflater: IntelDeflater
14:51:31.735 INFO FuncotatorDataSourceDownloader - Inflater: IntelInflater
14:51:31.736 INFO FuncotatorDataSourceDownloader - GCS max retries/reopens: 20
14:51:31.736 INFO FuncotatorDataSourceDownloader - Requester pays: disabled
14:51:31.737 INFO FuncotatorDataSourceDownloader - Initializing engine
14:51:31.737 INFO FuncotatorDataSourceDownloader - Done initializing engine
14:51:31.737 INFO FuncotatorDataSourceDownloader - HG38_Somatic data sources selected.
14:51:31.757 INFO FuncotatorDataSourceDownloader - Collecting expected checksum...
14:51:33.131 INFO FuncotatorDataSourceDownloader - Shutting down engine
[February 22, 2025 at 2:51:33 PM GMT] org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader done. Elapsed time: 0.03 minutes.
Runtime.totalMemory()=285212672
code: 400
message: Bucket is a requester pays bucket but no user project provided.
reason: required
location: null
retryable: false
com.google.cloud.storage.StorageException: Bucket is a requester pays bucket but no user project provided.
at com.google.cloud.storage.StorageException.translate(StorageException.java:170)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:329)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:554)
at com.google.cloud.storage.StorageImpl.lambda$internalGetBlob$63(StorageImpl.java:1585)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:65)
at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1524)
at com.google.cloud.storage.StorageImpl.internalGetBlob(StorageImpl.java:1583)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:328)
at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.fetchSize(CloudStorageReadChannel.java:231)
at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.<init>(CloudStorageReadChannel.java:110)
at com.google.cloud.storage.contrib.nio.CloudStorageReadChannel.create(CloudStorageReadChannel.java:90)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newReadChannel(CloudStorageFileSystemProvider.java:401)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newByteChannel(CloudStorageFileSystemProvider.java:316)
at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newInputStream(CloudStorageFileSystemProvider.java:493)
at java.base/java.nio.file.Files.newInputStream(Files.java:160)
at java.base/java.nio.file.Files.newBufferedReader(Files.java:2922)
at java.base/java.nio.file.Files.lines(Files.java:4141)
at java.base/java.nio.file.Files.lines(Files.java:4227)
at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.readSha256SumFromPath(FuncotatorDataSourceDownloader.java:322)
at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.downloadDataSources(FuncotatorDataSourceDownloader.java:297)
at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.downloadAndValidateDataSources(FuncotatorDataSourceDownloader.java:275)
at org.broadinstitute.hellbender.tools.funcotator.FuncotatorDataSourceDownloader.doWork(FuncotatorDataSourceDownloader.java:248)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:150)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:203)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:222)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:166)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:209)
at org.broadinstitute.hellbender.Main.main(Main.java:306)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
GET https://storage.googleapis.com/storage/v1/b/broad-public-datasets/o/funcotator%2Ffuncotator_dataSources.v1.8.hg38.20230908s.sha256?fields=bucket,name,generation,size&projection=full
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Bucket is a requester pays bucket but no user project provided.",
"reason" : "required"
} ],
"message" : "Bucket is a requester pays bucket but no user project provided."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:551)
... 31 more
(gatk) root@3996f55bb7a7:/gatk# ./gatk --version
Using GATK jar /gatk/gatk-package-4.6.1.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gatk/gatk-package-4.6.1.0-local.jar --version
The Genome Analysis Toolkit (GATK) v4.6.1.0
HTSJDK Version: 4.1.3
Picard Version: 3.3.0
-
Looks like there is a problem with the bucket we use to store that data. We are working on a fix.
We are sorry for any inconvenience.
Regards.
-
Thanks!
-
Hi again.
Funcotator bucket is now fixed. You should be able to download datasources again.
Please sign in to leave a comment.
3 comments