Cannot lock consolidation filelock; Cannot lock.
Hi ! When I use GenomicsDBImport and GenotypeGVCFs , I get the following error, I have no problem with running CombineGVCFs with CombineGVCFs, but CombineGVCFs is too slow.
a) GATK version used : 4.1.7.0
b) Exact GATK commands used : gatk GenotypeGVCFs -R path/hg38ncbi.fa -V gendb://mydatabase -O rawvariants.vcf
c) The entire error log if applicable.
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home//miniconda3/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar GenotypeGVCFs -R /home//workdir/data_single_cell/sperm/hg38ncbi.fa -V gendb://mydatabase -O rawvariants.vcf
21:14:29.330 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/miniconda3/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
May 25, 2020 9:14:29 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
21:14:29.494 INFO GenotypeGVCFs - ------------------------------------------------------------
21:14:29.495 INFO GenotypeGVCFs - The Genome Analysis Toolkit (GATK) v4.1.7.0
21:14:29.495 INFO GenotypeGVCFs - For support and documentation go to https://software.broadinstitute.org/gatk/
21:14:29.495 INFO GenotypeGVCFs - Executing as lbjiang@mu01 on Linux v3.10.0-327.el7.x86_64 amd64
21:14:29.495 INFO GenotypeGVCFs - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
21:14:29.495 INFO GenotypeGVCFs - Start Date/Time: May 25, 2020 9:14:29 PM CST
21:14:29.495 INFO GenotypeGVCFs - ------------------------------------------------------------
21:14:29.495 INFO GenotypeGVCFs - ------------------------------------------------------------
21:14:29.496 INFO GenotypeGVCFs - HTSJDK Version: 2.21.2
21:14:29.496 INFO GenotypeGVCFs - Picard Version: 2.21.9
21:14:29.496 INFO GenotypeGVCFs - HTSJDK Defaults.COMPRESSION_LEVEL : 2
21:14:29.496 INFO GenotypeGVCFs - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
21:14:29.496 INFO GenotypeGVCFs - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
21:14:29.496 INFO GenotypeGVCFs - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
21:14:29.496 INFO GenotypeGVCFs - Deflater: IntelDeflater
21:14:29.496 INFO GenotypeGVCFs - Inflater: IntelInflater
21:14:29.496 INFO GenotypeGVCFs - GCS max retries/reopens: 20
21:14:29.496 INFO GenotypeGVCFs - Requester pays: disabled
21:14:29.496 INFO GenotypeGVCFs - Initializing engine
[TileDB::StorageManager] Error: Cannot lock consolidation filelock; Cannot lock.
21:14:30.336 INFO GenotypeGVCFs - Shutting down engine
[May 25, 2020 9:14:30 PM CST] org.broadinstitute.hellbender.tools.walkers.GenotypeGVCFs done. Elapsed time: 0.02 minutes.
Runtime.totalMemory()=1199570944
***********************************************************************
A USER ERROR has occurred: Couldn't create GenomicsDBFeatureReader
***********************************************************************
Set the system property GATK_STACKTRACE_ON_USER_EXCEPTION (--java-options '-DGATK_STACKTRACE_ON_USER_EXCEPTION=true') to print the stack trace.
and last GATK commands used:
gatk GenomicsDBImport \
-V sample1.g.vcf \
-V sample2.g.vcf \
-V sample3.g.vcf \
--genomicsdb-workspace-path mydatabase \
--intervals chr22
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/miniconda3/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar GenomicsDBImport -V SRR630496.erc.g.vcf -V SRR630498.erc.g.vcf --genomicsdb-workspace-path mydatabase -L chr22
21:21:17.641 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/miniconda3/share/gatk4-4.1.7.0-0/gatk-package-4.1.7.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
May 25, 2020 9:21:17 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
21:21:17.806 INFO GenomicsDBImport - ------------------------------------------------------------
21:21:17.807 INFO GenomicsDBImport - The Genome Analysis Toolkit (GATK) v4.1.7.0
21:21:17.807 INFO GenomicsDBImport - For support and documentation go to https://software.broadinstitute.org/gatk/
21:21:17.807 INFO GenomicsDBImport - Executing as lbjiang@mu01 on Linux v3.10.0-327.el7.x86_64 amd64
21:21:17.807 INFO GenomicsDBImport - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_192-b01
21:21:17.807 INFO GenomicsDBImport - Start Date/Time: May 25, 2020 9:21:17 PM CST
21:21:17.807 INFO GenomicsDBImport - ------------------------------------------------------------
21:21:17.808 INFO GenomicsDBImport - ------------------------------------------------------------
21:21:17.808 INFO GenomicsDBImport - HTSJDK Version: 2.21.2
21:21:17.808 INFO GenomicsDBImport - Picard Version: 2.21.9
21:21:17.808 INFO GenomicsDBImport - HTSJDK Defaults.COMPRESSION_LEVEL : 2
21:21:17.808 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
21:21:17.808 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
21:21:17.808 INFO GenomicsDBImport - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
21:21:17.808 INFO GenomicsDBImport - Deflater: IntelDeflater
21:21:17.808 INFO GenomicsDBImport - Inflater: IntelInflater
21:21:17.808 INFO GenomicsDBImport - GCS max retries/reopens: 20
21:21:17.809 INFO GenomicsDBImport - Requester pays: disabled
21:21:17.809 INFO GenomicsDBImport - Initializing engine
21:21:20.496 INFO IntervalArgumentCollection - Processing 50818468 bp from intervals
21:21:20.498 INFO GenomicsDBImport - Done initializing engine
21:21:20.805 INFO GenomicsDBImport - Vid Map JSON file will be written to /lustre/data_single_cell/sperm1/mydatabase/vidmap.json
21:21:20.805 INFO GenomicsDBImport - Callset Map JSON file will be written to /lustre/data_single_cell/sperm1/mydatabase/callset.json
21:21:20.805 INFO GenomicsDBImport - Complete VCF Header will be written to /lustre/data_single_cell/sperm1/mydatabase/vcfheader.vcf
21:21:20.805 INFO GenomicsDBImport - Importing to array - /lustre/data_single_cell/sperm1/mydatabase/genomicsdb_array
21:21:20.805 INFO ProgressMeter - Starting traversal
21:21:20.805 INFO ProgressMeter - Current Locus Elapsed Minutes Batches Processed Batches/Minute
21:21:22.001 INFO GenomicsDBImport - Importing batch 1 with 2 samples
21:22:18.128 INFO ProgressMeter - chr22:1 1.0 1 1.0
21:22:18.128 INFO GenomicsDBImport - Done importing batch 1/1
21:22:18.129 INFO ProgressMeter - chr22:1 1.0 1 1.0
21:22:18.129 INFO ProgressMeter - Traversal complete. Processed 1 total batches in 1.0 minutes.
21:22:18.129 INFO GenomicsDBImport - Import completed!
21:22:18.129 INFO GenomicsDBImport - Shutting down engine
[May 25, 2020 9:22:18 PM CST] org.broadinstitute.hellbender.tools.genomicsdb.GenomicsDBImport done. Elapsed time: 1.01 minutes.
Runtime.totalMemory()=1821900800
Tool returned:
true
-
This is the result of GenomicsDBImport
-[ 201] callset.json
-[4.0K] chr22$1$50818468
|-[4.0K] __3d7b0c29-9d8b-4748-a08a-c1e0a9136cf647572466251520_1590412882146
||-[133K] AD.tdb
||-[ 54K] AD_var.tdb
||-[3.2M] ALT.tdb
||-[ 77K] ALT_var.tdb
||-[ 85K] BaseQRankSum.tdb
||-[192K] __book_keeping.tdb.gz
||-[4.1M] __coords.tdb
||-[1011K] DP_FORMAT.tdb
||-[114K] DP.tdb
||-[3.5M] END.tdb
||-[108K] ExcessHet.tdb
||-[ 64K] FILTER.tdb
||-[ 0] FILTER_var.tdb
||-[1.1M] GQ.tdb
||-[2.9M] GT.tdb
||-[120K] GT_var.tdb
||-[ 64K] ID.tdb
||-[ 0] ID_var.tdb
||-[ 64K] InbreedingCoeff.tdb
||-[1.0M] MIN_DP.tdb
||-[128K] MLEAC.tdb
||-[ 32K] MLEAC_var.tdb
||-[128K] MLEAF.tdb
||-[ 36K] MLEAF_var.tdb
||-[ 82K] MQRankSum.tdb
||-[ 98K] PGT.tdb
||-[5.8K] PGT_var.tdb
||-[ 99K] PID.tdb
||-[ 15K] PID_var.tdb
||-[2.9M] PL.tdb
||-[3.5M] PL_var.tdb
||-[ 77K] PS.tdb
||-[143K] QUAL.tdb
||-[163K] RAW_MQandDP.tdb
||-[ 84K] ReadPosRankSum.tdb
||-[3.2M] REF.tdb
||-[655K] REF_var.tdb
||-[174K] SB.tdb
||-[ 0] __tiledb_fragment.tdb
|-[ 535] __array_schema.tdb
|-[4.0K] genomicsdb_meta_dir
| -[ 60] genomicsdb_meta_a793298a-95f2-475e-b10b-726695483e3a.json
-[ 0] __tiledb_workspace.tdb
-[ 33K] vcfheader.vcf
-[ 39K] vidmap.jsonThanks in advance.
-
Can you try setting this environment variable and see if that solves the issue?
export TILEDB_DISABLE_FILE_LOCKING=1
-
Hi,Bhanu! Is the command I entered like this:
TILEDB_DISABLE_FILE_LOCKING=1 gatk GenomicsDBImport -V SRR630877.erc.g.vcf -V SRR630496.erc.g.vcf \
--genomicsdb-workspace-path my_database --intervals chr22If yes, it will still produce the same error。
-
And I checked my environment variable, the parameter is displayed as ' declare -x TILEDB_DISABLE_FILE_LOCKING="1" '
-
Hi hjwang
Take a look at this thread, a similar issue was resolved here: https://github.com/broadinstitute/gatk/issues/6616
-
Hi hjwang
We think this is a NFS related issue. The workaround for this is to set env variable TILEDB_DISABLE_FILE_LOCKING=1 in the GenotypeGVCF step as well. This is something we are looking to fix in the future.
Please sign in to leave a comment.
6 comments