Error import gcnvkernel during GATK-SV cohort mode on local HPC

Follow

Le Qi

• August 14, 2023 18:56

REQUIRED for all errors and issues:

I understand that the GATK-SV cromwell pipeline is meant for Terra, but I'm trying to implement it on local HPCs. I'm at the trainGCNV step and there's an error for the DetermineGermlineContigPloidyCohortMode task. The full error message is posted below for this step. I can reproduce the error if I try the following:

singularity shell docker://us.gcr.io/broad-dsde-methods/markw/gatk:2023-07-13-4.4.0.0-43-gd79823f9c-NIGHTLY-SNAPSHOT

python

import gcnvkernel

I tried a few different gatk dockers and they all show the same message. 

a) GATK version used: 4.4.0.0 from docker://us.gcr.io/broad-dsde-methods/markw/gatk:2023-07-13-4.4.0.0-43-gd79823f9c-NIGHTLY-SNAPSHOT

cromwell version: 85
b) Exact command used: 

java -Dconfig.file=$scripts_dir/singularity.conf -jar $cromwell run $scripts_dir/wdl/TrainGCNV.wdl -i $scripts_dir/trainGCNV.json

c) Entire program log:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell-executions/TrainGCNV/58896af2-52be-4fc8-af6f-f4df624ddf4e/call-CNVGermlineCohortWorkflow/CNVGermlineCohortWorkflow/eb2c9a7a-94de-4cf3-974f-c0752d86dd5a/call-DetermineGermlineContigPloidyCohortMode/attempt-2/tmp.ddf8c7c3

00:58:19.172 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.4.0.0-43-gd79823f-SNAPSHOT-local.jar!/com/intel/gkl/native/libgkl_compression.so

00:58:19.384 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /cromwell-executions/TrainGCNV/58896af2-52be-4fc8-af6f-f4df624ddf4e/call-CNVGermlineCohortWorkflow/CNVGermlineCohortWorkflow/eb2c9a7a-94de-4cf3-974f-c0752d86dd5a/call-DetermineGermlineContigPloidyCohortMode/attempt-2/tmp.ddf8c7c3/libgkl_compression9156501393605898667.so

00:58:19.396 INFO  DetermineGermlineContigPloidy - ------------------------------------------------------------

00:58:19.396 INFO  DetermineGermlineContigPloidy - The Genome Analysis Toolkit (GATK) v4.4.0.0-43-gd79823f-SNAPSHOT

00:58:19.396 INFO  DetermineGermlineContigPloidy - For support and documentation go to https://software.broadinstitute.org/gatk/

00:58:19.397 INFO  DetermineGermlineContigPloidy - Executing as leqi0001@n6055 on Linux v3.10.0-1160.80.1.el7.x86_64 amd64

00:58:19.397 INFO  DetermineGermlineContigPloidy - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10-Ubuntu-0ubuntu118.04.1

00:58:19.397 INFO  DetermineGermlineContigPloidy - Start Date/Time: August 14, 2023 at 12:58:18 AM GMT

00:58:19.397 INFO  DetermineGermlineContigPloidy - ------------------------------------------------------------

00:58:19.398 INFO  DetermineGermlineContigPloidy - ------------------------------------------------------------

00:58:19.399 INFO  DetermineGermlineContigPloidy - HTSJDK Version: 3.0.5

00:58:19.399 INFO  DetermineGermlineContigPloidy - Picard Version: 3.0.0

00:58:19.399 INFO  DetermineGermlineContigPloidy - Built for Spark Version: 3.3.1

00:58:19.400 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.BUFFER_SIZE : 131072

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.COMPRESSION_LEVEL : 2

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.CREATE_INDEX : false

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.CREATE_MD5 : false

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.CUSTOM_READER_FACTORY : 

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.REFERENCE_FASTA : null

00:58:19.401 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL

00:58:19.402 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false

00:58:19.402 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true

00:58:19.403 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false

00:58:19.403 INFO  DetermineGermlineContigPloidy - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false

00:58:19.403 DEBUG ConfigFactory - Configuration file values: 

00:58:19.408 DEBUG ConfigFactory - gcsMaxRetries = 20

00:58:19.408 DEBUG ConfigFactory - gcsProjectForRequesterPays = 

00:58:19.408 DEBUG ConfigFactory - gatk_stacktrace_on_user_exception = false

00:58:19.409 DEBUG ConfigFactory - samjdk.use_async_io_read_samtools = false

00:58:19.409 DEBUG ConfigFactory - samjdk.use_async_io_write_samtools = true

00:58:19.409 DEBUG ConfigFactory - samjdk.use_async_io_write_tribble = false

00:58:19.409 DEBUG ConfigFactory - samjdk.compression_level = 2

00:58:19.409 DEBUG ConfigFactory - spark.kryoserializer.buffer.max = 512m

00:58:19.409 DEBUG ConfigFactory - spark.driver.maxResultSize = 0

00:58:19.409 DEBUG ConfigFactory - spark.driver.userClassPathFirst = true

00:58:19.409 DEBUG ConfigFactory - spark.io.compression.codec = lzf

00:58:19.409 DEBUG ConfigFactory - spark.executor.memoryOverhead = 600

00:58:19.409 DEBUG ConfigFactory - spark.driver.extraJavaOptions = 

00:58:19.409 DEBUG ConfigFactory - spark.executor.extraJavaOptions = 

00:58:19.410 DEBUG ConfigFactory - codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]

00:58:19.410 DEBUG ConfigFactory - read_filter_packages = [org.broadinstitute.hellbender.engine.filters]

00:58:19.410 DEBUG ConfigFactory - annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]

00:58:19.410 DEBUG ConfigFactory - cloudPrefetchBuffer = 40

00:58:19.410 DEBUG ConfigFactory - cloudIndexPrefetchBuffer = -1

00:58:19.410 DEBUG ConfigFactory - createOutputBamIndex = true

00:58:19.410 INFO  DetermineGermlineContigPloidy - Deflater: IntelDeflater

00:58:19.410 INFO  DetermineGermlineContigPloidy - Inflater: IntelInflater

00:58:19.411 INFO  DetermineGermlineContigPloidy - GCS max retries/reopens: 20

00:58:19.411 INFO  DetermineGermlineContigPloidy - Requester pays: disabled

00:58:19.411 INFO  DetermineGermlineContigPloidy - Initializing engine

00:58:19.417 DEBUG ScriptExecutor - Executing:

00:58:19.417 DEBUG ScriptExecutor -   python

00:58:19.417 DEBUG ScriptExecutor -   -c

00:58:19.418 DEBUG ScriptExecutor -   import gcnvkernel

 

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/__init__.py", line 11, in <module>

    from .postprocess import test_viterbiSegmentationEngine

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/test_viterbiSegmentationEngine.py", line 5, in <module>

    from gcnvkernel.postprocess.viterbi_segmentation import ViterbiSegmentationEngine

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/viterbi_segmentation.py", line 11, in <module>

    from .segment_quality_utils import HMMSegmentationQualityCalculator

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/segment_quality_utils.py", line 15, in <module>

    class HMMSegmentationQualityCalculator:

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/segment_quality_utils.py", line 129, in HMMSegmentationQualityCalculator

    _constrained_path_logp_theano_func = _get_compiled_constrained_path_logp_theano_func.__func__()

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configparser.py", line 117, in res

    return f(*args, **kwargs)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/postprocess/segment_quality_utils.py", line 126, in _get_compiled_constrained_path_logp_theano_func

    return th.function(inputs=inputs, outputs=output)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function.py", line 317, in function

    output_keys=output_keys)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc

    output_keys=output_keys)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 1841, in orig_function

    fn = m.create(defaults)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 1715, in create

    input_storage=input_storage_lists, storage_map=storage_map)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk

    storage_map=storage_map)[:3]

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/vm.py", line 1091, in make_all

    impl=impl))

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 866, in make_thunk

    on_unused_input='ignore')

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function.py", line 317, in function

    output_keys=output_keys)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc

    output_keys=output_keys)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 1841, in orig_function

    fn = m.create(defaults)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/function_module.py", line 1715, in create

    input_storage=input_storage_lists, storage_map=storage_map)

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk

    storage_map=storage_map)[:3]

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/vm.py", line 1142, in make_all

    self.updated_vars,

  File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/vm.py", line 992, in make_vm

    vm = CVM(

NameError: ('The following error happened while compiling the node', forall_inplace,cpu,scan_fn}(Elemwise{Maximum}[(0, 0)].0, Subtensor{int64:int64:int8}.0, Subtensor{int64:int64:int8}.0, IncSubtensor{InplaceSet;:int64:}.0), '\n', "name 'CVM' is not defined")

00:58:37.404 DEBUG ScriptExecutor - Result: 1

00:58:37.405 INFO  DetermineGermlineContigPloidy - Shutting down engine

[August 14, 2023 at 12:58:37 AM GMT] org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy done. Elapsed time: 0.31 minutes.

Runtime.totalMemory()=125829120

java.lang.RuntimeException: A required Python package ("gcnvkernel") could not be imported into the Python environment. This tool requires that the GATK Python environment is properly established and activated. Please refer to GATK README.md file for instructions on setting up the GATK Python environment.

at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.checkPythonEnvironmentForPackage(PythonScriptExecutor.java:228)

at org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy.onStartup(DetermineGermlineContigPloidy.java:299)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:147)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)

at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)

at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:166)

at org.broadinstitute.hellbender.Main.mainEntry(Main.java:209)

at org.broadinstitute.hellbender.Main.main(Main.java:306)

Caused by: org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException: 

python exited with 1

Command Line: python -c import gcnvkernel

 

at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)

at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:112)

at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:193)

at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeCommand(PythonScriptExecutor.java:78)

at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.checkPythonEnvironmentForPackage(PythonScriptExecutor.java:221)

... 7 more

Using GATK jar /root/gatk.jar defined in environment variable GATK_LOCAL_JAR

Running:

    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx5100m -jar /root/gatk.jar DetermineGermlineContigPloidy -L /cromwell-executions/TrainGCNV/58896af2-52be-4fc8-af6f-f4df624ddf4e/call-CNVGermlineCohortWorkflow/CNVGermlineCohortWorkflow/eb2c9a7a-94de-4cf3-974f-c0752d86dd5a/call-DetermineGermlineContigPloidyCohortMode/attempt-2/inputs/503995868/condensed_intervals.filtered.interval_list --arguments_file read_count_files.args --contig-ploidy-priors /cromwell-executions/TrainGCNV/58896af2-52be-4fc8-af6f-f4df624ddf4e/call-CNVGermlineCohortWorkflow/CNVGermlineCohortWorkflow/eb2c9a7a-94de-4cf3-974f-c0752d86dd5a/call-DetermineGermlineContigPloidyCohortMode/attempt-2/inputs/1625757417/hg38.contig_ploidy_priors_homo_sapiens.tsv --interval-merging-rule OVERLAPPING_ONLY --output out --output-prefix full_batch --verbosity DEBUG --mean-bias-standard-deviation 1.0 --mapping-error-rate 0.01 --global-psi-scale 0.05 --sample-psi-scale 0.001

 

 

 

 

0

• 

  Gökalp Çelik

  • August 15, 2023 17:08
  • Edited

  Hi Le Qi

  Docker nightly images may cause certain issues and they are not recommended to be used unless there is a very specific need or an advice from the developer team. Can you try running your workflow using the image from hub.docker.com ?

  docker pull broadinstitute/gatk:4.4.0.0

  This is the official stable image that is supported currently. 

  Regards. 

  0

  Comment actions Permalink
• 

  Le Qi

  • August 15, 2023 17:54

  Hi Gökalp Çelik

  Thanks for your suggestions. I've tried to import gcnvkernel after running this version of gatk docker and it still shows the same error. Maybe I'm missing something else and I'm running the whole trainGCNV workflow using the stable version of gatk.

  0

  Comment actions Permalink
• 

  Gökalp Çelik

  • August 15, 2023 19:22

  Hi Le Qi

  You seem to hit the problem of not having access to the temporary folder in the docker container which is only accessible by the root account. 

  Can you modify your Singularity configuration or whatever you are using to run containers to have the environment variable 

  THEANO_FLAGS="base_compiledir=/path/to/temporary/folder"

   

  to point to a temporary folder where your user has access and read, write and execute permissions?

   

  Regards. 
  0

  Comment actions Permalink
• 

  Le Qi

  • August 15, 2023 19:52

  Hi Gökalp Çelik

  I tried the new environmental variable. I have the same errors and I can see the cache files generated in the specified path when I just try to import gcnvkernel. I had similar cache files generated in previous runs as well in $HOME. 

  Running the whole workflow with the stable docker gave me the same error again.

  0

  Comment actions Permalink
• 

  Gökalp Çelik

  • August 17, 2023 16:04
  • Edited

  Hi Le Qi

  By the current status of the workflow GATK-SV workflow is not supported on any other platform but GCP. So we may not be able to solve all your issues related to this workflow specifically on your platform. Still your container instance shows a username other than the root user therefore not only THEANO temporary folder but also gatk temporary folder access might be causing additional issues. Can you share your trainGCNV.wdl file may be we can pinpoint some additional spots for errors?

  One last suggestion. It was indicated in our github issues pages that singularity instances do not automatically initialize the gatk conda environment as in docker runtime. Can you try

  conda activate gatk 

  command before python -c "import gcnvkernel" ? Of course pay attention to the THEANO_FLAGS for proper compilation of temporary files. 

  0

  Comment actions Permalink
• 

  Le Qi

  • August 17, 2023 19:19

  Hi Gökalp Çelik

  I managed to activate gatk conda environment and set THEANO_FLAGS environmental variable, but import gcnvkernel still showed the same error. I've successfully adopted the previous 2 steps (GatherSampleEvidenceBatch and EvidenceQC) on my platform, therefore I'm guessing accessing GATK temporary folder is probably not the problem.

  Let me try a small dataset on Terra and get back to you.

  0

  Comment actions Permalink
• 

  Le Qi

  • August 18, 2023 23:46

  Hi Gökalp Çelik

  The workflow works fine on Terra and on my own MacOS with docker, which is not allowed on our HPC. I think the problem is with singularity somehow. I managed to get around the problem using udocker. 

   

  0

  Comment actions Permalink
• 

  Gökalp Çelik

  • August 19, 2023 14:10

  Hi Le Qi

  It is good to hear that you have found the solution already. Normally GATK docker images are not tested against singularity runtime therefore the team cannot approve of its use instead of the official docker runtime. However some users have successful results with some of the tools but not every tool is supposed to run the same. 

  Thank you for sharing your solution with everyone. 

  0

  Comment actions Permalink

Please sign in to leave a comment.