Hello,

I am attempting to run the CNNScoreVariants command on an HPC system. I followed the instructions to create a conda environment posted here: https://gatk.broadinstitute.org/hc/en-us/articles/360035889851--How-to-Install-and-use-Conda-for-GATK4.

I am using GATK version 4.4.0. Below is the exact command I used:

conda env create --prefix PATH/TO/conda_envs/GATK -f gatkcondaenv.yml

c) Entire program log:

Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - defaults::intel-openmp=2019.4
  - defaults::tensorflow=1.15.0

ATTEMPT TO FIX:

I was able to overcome the intel-openmp failure by adding anaconda to the channels list in the yml file and then changing -defaults to -anaconda::intel-openmp=2019.4.

I am unable to find an alternate channel that has tensorflow 1.15.0. Owing to the python version specified in the environment, I am unable to upgrade tensorflow. I have been able to install 1.10, but the analysis fails with the following:

[0m15:41:10.379 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/project/software/el9/apps/gatk/4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
15:41:10.493 INFO  CNNScoreVariants - ------------------------------------------------------------
15:41:10.498 INFO  CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.4.0.0
15:41:10.500 INFO  CNNScoreVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
15:41:10.502 INFO  CNNScoreVariants - Executing as stephen.harding@ceres20-compute-6.scinet.local on Linux v5.14.0-284.30.1.el9_2.x86_64 amd64
15:41:10.505 INFO  CNNScoreVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v17.0.6+9-LTS-190
15:41:10.507 INFO  CNNScoreVariants - Start Date/Time: October 26, 2023 at 3:41:10 PM CDT
15:41:10.509 INFO  CNNScoreVariants - ------------------------------------------------------------
15:41:10.511 INFO  CNNScoreVariants - ------------------------------------------------------------
15:41:10.516 INFO  CNNScoreVariants - HTSJDK Version: 3.0.5
15:41:10.518 INFO  CNNScoreVariants - Picard Version: 3.0.0
15:41:10.522 INFO  CNNScoreVariants - Built for Spark Version: 3.3.1
15:41:10.524 INFO  CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
15:41:10.526 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:41:10.530 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:41:10.532 INFO  CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:41:10.534 INFO  CNNScoreVariants - Deflater: IntelDeflater
15:41:10.537 INFO  CNNScoreVariants - Inflater: IntelInflater
15:41:10.539 INFO  CNNScoreVariants - GCS max retries/reopens: 20
15:41:10.541 INFO  CNNScoreVariants - Requester pays: disabled
15:41:10.544 INFO  CNNScoreVariants - Initializing engine
15:41:10.858 INFO  FeatureManager - Using codec VCFCodec to read file file:///project/mpm_fusarium_genomics/stephen/GATK_Test/Fpro10717_TrimmedPair_OutFiles/Fpro10717.g.vcf.gz
15:41:10.964 INFO  CNNScoreVariants - Done initializing engine
15:41:10.968 INFO  NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/project/software/el9/apps/gatk/4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
15:41:32.284 INFO  CNNScoreVariants - Using key:CNN_1D for CNN architecture:/tmp/1d_cnn_mix_train_full_bn.693313877541357885.json and weights:/tmp/1d_cnn_mix_train_full_bn.8688214035610729398.hd5
15:41:32.297 INFO  CNNScoreVariants - Done scoring variants with CNN.
15:41:32.300 INFO  CNNScoreVariants - Shutting down engine
[October 26, 2023 at 3:41:32 PM CDT] org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants done. Elapsed time: 0.37 minutes.
Runtime.totalMemory()=285212672
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException: A nack was received from the Python process (most likely caused by a raised exception caused by): nkm received

: Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/project/mpm_fusarium_genomics/stephen/conda_envs/GATK/lib/python3.6/site-packages/vqsr_cnn/vqsr_cnn/models.py", line 21, in start_session_get_args_and_model
    K.clear_session()
 AttributeError: module 'keras.backend' has no attribute 'clear_session'

        at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.waitForAck(StreamingPythonScriptExecutor.java:222)
        at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.sendSynchronousCommand(StreamingPythonScriptExecutor.java:183)
        at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.initializePythonArgsAndModel(CNNScoreVariants.java:559)
        at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.onTraversalStart(CNNScoreVariants.java:319)
        at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1096)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)
Using GATK jar /project/software/el9/apps/gatk/4.4.0.0/gatk-package-4.4.0.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /project/software/el9/apps/gatk/4.4.0.0/gatk-package-4.4.0.0-local.jar CNNScoreVariants -V Fpro10717_TrimmedPair_OutFiles/Fpro10717.g.vcf.gz -R F_proliferatum_62905_clean_sorted_AddChrInfo.fa -O Fpro10717_TrimmedPair_OutFiles/Fpro10717.annotated.vcf