CNNScoreVariants ImportError: cannot import name 'Adam'
AnsweredI'm trying to run CNNScoreVariants and have set up the conda environment as described here:
There seems to be an issue with tensorflow, as the error appears to be related to importing 'Adam' from tensorflow/keras.
I am not sure if I need to load tensorflow prior to CNNScoreVariants, but if I do, it breaks the python environment.
Of note, I am on a HPC with no root privileges. The available version of tensorflow is 1.4.1, running on python 3.4, which is a different python version compared to the anaconda version I am using (python 3.8). Maybe this matters, maybe it doesn't.
REQUIRED for all errors and issues:
a) GATK version used:
GATK 4.1.4.0
b) Exact command used:
gatk CNNScoreVariants -V Farrell_30_Bulk_S2.g.vcf.gz -R ../../../../../../../../../work/InternalMedicine/s184335/genome_folder/alias/hg38/fasta/default/hg38.fa -O BamFiles/HaplotypeCaller_GVCF_output/ScoredVariants/test_scored.g.vcf
c) Entire program log:
Using GATK jar /cm/shared/apps/gatk/4.1.4.0/gatk-package-4.1.4.0-local.jar defined in environment variable GATK_LOCAL_JAR
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /cm/shared/apps/gatk/4.1.4.0/gatk-package-4.1.4.0-local.jar CNNScoreVariants -V Farrell_30_Bulk_S2.g.vcf.gz -R ../../../../../../../../../work/InternalMedicine/s184335/genome_folder/alias/hg38/fasta/default/hg38.fa -O BamFiles/HaplotypeCaller_GVCF_output/ScoredVariants/test_scored.g.vcf
15:56:59.052 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/cm/shared/apps/gatk/4.1.4.0/gatk-package-4.1.4.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Aug 17, 2022 3:56:59 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
15:56:59.326 INFO CNNScoreVariants - ------------------------------------------------------------
15:56:59.327 INFO CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.1.4.0
15:56:59.327 INFO CNNScoreVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
15:56:59.327 INFO CNNScoreVariants - Executing as s184335@Nucleus005 on Linux v3.10.0-957.el7.x86_64 amd64
15:56:59.327 INFO CNNScoreVariants - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-b08
15:56:59.328 INFO CNNScoreVariants - Start Date/Time: August 17, 2022 3:56:59 PM CDT
15:56:59.328 INFO CNNScoreVariants - ------------------------------------------------------------
15:56:59.328 INFO CNNScoreVariants - ------------------------------------------------------------
15:56:59.328 INFO CNNScoreVariants - HTSJDK Version: 2.20.3
15:56:59.328 INFO CNNScoreVariants - Picard Version: 2.21.1
15:56:59.329 INFO CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
15:56:59.329 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
15:56:59.329 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
15:56:59.329 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
15:56:59.329 INFO CNNScoreVariants - Deflater: IntelDeflater
15:56:59.329 INFO CNNScoreVariants - Inflater: IntelInflater
15:56:59.329 INFO CNNScoreVariants - GCS max retries/reopens: 20
15:56:59.329 INFO CNNScoreVariants - Requester pays: disabled
15:56:59.329 INFO CNNScoreVariants - Initializing engine
15:56:59.837 INFO FeatureManager - Using codec VCFCodec to read file file:///project/InternalMedicine/Chung_lab/shared/BenKroger/MDS_Relapse/June2022_run/BamFiles/HaplotypeCaller_GVCF_output/Farrell_30_Bulk_S2.g.vcf.gz
15:57:00.046 INFO CNNScoreVariants - Done initializing engine
15:57:00.047 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/cm/shared/apps/gatk/4.1.4.0/gatk-package-4.1.4.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
15:57:03.346 INFO CNNScoreVariants - Done scoring variants with CNN.
15:57:03.347 INFO CNNScoreVariants - Shutting down engine
[August 17, 2022 3:57:03 PM CDT] org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants done. Elapsed time: 0.07 minutes.
Runtime.totalMemory()=2082471936
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException: A nack was received from the Python process (most likely caused by a raised exception caused by): nkm received
: Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home2/s184335/.conda/envs/gatk/lib/python3.6/site-packages/vqsr_cnn/__init__.py", line 1, in <module>
from .vqsr_cnn.models import build_2d_annotation_model_from_args, build_1d_annotation_model_from_args
File "/home2/s184335/.conda/envs/gatk/lib/python3.6/site-packages/vqsr_cnn/vqsr_cnn/__init__.py", line 1, in <module>
from .models import build_2d_annotation_model_from_args, build_1d_annotation_model_from_args
File "/home2/s184335/.conda/envs/gatk/lib/python3.6/site-packages/vqsr_cnn/vqsr_cnn/models.py", line 8, in <module>
from keras.optimizers import Adam
ImportError: cannot import name 'Adam'
at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.waitForAck(StreamingPythonScriptExecutor.java:222)
at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.sendSynchronousCommand(StreamingPythonScriptExecutor.java:183)
at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.onTraversalStart(CNNScoreVariants.java:317)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1046)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
at org.broadinstitute.hellbender.Main.main(Main.java:292)
-
Hi albert son,
Could you upgrade your GATK version? There was a bug in CNNScoreVariants in this older version of CNNScoreVariants and it might be related to the issue you are seeing: https://gatk.broadinstitute.org/hc/en-us/articles/360052489832-Known-Issue-with-CNNScoreVariants-version-4-1-9-0
Let me know how that goes!
Best,
Genevieve
Please sign in to leave a comment.
1 comment