CNNScoreVariants error:AttributeError: module 'keras.backend' has no attribute 'clear_session'
REQUIRED for all errors and issues:
a) GATK version used:
b) Exact command used:
c) Entire program log:
gatk=/mnt/sdc/tanght/wes_cancer_2023711/biosoft/gatk-4.4.0.0/gatk
ref=/mnt/sdc/tanght/human_reference_data/Homo_sapiens_assembly38.fasta
$gatk CNNScoreVariants \
-V $1.vcf.gz \
-R $ref \
-O $1.annotated.vcf
c) Entire program log:
Using GATK jar /mnt/sdc/tanght/wes_cancer_2023711/biosoft/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /mnt/sdc/tanght/wes_cancer_2023711/biosoft/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar CNNScoreVariants -V Normal1.vcf.gz -R /mnt/sdc/tanght/human_reference_data/Homo_sapiens_assembly38.fasta -O Normal1.annotated.vcf
13:49:54.647 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/sdc/tanght/wes_cancer_2023711/biosoft/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
13:49:54.680 INFO CNNScoreVariants - ------------------------------------------------------------
13:49:54.683 INFO CNNScoreVariants - The Genome Analysis Toolkit (GATK) v4.4.0.0
13:49:54.683 INFO CNNScoreVariants - For support and documentation go to https://software.broadinstitute.org/gatk/
13:49:54.683 INFO CNNScoreVariants - Executing as tht@ecs-d32a on Linux v5.4.0-100-generic amd64
13:49:54.683 INFO CNNScoreVariants - Java runtime: Java HotSpot(TM) 64-Bit Server VM v20.0.1+9-29
13:49:54.683 INFO CNNScoreVariants - Start Date/Time: February 17, 2024, 1:49:54 PM CST
13:49:54.683 INFO CNNScoreVariants - ------------------------------------------------------------
13:49:54.683 INFO CNNScoreVariants - ------------------------------------------------------------
13:49:54.684 INFO CNNScoreVariants - HTSJDK Version: 3.0.5
13:49:54.684 INFO CNNScoreVariants - Picard Version: 3.0.0
13:49:54.684 INFO CNNScoreVariants - Built for Spark Version: 3.3.1
13:49:54.685 INFO CNNScoreVariants - HTSJDK Defaults.COMPRESSION_LEVEL : 2
13:49:54.685 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
13:49:54.685 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
13:49:54.685 INFO CNNScoreVariants - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
13:49:54.685 INFO CNNScoreVariants - Deflater: IntelDeflater
13:49:54.685 INFO CNNScoreVariants - Inflater: IntelInflater
13:49:54.685 INFO CNNScoreVariants - GCS max retries/reopens: 20
13:49:54.685 INFO CNNScoreVariants - Requester pays: disabled
13:49:54.686 INFO CNNScoreVariants - Initializing engine
13:49:54.893 INFO FeatureManager - Using codec VCFCodec to read file file:///mnt/sdc/tanght/ad_sc_compare_study/5.gatk/Normal1.vcf.gz
13:49:54.969 INFO CNNScoreVariants - Done initializing engine
13:49:54.970 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/mnt/sdc/tanght/wes_cancer_2023711/biosoft/gatk-4.4.0.0/gatk-package-4.4.0.0-local.jar!/com/intel/gkl/native/libgkl_utils.so
13:49:56.223 INFO CNNScoreVariants - Using key:CNN_1D for CNN architecture:/tmp/1d_cnn_mix_train_full_bn.10711659785906526281.json and weights:/tmp/1d_cnn_mix_train_full_bn.12532684413802336162.hd5
13:49:56.224 INFO CNNScoreVariants - Done scoring variants with CNN.
13:49:56.224 INFO CNNScoreVariants - Shutting down engine
[February 17, 2024, 1:49:56?PM CST] org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants done. Elapsed time: 0.03 minutes.
Runtime.totalMemory()=629145600
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException: A nack was received from the Python process (most likely caused by a raised exception caused by): nkm received
: Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/tht/miniconda3/envs/gatk/lib/python3.6/site-packages/vqsr_cnn/vqsr_cnn/models.py", line 21, in start_session_get_args_and_model
K.clear_session()
AttributeError: module 'keras.backend' has no attribute 'clear_session'
at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.waitForAck(StreamingPythonScriptExecutor.java:222)
at org.broadinstitute.hellbender.utils.python.StreamingPythonScriptExecutor.sendSynchronousCommand(StreamingPythonScriptExecutor.java:183)
at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.initializePythonArgsAndModel(CNNScoreVariants.java:559)
at org.broadinstitute.hellbender.tools.walkers.vqsr.CNNScoreVariants.onTraversalStart(CNNScoreVariants.java:319)
at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1096)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
(gatk)
I bulit a conda environment named gatk according to file named "gatkcondaenv.yml
".
However this issue only happens when I use cloud server.I bulid same conda environment in my wsl ubuntu and it works with no errors.
-
Hi Tang Huatao
The issue you are observing is most likely due to conda environment installation. Either it is not installed properly or conda environment contains an incompatible version of the required libraries. Can you post the conda list command within the conda gatk environment?
It is also possible that the conda installation might be outdated on the compute environment therefore it pulls wrong package versions.
Another quick reminder about the tool that you are using. GATK team is about to retire CNNScoreVariants tools due to its dependency on old python libraries. A new tool is in the works but not ready for primetime. We cannot provide an exact timeframe as to when it will be ready but we suggest our users not to depend on CNNScoreVariants tool too tightly.
Regards.
-
Thank you for your reply!
Here is the output of conda list command within the conda gatk environment:
$ conda list
# packages in environment at /home/tht/miniconda3/envs/gatk:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
_r-mutex 1.0.1 anacondar_1 conda-forge
_tflow_select 2.3.0 mkl
absl-py 0.15.0 pyhd8ed1ab_0 conda-forge
astor 0.8.1 pyh9f0ad1d_0 conda-forge
bcftools 1.12 h45bccc9_1 bioconda
binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge
binutils_linux-64 2.36 hf3e587d_33 conda-forge
biopython 1.76 py36h516909a_0 conda-forge
blas 1.1 openblas conda-forge
bwidget 1.9.14 ha770c72_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.26.0 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
cairo 1.16.0 hcf35c78_1003 conda-forge
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
curl 7.26.0 1 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/free
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 hfdff14a_1 conda-forge
dill 0.3.4 pyhd8ed1ab_0 conda-forge
expat 2.5.0 hcb278e6_1 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
gast 0.2.2 py_0 conda-forge
gcc_impl_linux-64 7.5.0 habd7529_20 conda-forge
gcc_linux-64 7.5.0 h47867f9_33 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gfortran_impl_linux-64 7.5.0 h56cb351_20 conda-forge
gfortran_linux-64 7.5.0 h78c8a43_33 conda-forge
glib 2.66.3 h58526e2_0 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
grpcio 1.38.1 py36h8e87921_0 conda-forge
gsl 2.6 he838d99_2 conda-forge
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
gxx_impl_linux-64 7.5.0 hd0bb8aa_20 conda-forge
gxx_linux-64 7.5.0 h555fc39_33 conda-forge
h5py 2.10.0 nompi_py36h4510012_106 conda-forge
harfbuzz 2.4.0 h9f30f68_3 conda-forge
hdf5 1.10.6 nompi_h3c11f04_101 conda-forge
htslib 1.12 h9093b5e_1 bioconda
icu 64.2 he1b5a44_1 conda-forge
importlib-metadata 4.8.1 py36h5fab9bb_0 conda-forge
intel-openmp 2019.4 243
joblib 1.1.1 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keras 2.2.4 py36_1 conda-forge
keras-applications 1.0.8 py_1 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_17 conda-forge
kiwisolver 1.3.1 py36h605e78d_1 conda-forge
krb5 1.17.1 h173b8e3_0 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
libblas 3.9.0 13_linux64_openblas conda-forge
libcblas 3.9.0 13_linux64_openblas conda-forge
libcurl 7.76.1 hc4aaa36_1 conda-forge
libdeflate 1.17 h5eee18b_1 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h7f8727e_1 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc 7.2.0 h69d50b8_2 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
libgcc-devel_linux-64 7.5.0 hda03d7c_20 conda-forge
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgfortran 3.0.0 1 conda-forge
libgfortran-ng 7.5.0 h14aa051_20 conda-forge
libgfortran4 7.5.0 h14aa051_20 conda-forge
libglib 2.66.3 hbe7bbb4_0 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
libgpuarray 0.7.6 h7f98852_1003 conda-forge
libiconv 1.17 hd590300_2 conda-forge
liblapack 3.9.0 13_linux64_openblas conda-forge
libnghttp2 1.52.0 ha637b67_1 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
libnsl 2.0.0 h5eee18b_0 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
libopenblas 0.3.18 hf726d26_0
libpng 1.6.42 h2797004_0 conda-forge
libprotobuf 3.18.0 h780b84a_1 conda-forge
libsqlite 3.45.1 h2797004_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx-devel_linux-64 7.5.0 hb016644_20 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libtiff 4.2.0 hf544144_3 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.2 hd590300_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
llvm-openmp 17.0.6 h4dfa4b3_0 conda-forge
make 4.3 hd18ef5c_1 conda-forge
mako 1.3.2 pyhd8ed1ab_0 conda-forge
markdown 3.5.2 pyhd8ed1ab_0 conda-forge
markupsafe 2.0.1 py36h8f6f2f9_0 conda-forge
matplotlib 3.2.1 0 conda-forge
matplotlib-base 3.2.1 py36hb8e4980_0 conda-forge
mkl 2019.5 281 conda-forge
mkl-service 2.3.0 py36h516909a_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
numpy 1.17.5 py36h2aa4a07_1 conda-forge
openblas 0.2.20 8 conda-forge
openssl 1.1.1w hd590300_0 conda-forge
opt_einsum 3.3.0 pyhc1e730c_2 conda-forge
pandas 1.0.3 py36h830a2c2_1 conda-forge
pango 1.42.4 h7062337_4 conda-forge
patsy 0.5.6 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.35 h14c3975_1 https://mirrors.tun a.tsinghua.edu.cn/anaconda/pkgs/main
perl 5.32.1 0_h5eee18b_perl5 https://mirrors.tu na.tsinghua.edu.cn/anaconda/pkgs/main
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.38.0 h516909a_1003 conda-forge
protobuf 3.18.0 py36hc4f0c31_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pygpu 0.7.6 py36h92226af_1002 conda-forge
pymc3 3.1 py36_0 conda-forge
pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge
pyqt 5.9.2 py36hcca6a23_4 conda-forge
pysam 0.11.2.2 py36_1 bioconda
python 3.6.10 h8356626_1011_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.6 2_cp36m conda-forge
pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge
pyvcf 0.6.8 py36h9f0ad1d_1002 conda-forge
pyyaml 5.4.1 py36h8f6f2f9_1 conda-forge
qt 5.9.7 h0c104cb_3 conda-forge
r-assertthat 0.2.1 r40hc72bb7e_2 conda-forge
r-backports 1.4.1 r40hcfec24a_0 conda-forge
r-base 4.0.2 h95c6c4b_0 conda-forge
r-bitops 1.0_7 r40h06615bd_0 conda-forge
r-brio 1.1.3 r40hcfec24a_0 conda-forge
r-callr 3.7.2 r40hc72bb7e_0 conda-forge
r-catools 1.18.2 r40h7525677_0 conda-forge
r-cli 3.4.1 r40h7525677_0 conda-forge
r-colorspace 2.0_3 r40h06615bd_0 conda-forge
r-crayon 1.5.1 r40hc72bb7e_0 conda-forge
r-data.table 1.14.2 r40hcfec24a_0 conda-forge
r-desc 1.4.2 r40hc72bb7e_0 conda-forge
r-diffobj 0.3.5 r40hcfec24a_0 conda-forge
r-digest 0.6.29 r40h03ef668_0 conda-forge
r-dplyr 1.0.10 r40h7525677_0 conda-forge
r-ellipsis 0.3.2 r40hcfec24a_0 conda-forge
r-evaluate 0.16 r40hc72bb7e_0 conda-forge
r-fansi 1.0.3 r40h06615bd_0 conda-forge
r-farver 2.1.1 r40h7525677_0 conda-forge
r-fs 1.5.2 r40h7525677_1 conda-forge
r-gdata 2.18.0.1 r40hc72bb7e_0 conda-forge
r-generics 0.1.3 r40hc72bb7e_0 conda-forge
r-getopt 1.20.3 r40ha770c72_2 conda-forge
r-ggplot2 3.3.6 r40hc72bb7e_0 conda-forge
r-glue 1.6.2 r40h06615bd_0 conda-forge
r-gplots 3.1.3 r40hc72bb7e_0 conda-forge
r-gsalib 2.1 r40hc72bb7e_1003 conda-forge
r-gtable 0.3.1 r40hc72bb7e_0 conda-forge
r-gtools 3.9.3 r40h06615bd_0 conda-forge
r-isoband 0.2.5 r40h03ef668_0 conda-forge
r-jsonlite 1.8.0 r40h06615bd_0 conda-forge
r-kernsmooth 2.23_18 r40h7679c2e_0 conda-forge
r-labeling 0.4.2 r40hc72bb7e_1 conda-forge
r-lattice 0.20_45 r40hcfec24a_0 conda-forge
r-lifecycle 1.0.2 r40hc72bb7e_0 conda-forge
r-magrittr 2.0.3 r40h06615bd_0 conda-forge
r-mass 7.3_58.1 r40h06615bd_0 conda-forge
r-matrix 1.4_1 r40h0154571_0 conda-forge
r-mgcv 1.8_40 r40h0154571_0 conda-forge
r-munsell 0.5.0 r40hc72bb7e_1004 conda-forge
r-nlme 3.1_150 r40h31ca83e_0 conda-forge
r-optparse 1.7.3 r40hc72bb7e_0 conda-forge
r-pillar 1.8.1 r40hc72bb7e_0 conda-forge
r-pkgconfig 2.0.3 r40hc72bb7e_1 conda-forge
r-pkgload 1.3.0 r40hc72bb7e_0 conda-forge
r-plogr 0.2.0 r40hc72bb7e_1003 conda-forge
r-praise 1.0.0 r40hc72bb7e_1005 conda-forge
r-processx 3.7.0 r40h06615bd_0 conda-forge
r-ps 1.7.1 r40h06615bd_0 conda-forge
r-purrr 0.3.4 r40hcfec24a_1 conda-forge
r-r6 2.5.1 r40hc72bb7e_0 conda-forge
r-rcolorbrewer 1.1_3 r40h785f33e_0 conda-forge
r-rcpp 1.0.9 r40h7525677_1 conda-forge
r-rematch2 2.1.2 r40hc72bb7e_1 conda-forge
r-rlang 1.0.6 r40h7525677_0 conda-forge
r-rprojroot 2.0.3 r40hc72bb7e_0 conda-forge
r-rstudioapi 0.14 r40hc72bb7e_0 conda-forge
r-scales 1.2.1 r40hc72bb7e_0 conda-forge
r-testthat 3.1.4 r40h7525677_0 conda-forge
r-tibble 3.1.8 r40h06615bd_0 conda-forge
r-tidyselect 1.1.2 r40hc72bb7e_0 conda-forge
r-utf8 1.2.2 r40hcfec24a_0 conda-forge
r-vctrs 0.4.1 r40h7525677_0 conda-forge
r-viridislite 0.4.1 r40hc72bb7e_0 conda-forge
r-waldo 0.4.0 r40hc72bb7e_0 conda-forge
r-withr 2.5.0 r40hc72bb7e_0 conda-forge
readline 8.2 h8228510_1 conda-forge
samtools 1.5 1 bioconda
scikit-learn 0.23.1 py36h0e1014b_0 conda-forge
scipy 1.0.0 py36_blas_openblas_201 [blas_openblas ] conda-forge
sed 4.8 he412f7d_0 conda-forge
setuptools 58.0.4 py36h5fab9bb_2 conda-forge
sip 4.19.8 py36hf484d3e_1000 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.45.1 h2c6b66d_0 conda-forge
sysroot_linux-64 2.12 he073ed8_17 conda-forge
tensorboard 1.15.0 py36_0 conda-forge
tensorflow 1.15.0 mkl_py36h4920b83_0
tensorflow-base 1.15.0 mkl_py36he1670d9_0
tensorflow-estimator 1.15.1 pyh2649769_0
termcolor 1.1.0 pyhd8ed1ab_3 conda-forge
theano 1.0.4 py36h831f99a_1002 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tktable 2.10 h0c5db8f_5 conda-forge
tornado 6.1 py36h8f6f2f9_1 conda-forge
tqdm 4.65.0 pyhd8ed1ab_0 conda-forge
typing_extensions 4.1.1 pyha770c72_0 conda-forge
vcftools 0.1.16 pl5321hdcf5f25_9 bioconda
werkzeug 0.16.1 py_0 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
wrapt 1.13.1 py36h8f6f2f9_0 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.1.1 hd590300_0 conda-forge
xorg-libsm 1.2.4 h7391055_0 conda-forge
xorg-libx11 1.8.7 h8ee46fc_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.11 hd590300_0 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hfc55251_0 conda-forge
(gatk) -
Hi again.
Looking at this list and it looks like this environment is not %100 the same environment that we have in our docker image. It is possible that the environment was created using a prior version. Certain packages are not present that we already have in the docker gatk environment. Can you remove this gatk environment, update conda and reinstall the environment again? This may solve your issues.
Regards.
-
Hello!
I have tried removing gatk environment,updating conda and reinstalling the environment again and I'm sorry that it didn't work.
Can you please provide any other suggestions？
Thanks in advance.
Regards
-
Hi Tang Huatao
Can you setup a tmp dir where you have proper read write permissions on the cloud server? It is possible that python module compilation might not be running properly.
You can check the article below
-
Thank you!
I am really sorry that I totally forget about this.
I think you're right,conda can mess up things sometimes.But as I've learned how to use the docker rencently ,the problem no longer exists for me.
Thank you for your help again!
Please sign in to leave a comment.
6 comments