Problem Installing GATK python environment - SOLUTION POSTED!
Featured
Hi
When gatk python environment is installed to a fresh installation I am getting an error regarding missing dependencies however it is not stated anywhere during installation.
18:51:11.774 DEBUG ScriptExecutor - -c
18:51:11.774 DEBUG ScriptExecutor - import gcnvkernel
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/gcnvkernel/__init__.py", line 1, in <module>
from pymc3 import __version__ as pymc3_version
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/pymc3/__init__.py", line 5, in <module>
from .distributions import *
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/__init__.py", line 1, in <module>
from . import timeseries
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 5, in <module>
from .continuous import get_tau_sd, Normal, Flat
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/continuous.py", line 12, in <module>
from scipy import stats
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/scipy/stats/__init__.py", line 345, in <module>
from .morestats import *
File "/home/exome/miniconda3/envs/gatk/lib/python3.6/site-packages/scipy/stats/morestats.py", line 12, in <module>
from numpy.testing.decorators import setastest
ModuleNotFoundError: No module named 'numpy.testing.decorators'
18:51:12.924 DEBUG ScriptExecutor - Result: 1
18:51:12.924 INFO DetermineGermlineContigPloidy - Shutting down engine
[February 5, 2020 at 6:51:12 PM EET] org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy done. Elapsed time: 0.02 minutes.
Runtime.totalMemory()=2109734912
java.lang.RuntimeException: A required Python package ("gcnvkernel") could not be imported into the Python environment. This tool requires that the GATK Python environment is properly established and activated. Please refer to GATK README.md file for instructions on setting up the GATK Python environment.
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.checkPythonEnvironmentForPackage(PythonScriptExecutor.java:205)
at org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy.onStartup(DetermineGermlineContigPloidy.java:280)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:137)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:163)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:206)
at org.broadinstitute.hellbender.Main.main(Main.java:292)
Caused by: org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
python exited with 1
Command Line: python -c import gcnvkernel
at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
at org.broadinstitute.hellbender.utils.runtime.ScriptExecutor.executeCuratedArgs(ScriptExecutor.java:126)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeArgs(PythonScriptExecutor.java:170)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.executeCommand(PythonScriptExecutor.java:79)
at org.broadinstitute.hellbender.utils.python.PythonScriptExecutor.checkPythonEnvironmentForPackage(PythonScriptExecutor.java:198)
... 7 more
numpy.testing.decorators is somehow causing an issue and I have no idea where to get that. This was not a problem in previous versions however now it is a problem.
Can someone elaborate on this.
Docker image is not my target now since I observed a different behavior compared to installed version and I am trying to replicate the problem.
GATK version 4.1.4.1
miniconda3 latest version
debian buster 10.2
-
Looks like I found the solution to this problem but this may only be a temporary solution so devs please check this requirement of gatk environment and merge a solution.
This gatk conda environment installation was performed on a clean anaconda/miniconda 3 instance. The main issue is that numpy developers have already warned everyone to not import from numpy.testing.decorators as it was a deprecated call since version 1.15.0. However fresh installs of numpy into gatk environment is causing repositories to load a newer version which loaded 1.17.4 to my personal computer on a manjaro system which was installed couple of months ago and installs 1.18.1 to my fresh debian install on the workstation.
Fortunately 1.17.4 still supports the deprecated import from numpy.testing.decorators however 1.18.1 does not which results in package not found error in scipy code.
The version installed on the docker gatk is 1.13.3 and uses the old import style.
If any one wants to install gatk environment properly they should make sure that numpy version stays below 1.15 although it looks like 1.17 may also do the job and avoid 1.18 install completely or
There may be a workaround in the problematic python code in scipy.
Here is my workaround
Convert this line in morestats.py in scipy
from numpy.testing.decorators import setastest
to this
from numpy.testing import dec
and also convert all instances of
@setastest(False)
to
@dec.setastest(False)
in the same file and save it.
Now try
python -c "import gcnvkernel"
This line should not throw any errors of unfound packages.
This trick was from
https://docs.scipy.org/doc/numpy/reference/generated/numpy.testing.decorators.setastest.html
They clearly stated that setastest import must be made this way from now on.
Regards.
-
Hi SkyWarrior
Thank you so much for posting this solution. Your posts are always a great resource for the entire community!
I will check with the dev team on this and post their response here.
-
Thanks Bhanu Gandham
I also checked forced installation of an earlier version of numpy such as 1.13.3 from conda environment but it does not solve problems. Instead it creates a new level of uncertainty among package dependencies and kills the whole environment irreversibly.
-
Hi SkyWarrior
Did you face this issue using the gatk conda environment straight out of the box? Or was this triggered after you changed the numpy/other library versions in the conda env?
-
This error jumps out of the box. I just tried to fix it with the ways I mentioned and the only solution that worked for me was modifying morestats.py after oob installation. This must be happening to more people. As a test your devs can try to install gatk environment to a fresh miniconda installation on a docker ubuntu instance. They will observe the same behavior even with gatk version 4.1.3.0.
I posted a short video about the problem and the solution.
-
Hi SkyWarrior thanks for the movie and posting the solution. I think the numpy issue has been addressed (in the next release) by pinning it to 1.13.3 in the gatk env (longer term we probably need to update our dependencies more widely). Do let us know if you see it after the upgrade. Thanks again.
-
Thanks. I will give it a shot once arrives.
-
Hi its me again and you will not like this I am sure.
Looks like keeping numpy 1.13.3 version intact is working when gatk 4.1.5.0 environment is installed fresh onto conda. I tried this under 3 instances all of them had 1.13.3. However out of 3 instances only Ubuntu vanilla 18.04 liked the way numpy 1.13.3 works without throwing any errors upon import. Other 2 instances one on fresh debian 10.2 and arch (manjaro) did not likte numpy 1.13.3 due to some GCC python lib incompatibilities. Debian uses GCC 8.X and manjaro uses GCC 9.X whereas ubuntu is still stuck with GCC 7.X release. The only way to fix this incompatibility was to force numpy update to 1.17.0 without disturbing scipy lib as happened above.
I am sure more people will have this problem as they want to upgrade their installations to 4.1.5.0 however this problem will get in the way.
The way to upgrade numpy to 1.17.0 is to execute
pip install --upgrade numpy==1.17.0
Then imports will not cause any issues with systems using latest GCC stable.
You may wonder why docker is not a solution for this and I will say docker instance of gatk does not load python environment properly when userid is set anything but root. And this causes file access problems after a proper execution.
Thanks for reading. I am sorry if I am bugging you too much on this matter.
-
Hi SkyWarrior,
You are not bugging us at all. In fact you are doing just the opposite, you are helping us with very useful information.
I will share this with the dev team and get back to you.
-
Hi @SkyWarrior Thanks for the updates. Have you had any success running the CNN/CNV tools using numpy 1.17 ? IIRC, I tried this previously (on Ubuntu) and the CNN tools failed and will require some code changes to work. Curious if you're seeing different results. Thanks!
-
Hi Chris Norman
My CNV pipeline has been running on numpy 1.17 (4.1.5.0) and 1.18 (4.1.3.0 to 4.1.4.1) for a while and my results are quite reproducible from one run to another but I have not had any experience with CNN. I am hesitant to move my workflow to CNN but I can give a try sometime next week with a couple of samples.
-
Hi SkyWarrior,
As always, thanks for your detailed report. I am working on cleaning up the conda environment at https://github.com/broadinstitute/gatk/pull/5026. I updated the GATK Docker image to Ubuntu 18.04 and this environment seems to work, although we may need to do some more detailed tests and numerical checks to confirm this.
However, I am not sure whether the conda environment will work across other architectures and distributions. Although we cannot really test all environments, I'd like to understand more about your specific use case. If the Docker was updated to add a non-root user, would you be able to (or even prefer to) use that to guarantee the correct GATK conda environment? Or do you need to be able to run under all of the various distributions you mentioned?
-
Hi Samuel Lee
I agree that you cannot test each and every single scenario. I am able to solve my problems temporarily but I opt for the proper docker image where anyone can use -u $UID:$GID in the docker commandline if possible. This way monitoring gatk from HTOP or ps is posslble since user will see the gatk instance as a local application rather than some hidden process underneath docker. I am not sure if this is possible though. Users might need to compose their own Dockerfile for their own docker instance of gatk to get their UIDs and GIDs into the conda environment. I have been checking this for a while and could not get a definitive way of making an universal docker image that works under any UID:GID.
If there is a way I would love to use that.
-
Thanks for the very quick response! I'll pass this along to our engine team.
-
Hi @SkyWarrior,
Would something along the lines of the patch suggested at https://github.com/broadinstitute/gatk/issues/5959 do the trick?
I must admit that Docker best practices is something that is a little outside my purview, but I want to make sure it's as easy as possible to e.g. run gCNV in the intended environment.
-
I am not really sure but I will check.
Hi Samuel Lee
As a duct tape solution I gave read write execute permission recursively to the root directory of the latest gatk docker image and tried to run this image with my own UID:GID and python import commands now work properly. Top image is the one that I performed chmod -R 777 /root and the bottom image is the vanilla image
(gatk) [sky@K2-GOKALP ~]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
broadinstitute/gatk rootpermitted 6ef9568d652d 17 minutes ago 4.12GB
broadinstitute/gatk latest bc944dc20373 3 weeks ago 4.12GB
(gatk) [sky@K2-GOKALP ~]$(gatk) [sky@K2-GOKALP ~]$ docker run --rm -t -u $UID:$GID broadinstitute/gatk python -c "import gcnvkernel"
Traceback (most recent call last):
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1856, in filter_compiledir
os.makedirs(path, 0o770) # read-write-execute for user and group
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/miniconda/envs/gatk/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.theano'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/gcnvkernel/__init__.py", line 1, in <module>
from pymc3 import __version__ as pymc3_version
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/__init__.py", line 5, in <module>
from .distributions import *
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/__init__.py", line 1, in <module>
from . import timeseries
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 1, in <module>
import theano.tensor as tt
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/__init__.py", line 110, in <module>
from theano.compile import (
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/__init__.py", line 12, in <module>
from theano.compile.mode import *
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/compile/mode.py", line 11, in <module>
import theano.gof.vm
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/vm.py", line 674, in <module>
from . import lazylinker_c
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 42, in <module>
location = os.path.join(config.compiledir, 'lazylinker_ext')
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configparser.py", line 335, in __get__
self.__set__(cls, val_str)
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configparser.py", line 346, in __set__
self.val = self.filter(val)
File "/opt/miniconda/envs/gatk/lib/python3.6/site-packages/theano/configdefaults.py", line 1863, in filter_compiledir
" '%s'. Check the permissions." % path)
ValueError: Unable to create the compiledir directory '/root/.theano/compiledir_Linux-5.4--MANJARO-x86_64-with-debian-stretch-sid-x86_64-3.6.2-64'. Check the permissions.
(gatk) [sky@K2-GOKALP ~]$(gatk) [sky@K2-GOKALP ~]$ docker run --rm -t -u $UID:$GID broadinstitute/gatk:rootpermitted python -c "import gcnvkernel"
(gatk) [sky@K2-GOKALP ~]$ docker run --rm -t -u $UID:$GID broadinstitute/gatk:rootpermitted python -c "import vqsr_cnn"
Using TensorFlow backend.
(gatk) [sky@K2-GOKALP ~]$Do you think giving a permanent read write execute permission to the root directory of the gatk docker image will cause any security issues?
Alternatively
This configuration may be converted to the tmp dir or a parameter may be passed to the gCNV runner under gatk so that any user can use the unmodified gatk docker with their own UID:GID and will not have read write permission issues due to theano compile dir.
FINAL EDIT:
Forget about chmodding the root folder. It was a stupid idea.
I guess this line should be added to the environment variables or there may be a runtime modification by gatk python script runner to set this theano flag.
ENV THEANO_FLAGS="base_compiledir=/tmp"
Here is what I did. My Dockerfile
FROM broadinstitute/gatk:latest
ENV THEANO_FLAGS="base_compiledir=/tmp"and the result
(gatk) [sky@K2-GOKALP Desktop_Junk]$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
broadinstitute/gatk theanoenv 5410c5a3ca9e 2 minutes ago 4.12GB(gatk) [sky@K2-GOKALP Desktop_Junk]$ docker run --rm -it -u $UID:$GID broadinstitute/gatk:theanoenv python -c "import gcnvkernel"
(gatk) [sky@K2-GOKALP Desktop_Junk]$ -
Thanks, @SkyWarrior! Note that I added functionality along these lines a while back in https://github.com/broadinstitute/gatk/pull/6244 to address this and related issues (e.g, you can also get into trouble when scattering multiple instances of gCNV locally https://github.com/broadinstitute/gatk/issues/6235; setting THEANO_FLAGS allows you to assign each instance its own compilation directory, so they don't fight over it). The PR also allows you to set other flags that might improve theano performance and prevents gCNV itself from clobbering those flags with its own defaults.
I did add some tool documentation in that PR, but perhaps we can expand it or make it a bit more visible. I considered adding something like an --output-compilation-directory parameter to gCNV, but opted against it since it would've required additional code (and the THEANO_FLAGS approach is more flexible, anyway).
In any case, I think adding a non-root user to the Docker probably would've avoided the issue entirely and is more in line with Docker best practices. So I'll continue to explore that route.
Thanks again for your detailed responses!
Please sign in to leave a comment.
17 comments