Hi,

I am running GATK CNVs Germline in the Contig Ploidy step. I have an error related with theano folder. If a delete the folder, then the script goes well but when I tried a new running the error comes again. This error starts happening recently, but I can't figure out the reason behind.

a) GATK version used: gatk-4.2.5.0

b) Exact command used: 

   gatk DetermineGermlineContigPloidy        

c) Entire program log:

12:04:01.344 INFO  DetermineGermlineContigPloidy - Shutting down engine

[November 2, 2023 12:04:01 PM ART] org.broadinstitute.hellbender.tools.copynumber.DetermineGermlineContigPloidy done. Elapsed Time: 50.96 minutes.

Runtime.totalMemory()=3073376256

org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:

python exited with 1

Command Line: python /tmp/cohort_determine_ploidy_and_depth.2789903316707324493.py --sample_coverage_metadata=/tmp/samples-by-cov

erage-per-contig8948993380936208813.tsv --output_calls_path=/grupos/Marce/clinica/pipeline/tools/GATK-CNVs/output_dir/cohort_ploi

dy/less10reads/ploidy-calls --mapping_error_rate=1.000000e-02 --psi_s_scale=1.000000e-04 --mean_bias_sd=1.000000e-02 --psi_j_scal

e=1.000000e-03 --learning_rate=5.000000e-02 --adamax_beta1=9.000000e-01 --adamax_beta2=9.990000e-01 --log_emission_samples_per_ro

und=2000 --log_emission_sampling_rounds=100 --log_emission_sampling_median_rel_error=5.000000e-04 --max_advi_iter_first_epoch=100

0 --max_advi_iter_subsequent_epochs=1000 --min_training_epochs=20 --max_training_epochs=100 --initial_temperature=2.000000e+00 --

num_thermal_advi_iters=5000 --convergence_snr_averaging_window=5000 --convergence_snr_trigger_threshold=1.000000e-01 --convergenc

e_snr_countdown_window=10 --max_calling_iters=1 --caller_update_convergence_threshold=1.000000e-03 --caller_internal_admixing_rat

e=7.500000e-01 --caller_external_admixing_rate=7.500000e-01 --disable_caller=false --disable_sampler=false --disable_annealing=fa

lse --interval_list=/tmp/intervals5111471100799611651.tsv --contig_ploidy_prior_table=/grupos/Marce/clinica/pipeline/tools/GATK-C

NVs/bundle/hg38_contig_ploidy_priors.tsv --output_model_path=/grupos/Marce/clinica/pipeline/tools/GATK-CNVs/output_dir/cohort_plo

idy/less10reads/ploidy-model

Stdout: 11:53:48.626 INFO cohort_determine_ploidy_and_depth - THEANO_FLAGS environment variable has been set to: device=cpu,float

X=float64,optimizer=fast_run,compute_test_value=ignore,openmp=true,blas.ldflags=-lmkl_rt,openmp_elemwise_minsize=10

11:53:49.334 INFO gcnvkernel.structs.metadata - Generating intervals metadata...

11:53:49.556 INFO gcnvkernel.tasks.task_cohort_ploidy_determination - Instantiating the germline contig ploidy determination model...

11:53:58.767 INFO theano.gof.compilelock - Waiting for existing lock by process '877760' (I am process '877995')

11:53:58.767 INFO theano.gof.compilelock - To manually release the lock, delete /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir

11:58:34.823 INFO theano.gof.compilelock - Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

11:58:34.823 WARNING theano.gof.compilelock - Refreshing lock failed, we release the lock before raising again the exception

11:58:34.824 ERROR theano.gof.cmodule - [Errno 2] No such file or directory: '/home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock'

Stderr: INFO (theano.gof.compilelock): Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

INFO (theano.gof.compilelock): Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

INFO (theano.gof.compilelock): Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

INFO (theano.gof.compilelock): Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

INFO (theano.gof.compilelock): Waiting for existing lock by process '877760' (I am process '877995')

INFO (theano.gof.compilelock): To manually release the lock, delete /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir

INFO (theano.gof.compilelock): Waiting for existing lock by process '877760' (I am process '877995')

INFO (theano.gof.compilelock): To manually release the lock, delete /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir

INFO (theano.gof.compilelock): Refreshing lock /home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock

WARNING (theano.gof.compilelock): Refreshing lock failed, we release the lock before raising again the exception

ERROR (theano.gof.cmodule): [Errno 2] No such file or directory: '/home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock'

Traceback (most recent call last):

  File "/tmp/cohort_determine_ploidy_and_depth.2789903316707324493.py", line 122, in <module>    ploidy_task = gcnvkernel.CohortPloidyInferenceTask(ploidy_inference_params, ploidy_config, ploidy_workspace)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/gcnvkernel/tasks/task_cohort_ploidy_determination.py", line 92, in __init__

    ploidy_model = PloidyModel(ploidy_config, ploidy_workspace)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/model.py", line 197, in __call__

    instance.__init__(*args, **kwargs)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/gcnvkernel/models/model_ploidy.py", line 221, in __init__

    shape=(ploidy_workspace.num_contigs,))

File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/distribution.py", line 517, in __call__

    *args, **kwargs)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/distribution.py", line 39, in __new__

    return model.Var(name, dist, data, total_size)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/model.py", line 522, in Var

    model=self)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/model.py", line 1094, in __init__

    transformed_name, transform.apply(distribution), total_size=total_size)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/transforms.py", line 36, in apply

    return TransformedDistribution.dist(dist, self)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/distribution.py", line 49, in dist

    dist.__init__(*args, **kwargs)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/distributions/transforms.py", line 66, in __init__

    v = forward(FreeRV(name='v', distribution=dist))

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/pymc3/model.py", line 872, in __init__

    self.scaling = _get_scaling(total_size, self.shape, self.ndim)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/tensor/var.py", line 275, in <lambda>

    shape = property(lambda self: theano.tensor.basic.shape(self))

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 670, in __call__

    no_recycling=[])

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk

    no_recycling)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk

    output_storage=node_output_storage)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk

    keep_lock=keep_lock)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in __compile__

    keep_lock=keep_lock)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1624, in cthunk_factory

    key=key, lnk=self, keep_lock=keep_lock)

File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk

    keep_lock=keep_lock)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in __compile__

    keep_lock=keep_lock)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1624, in cthunk_factory

    key=key, lnk=self, keep_lock=keep_lock)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1189, in module_from_key

    module = lnk.compile_cmodule(location)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/cc.py", line 1517, in compile_cmodule

    get_lock()

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/compilelock.py", line 106, in _get_lock

    refresh_lock(lockpath)

  File "/grupos/Marce/clinica/tools/envs/gatk/lib/python3.6/site-packages/theano/gof/compilelock.py", line 329, in refresh_lock

    with open(lock_file, 'w') as lock_write:

FileNotFoundError: [Errno 2] No such file or directory: '/home/mmartinez/.theano/compiledir_Linux-5.4--generic-x86_64-with-debian-bullseye-sid-x86_64-3.6.10-64/lock_dir/lock'