How do i actually get memory_retry_multiplier to work in cromwell on GCP?

Follow

Gregory Dougherty

• June 07, 2024 20:07

Cromwell RetryWithMoreMemory instructions say I should set system.memory-retry-error-keys and memory_retry_multiplier in order to get retries to allocate more memory for a task when it fails with one of the strings in system.memory-retry-error-keys appearing in stderr.

I have a task failing with a memory error.  It has the string "Killed" in it, which is one of the system.memory-retry-error-keys.  The rc is 1.  But every retry has the same available memory as the first run (I know this because I run free -h right before the failure, and it gives the same message every time)

Is there example code showing successful use of this claimed feature?

0

• 

  Gökalp Çelik

  • June 14, 2024 13:55

  Hi Gregory Dougherty

  Can you provide us the log file created by cromwell for this task so that we can start debugging the issue? Specifically we will check for signatures of OOM within the log but there is no single standard code for that so we need to see the log. 

  Regards. 

  0

  Comment actions Permalink

Please sign in to leave a comment.