Cromwell call-caching size restriction
How do I restrict the space used for the Cromwell call caching database?
I am currently running workflows with Cromwell 84 with call caching. Call caching is working mostly fine, except that I have not figured out how to restrict the size of the call caching database. My current workaround is to periodically stop all Cromwell workflows and truncate the call caching database. In standard cache designs, there is a limited amount of space, and entries get evicted to stay within that limit.
The documentation for call caching has very few options. The blacklist-cache does not appear to address the issue:
The call caching blacklist cache is off by default. This cache is used to blacklist cache hits based on cache hit ids or buckets of cache hit paths that Cromwell has previously failed to copy for permissions reasons.
I want entries to be cleared when space runs out, not only when there is a failure to copy call caching results.
The buckets/hits sections of blacklist-cache basically looks like what I want (size control), but not applied to blacklisting. If this is what I want, what is the difference between the bucket and hit configuration?
hits {
# Guava cache concurrency.
concurrency: 10000
# How long entries in the cache should live from the time of their last access.
ttl: 1 hour
# Maximum number of entries in the cache.
size: 2000
}
-
Hi Matt!
The GATK devs are not Cromwell superusers, but you could try the Cromwell-specific slack workspace: https://join.slack.com/t/cromwellhq/shared_invite/zt-2r05hmi3 If no one there is able to help, let me know and I can add you to the DSP Workflows slack channel on Broad Slack (which is a privilege afforded to other Broadies, though they have limited bandwith for non-Terra questions).
Good luck!
Laura
-
The response from the Cromwell slack is there is no existing feature to restrict the call cache size.
-
Well that's disappointing, but thanks for reporting back.
Please sign in to leave a comment.
3 comments