Best practice when specifying Java RAM requirement

CWL Questions
1

I’m putting this out here for discussion. Consider a Java tool whose memory usage needs to be specified. Hardcoding this on the command line seems not to be a good practice for me, so perhaps something like this is good:

baseCommand: [java]
arguments:
  - "-Xmx$(Math.ceil(runtime.ram / 1000))G"

The runtime RAM can be altered using a ResourceRequirement. This allows the RAM to be specified in a way that could inform e.g. a scheduler that is scheduling the job and the final computed value passed to the command line at runtime. It does, of course, require that the tool uses a InlineJavascriptRequirement.

Thoughts?

2

Wouldn’t Math.ceil lead to Out of Memory errors?

3

On the Java side? In my limited testing this seems to work - expression by default results in -Xmx2G.

4

On an HPC system that kills jobs that go over their requested memory, rounding up could lead to java using more than the allocation.

5

Yes that is a danger. The ResourceRequirement ramMin can be a fractional value that gets rounded up, but that is mebibytes, not GB. So then Math.floor would be better and default would be -Xmx1G.

6

$(runtime.ram) will give the rounded value, so one could skip the rounding and InlineJavascriptRequirement and use "-Xmx$(runtime.ram)m"

7

Indeed one could. Re: your comment about InlineJavascriptRequirement, is $(runtime.ram) not an expression (and thus requiring that requirement)?

8

In the CWL community call that happened after this forum discussion, @mrc pointed out that a parameter reference is not the same thing as an Expression. So this $(runtime.ram) is a parameter reference because it simply references a parameter of the tool or workflow and it only becomes an Expression when computation (e.g. $(runtime.ram * 2)) is involved. Parameter references do not, as was previously pointed out, require an InlineJavascriptRequirement.