I feel like I’m missing something obvious.
I am using cromwell to run cwl on an HPC cluster. The cluster uses slurm for job management. I want to be able to adjust the resources requested of slurm, dependent on what different tasks within the cwl workflow. i.e. I would like to request more memory, time and cpus when running bwa than when I am running a samtools sort. To me, this seems like a fairly basic requirement, there must be documentation around this somewhere surely?
Cromwell allows you to specify a config file that overrules the defaults, creating the slurm job file. I can’t find anything that tells me how to get information from the cwl file, to that config file though, which means that at the moment, I can only specify a single set of slurm job requirements with no ability to customize.
I thought it might have involved runtime variables - but a) I don’t appear to be able to pass them to the cromwell config file and b) I can’t see a way to pass variables that I define, like how much time I want the job to reserve.
Hi @tirohia you should be using https://www.commonwl.org/v1.1/CommandLineTool.html#ResourceRequirement then it is the job of the CWL runner (Cromwell in this case) to translate that into the slurm resource request.
I checked and I see that ResourceRequirement is missing from the User Guide, so that’s probably why you are confused. We have a grant starting soon to help fund improvements to CWL documentation so hopefully it will get some attention in the next few months.
Can you use custom ResourceRequirements? One of the parameters out implementation of slurm requires is a maximum run time. Without it, it defaults to an hour.
I’m having the additional problem of cromwell not picking up the ResourceRequirements, but from what you say, that’s a cromwell issue?
I would be surprised if Cromwell did not support
ResourceRequirement but I haven’t ever used it so I cannot say for sure.
CWL v1.1 adds a ToolTimeLimit to set the runtime. However the implementation you are using needs to support it. I have not checked recently but I do not believe Cromwell supports CWL v1.1. You might want to look at Toil or Arvados which both support Slurm.
Worse case you can edit the Cromwell slurm template to specify a large maximum wall time like 24 hours. But it would better if Cromwell supported the ToolTimeLimit