Using JavaScript in requirements such as DockerPull

CWL Questions
1

Hello,

I have been using JavaScript in my resource requirements to dynamically determine the instance type to use based on my workflow inputs.

Backstory:
We’re running Illumina’s BCLConvert. This workflow scatters over the bclconvert tool over as many times as there are samplesheet configurations in the input list.

I have attached a diagram of the workflow cwl-ica/.github/catalogue/docs/workflows/bclconvert-with-qc-pipeline/4.0.3/bclconvert-with-qc-pipeline__4.0.3.md at main · umccr/cwl-ica · GitHub

And linked the tool cwl-ica/tools/bcl-convert/4.0.3/bcl-convert__4.0.3.cwl at main · umccr/cwl-ica · GitHub

For each bclconvert tool invocation, I would like to run the tool with cpu hardware unless the parameter --compression-format is set to --dragen or --dragen-interleaved (in which case I’d need to run this on CPU hardware) for each configuration in the list.

I am able to update the resource requirements dynamically (and the script InitialWorkDirRequirement can handle both dragen invocation of bclconvert and the cpu invocation), however, I’d like to also be able to change the docker file (since FPGA hardware needs a different docker image to the CPU BCLConvert).

I can use the cwltool overrides option for dockerPull , however this would override the image for all tool invocations, and this might be different for different items.

I note that in 2.3. Expressions — Common Workflow Language User Guide 0.1 documentation (commonwl.org) it states JavaScript expressions are only allowed in certain spots (like ResourceRequirement and InitialWorkDirRequirement), is there a reason this is forbidden in DockerRequirement?

Alexis

2

Hey Alexis. Overrides can apply to a specific step or sub-step

3

Hi @mrc

Thank you for getting back to me so quickly.

Because the workflow is scattered I don’t think there’d be a way to apply the override to one element of the scatter and not the rest as they’d have the same step names (bcl_convert_step).

4

Can you split the scattered step into 2? One CPU and one GPU? Then merge the downstream links.

5

The scatter calls a workflow (rather than a tool), so I can’t specify the overrides at the scatter step level.

Without writing a separate tool for the different hardwares, I don’t think this is possible.

Regardless, writing separate workflows / tools for each hardware use case may be the most ‘portable’ CWL (over using conditionals inside the scatter).

6

Here is the previous discussion about that question: Expressions in dockerPull DockerRequirement · Issue #476 · common-workflow-language/common-workflow-language · GitHub

7

Thanks for finding this

I should mention, one of the reasons I’d prefer to avoid introducing expressions is that it confounds static analysis of the CWL document. The Arvados runner, for example, ensures that all the Docker images that will be used by the workflow are available before beginning execution, and it wouldn’t be able to do that if the Docker image was not known until runtime.

Makes a lot of sense