Currently, the docker pull is specified at the command line tool level, as an example below
- class: DockerRequirement
dockerPull: 'ghcr.io/msk-access/abra2:2.22'
I wanted to check is there a way to specify the docker pull registry location on the fly when running a workflow level and not at the command line tool level. Can an additional input be provided which can tell where the image should be pulled from?
Hello @rhshah , that is not a feature in CWL itself, no. A particular CWL implementation (a workflow runner or platform) might offer such a feature.
Thank you @mrc , Any implementation that you can think of that offers such functionality? I agree it has nothing to do with CWL specification but if cwltool the core implementation can offer something like this it will definitely help.
@rhshah I don’t know off-hand. For cwltool you can help us by requesting the feature at https://github.com/common-workflow-language/cwltool/issues/new ; let us know how you’d like to work. A default registry if no registry is specified? A registry override? Per-step registries?
not an answer but this reminds me of the discussion here; Working offline with singularity
Its pretty much the same issue; the need to be able to parameterize the container locations so that you can more easily change the container configs for workflows and tools without having to hard-code everything into each CWL. One of the solutions they had for this in Nextflow was the use of “profiles” that could define all the settings for each process and you could select different profiles at run time. Essentially allowing a stronger separation between the task definition (the tool you are running and its inputs & outputs) and the task execution criteria (containers, environment, CPU & mem resources, etc).
Not sure if anything similar exists in CWL yet? As it is, I have to use a Perl one-liner every single time I version bump a container and need to propagate the change to every CWL.