What would we need to do (or change) to build an isolated Conda environment for each workflow step?
We have tools with SoftwarePackage declarations that have conflicting dependencies, that cannot be resolved in a single Conda environment. Using
--no-container, cwltool creates a directory
./cwltool_deps that is shared by all workflow steps. This can be overridden in a dependency resolver file, but seems to be always global for the workflow. We need to build a separate environment for each step.
I think that the backend
galaxy.tools.deps.conda package will build an isolated environment if
cache_path is empty, and guess that we could pin that in the Conda entry in a custom dependency resolver file. But I don’t see how to push that value towards the resolver.
This might also provide the second mechanism suggested in workflow with parallel steps and --beta-conda-dependencies failed due to many conda install at the same time · Issue #1331 · common-workflow-language/cwltool · GitHub.
I understand of course that we could use a user-space container, but we would specifically prefer to leverage our existing Conda setup, without building a bespoke container image each time.