I’ve been asked to provide the following information about
cwltool for a catalogue of workflow environments are being elaborated the HealthyCloud project. So I thought I’d list my answers publicly, in case I missed something.
Main communityMain user community
Common Workflow Language, a member project of the Software Freedom Conservancy
cwltoolis used to
- By workflow language developers to prototype new features for future versions of the CWL standards
- By workflow engine developers, as an extensible CWL parsing and execution library
- By workflow description authors, for local testing of CWL documents.
cwltool · PyPI
Open source (Y/N, indicate license)
Yes, Apache 2.0
Maturity (TRL level or similar)
TRL-6 (“Technology demonstrated in relevant environment”;
cwltoolis not for production usage by itself)
Orchestrator level (Platform, Application, Service, Infrastructure)
Not an orchestrator of external systems.
Main execution paradigm (Task base, MPI, web services …)
Only for local running of tools, perhaps in parallel, perhaps containerized
Being used with health data (Y/N, which data)
Yes. See Notable CWL mentions, adoption, and users · common-workflow-language/common-workflow-language Wiki · GitHub
Support for distributed data (Y/N, indicate short summary of how this is achieved)
cwltooldoes support HTTP(S) downloading of remote data.
Other systems that use the cwltool library do support distributed data, for example: Arvados
Support for execution in multi-domain computing infrastructure (Y/N)
cwltooldoes not distribute tasks, but other workflow systems based upon
cwltooldo: toil-cwl-runner, arvados, streamflow, reana
Support for sensitive data (Y/N, how this is dealt with)
Support for reproducibility/provenance (Y/N, mechanism, i.e. containers, etc)
Yes. Docker format containers can be run using podman, singularity, udocker, or the docker container engines.
Provenance reporting is done using common provenance standards in a profile we call CWLProv GitHub - common-workflow-language/cwlprov: Profile for provenance research object of a CWL workflow run
Farah Zaib Khan, Stian Soiland-Reyes, Richard O Sinnott, Andrew Lonie, Carole Goble, Michael R Crusoe, Sharing interoperable workflow provenance: A review of best practices and their practical application in CWLProv, GigaScience, Volume 8, Issue 11, November 2019, giz095, https://doi.org/10.1093/gigascience/giz095
Main initiatives where it has been used (give number and a short list in case of many)
As the reference runner for the CWL standards, there are a great many initiatives benefiting from or using cwltool directly: See Notable CWL mentions, adoption, and users · common-workflow-language/common-workflow-language Wiki · GitHub