Cwl-runner and podman

CWL Questions
1

I read that there is interim support for podman (docker replacement) in CWL.

I am running CWL via conda, is the version difference too great to support podman at the moment ?

Are there alternative way to get the latest CWL (1.2?) without building lots of packages ? I am running CentOS 8.2

Cheers

2

Hello @Nicholas_Yue

I read that there is interim support for podman (docker replacement) in CWL.

CWL, as a standard, does not specify that only certain software container engines are supported. So from the perspective of the standard there is nothing to prevent any CWL aware runner using podman instead of docker, singularity, udocker , and so on.

cwl-runner is the generic name for any CWL capable runner. A well known CWL runner is the CWL reference implementation, cwltool. There does not appear to be any interim support for podman in cwltool: https://github.com/common-workflow-language/cwltool/search?q=podman However, support for additional non-docker software container runtimes has been identified as a desired enhancement to cwltool: Investigate rootless containers using runc / umoci · Issue #1086 · common-workflow-language/cwltool · GitHub

I am running CWL via conda,

I think you are referring to cwltool’s conda package?

is the version difference too great to support podman at the moment ?

According to Podman and Buildah for Docker users | Red Hat Developer (under “Podman commands are the same as Docker’s”) podman can act identically to docker. Seems that in some Linux distribution you can install the podman-docker package; if not you need to otherwise cause podman to be available as docker. To confirm that this works I recommend running the CWL conformance tests: https://github.com/common-workflow-language/common-workflow-language/blob/0c28d10a188e5da9a730d4db35511377337af9c1/CONFORMANCE_TESTS.md

If this is successful then cwltool can be enhanced to use podman if docker is not available.

Are there alternative way to get the latest CWL (1.2?) without building lots of packages ? I am running CentOS 8.2

The most recommend method is to make a Python 3 virtualenv and to pip install cwltool inside of it. I’m not able to personally support other methods on installing the CWL reference runner; perhaps others can.

Good luck and let us know!

3

I did a simple test using podman version 2.0.6 (and I linked /usr/bin/podman to /usr/local/bin/docker) on Debian with the CWL v1.0 conformance tests against cwltool version 3.0.20200807132242 and “175 tests passed, 22 failures”. This might be enough for your needs.

4

Seems the podman+cwltool conformance tests failures were related to permissions, so I re-ran as the root user and all the CWL v1.0 conformance tests passed. Likewise for the CWL v1.1 and CWL v1.2 conformance tests.

5

Hello @mrc

We got everything running with podman (on jupyterhub/kubernetes) with a small “hack” line

with:

    if not os.path.exists(source):
        os.makedirs(source)
    os.chmod(source, 0o777)

This fixes the issue with the permissions on the volumes. I’ve been wondering if you guys would find it acceptable to fully support podman as a replacement for docker.

(related to Document podman compatability · Issue #1344 · common-workflow-language/cwltool · GitHub)

1 Like
6

Yes, can you please send a pull request?

Then we can wire up testing with podman as well as docker

7

We have the same issue with calrissian already PR here: Ensure `tmp_out` is readable for all users by emmanuelmathot · Pull Request #111 · Duke-GCB/calrissian · GitHub. But the future PR on CWLtool could solve the issue at the root.

1 Like
8

For traceability: chmods 777 for podman by fabricebrito · Pull Request #1516 · common-workflow-language/cwltool · GitHub

1 Like