I am trying to run CWL on a server that does not allow normal docker, but uses rootless docker instead (link at
https://docs.docker.com/engine/security/rootless/, but I’m not allowed >2 links in my message as a new user ).
I am testing a simple tool (cwl file here, job file here) and find that it works on my own computer with normal docker, but I get an error on the server with rootless docker. I’m not sure if there are other differences between my server and local machine that may be causing the problem though. Can somebody clarify if rootless docker is supported for CWL?
Welcome @jnitta !
Yes, CWL as a standard supports the use of rootless software container engines like Docker in that mode, or udocker.
However, different CWL aware workflow systems might not support every software container engine in every mode. I’m guessing that you are using the CWL reference runner,
That should work with rootless Docker, though I have not personally tested that.
Can you share the error you receive?
Okay, I did some local experimentation and I’ve been able to reproduce the issue. The working directory we mount into the container has the wrong permissions when using rootless docker:
The listed workaround would be quite complex to implement for
cwltool, but contributions to enhance cwltool this way would be welcome.
I think podman offers a rootless mode? Perhaps that doesn’t have this issue.
Thanks so much for looking into this! I hope it is also useful information for anybody else thinking about using rootless docker. Unfortunately making modifications to
cwltool to enable rootless docker is beyond my abilities.
I am going to try udocker as mentioned here. I am not familiar with podman. Is there a reason to use podman over udocker?
(And yes, I was using the reference runner. The original error is here).
Thanks so much for looking into this!
You are welcome!
I am not familiar with podman. Is there a reason to use podman over udocker?
podman should run faster than
udocker is a beautiful hack that runs on almost any Linux system without special privileges.
podman needs a somewhat recent Linux kernel, but that shouldn’t be an issue if rootless docker was installed as well.
https://github.com/containers/podman/blob/master/README.md#rootless and https://github.com/containers/podman/blob/9c5fe954cca8b4bcb8f552645e1f52a5d9824134/docs/tutorials/rootless_tutorial.md may be helpful.
I should add that I have not personally tested rootless
cwltool. Please let us know how it goes, if you do!
From your logs, I see that you are running a very old version of
cwltool as you were using Python 2:
While you mentioned you tried with Python 3 as well, I don’t know which version you tried.
However, I did experience the same issues with rootless docker and the latest version of
cwltool, but I wanted to make sure you didn’t run into other problems that we have already fixed.
When I tried with Python 3, I was running Python 3.8.6 and cwl-runner 3.0.20201121085451.
I will look into podman, thanks!
Just a side-note: we are using the
--singularity option since that seem to work nicely with our Docker registries. The environment we are running CWL are HPC systems where we have difficulties with
udocker or similar solutions. Singularity however works fine and you can even cache the created Singularity images so that everything works instantly.