I am trying to run CWL on a server that does not allow normal docker, but uses rootless docker instead (link at https://docs.docker.com/engine/security/rootless/, but I’m not allowed >2 links in my message as a new user ).
I am testing a simple tool (cwl file here, job file here) and find that it works on my own computer with normal docker, but I get an error on the server with rootless docker. I’m not sure if there are other differences between my server and local machine that may be causing the problem though. Can somebody clarify if rootless docker is supported for CWL?
Yes, CWL as a standard supports the use of rootless software container engines like Docker in that mode, or udocker.
However, different CWL aware workflow systems might not support every software container engine in every mode. I’m guessing that you are using the CWL reference runner, cwltool?
That should work with rootless Docker, though I have not personally tested that.
Okay, I did some local experimentation and I’ve been able to reproduce the issue. The working directory we mount into the container has the wrong permissions when using rootless docker:
The listed workaround would be quite complex to implement for cwltool, but contributions to enhance cwltool this way would be welcome.
I think podman offers a rootless mode? Perhaps that doesn’t have this issue.
Thanks so much for looking into this! I hope it is also useful information for anybody else thinking about using rootless docker. Unfortunately making modifications to cwltool to enable rootless docker is beyond my abilities.
I am going to try udocker as mentioned here. I am not familiar with podman. Is there a reason to use podman over udocker?
(And yes, I was using the reference runner. The original error is here).
I am not familiar with podman. Is there a reason to use podman over udocker?
podman should run faster than udocker as udocker is a beautiful hack that runs on almost any Linux system without special privileges. podman needs a somewhat recent Linux kernel, but that shouldn’t be an issue if rootless docker was installed as well.
From your logs, I see that you are running a very old version of cwltool as you were using Python 2: 1.0.20191225192155
While you mentioned you tried with Python 3 as well, I don’t know which version you tried.
However, I did experience the same issues with rootless docker and the latest version of cwltool, but I wanted to make sure you didn’t run into other problems that we have already fixed.
Just a side-note: we are using the --singularity option since that seem to work nicely with our Docker registries. The environment we are running CWL are HPC systems where we have difficulties with udocker or similar solutions. Singularity however works fine and you can even cache the created Singularity images so that everything works instantly.
).