"Singularity exec" vs "docker run"

CWL Questions
1

Hi there! New CWL/Toil user here.
We have a workflow where we noticed that depending on whether singularity or docker container choice it uses exec vs run. I am trying to understand what was the rational behind this difference and if there is a way to force to use singularity run in toil.

2

Hi Samee,

We have a workflow where we noticed that depending on whether singularity or docker container choice it uses exec vs run .

Can you clarify this? As in toil behaves differently between if presented with an .sif file compared to when presented with a Docker Image URI?

3

I haven’t used it, but from your description looks like you are talking about singularity exec vs. docker run? If so, that’d make sense, as both singularity exec and docker run allow you to start a container and pass the command you want it to run (e.g. python, bash, some-script.sh, etc.).

There’s also singularity run, but that does not behave like singularity exec or docker run. singularity run runs the default command in the Singularity container definition, defined by the maintainers of the container image. There’s also docker exec, but this is used to run a command in an existing container that’s already running.

Not sure if that answers your question, but maybe you are confused because these two tools use exec & run for different things (i.e. exec&run are semantically different in singularity & docker).

4

I have a cwl workflow that I can run either with docker or singularity. I understand the difference betweeen singularity exec vs docker run. My question is slightly different. Sorry for not being clear on that. Let me try to add some clarity.

Here is how I run a workflow with toil and choosing to use the docker container.

toil_cwl_runner /home/samee/bin/miniforge3/envs/sophios_dev/bin/sophios --disable-pull --cachedir cachedir --provenance provenance/bbbc --write-summary output_bbbc.json --skip-schemas --relax-path-checks --leave-outputs autogenerated/bbbc.cwl autogenerated/bbbc_inputs.yml

In this case I see that docker uses the run command to run the container.

docker \
    run \
    -i \
    --mount=type=bind,source=/home/samee/axle_dev/image-workflows/cachedir/b4aed9ba9b80fef2ca574a2febca212d,target=/zXKUXM \
    --mount=type=bind,source=/tmp/_a41e33r,target=/tmp \
    --workdir=/zXKUXM \
    --read-only=true \
...

However if I run the same workflow using singularity (by passing --singularity) in the command line, like following:

toils_cwl_runner /home/samee/bin/miniforge3/envs/sophios_dev/bin/sophios --disable-pull --cachedir cachedir --provenance provenance/bbbc --singularity --write-summary output_bbbc.json --skip-schemas --relax-path-checks --leave-outputs autogenerated/bbbc.cwl autogenerated/bbbc_inputs.yml

I see singularity exec is used by toil:

singularity \
    --quiet \
    exec \
    --contain \
    --ipc \
    --cleanenv \
    --userns \
....

So, my question is why the toil devs chose to use exec for singularity but run for docker?

Hope this makes my question clear :slight_smile:

5

I have traced the issue to these few lines of code in cwlrunner.
In singularity.py, we have the following:

and in docker.py, we have the following:

So, my question is what made the devs choose these two different way to launch the containers? Depending on whether the image has an entrypoint or not, it produces different outcome.

6

Here’s some context on why this decision was made: run doesn't respect quotes and escapes the same way exec does · Issue #3673 · apptainer/singularity · GitHub and the follow-up Compatibility mode to match Docker/OCI arg and env escaping / quoting behaviour · Issue #487 · sylabs/singularity · GitHub

2019 was a long time ago, so if there are options that we can pass to singularity and/or apptainer that better match the behavior of docker run, then I’m happy to review such change for cwltool.

Maybe adding --no-eval to the invocation for newer versions of singularity and apptainer , in addition to switching to run from exec will work?

7

A new cwltool release has been made with your fix; thank you @Samee !

Release 3.1.20241112140730 · common-workflow-language/cwltool · GitHub / Release 3.1.20241112140730 · common-workflow-language/cwltool · GitHub