Synopsis: Singularity is installed and it works standalone, the CWL file can be run through the docker container using docker. The container and program can be called through singularity. The problem is that when I attempt to instruct CWL to use singularity, it throws an error that singularity is not detected.
- Alright so I have singularity in two places:
- In a typical space:
/usr/bin/singularity - And in one that is more specific to our setup:
/mnt/bioinformatics/auxiliary_programs/Singularity/bin/singularity - Both are exported in $PATH
- Singularity can be called alone:
- Running:
singularity exec docker://ubuntu echo "hello world" - Yields:
INFO: Using cached SIF image hello world - Running:
singularity exec docker://my.registry/signalp-5 signalp -h - Yields:
INFO: Using cached SIF image Usage of /bin/signalp-5.0b/bin/signalp: -batch int Number of sequences that the tool will run simultaneously. Decrease or increase size depending on your system memory. (default 10000) -fasta string Input file in fasta format. -format string Output format. 'long' for generating the predictions with plots, 'short' for the predictions without plots. (default "short") -gff3 Make gff3 file of processed sequences. -mature Make fasta file with mature sequence. -org string Organism. Archaea: 'arch', Gram-positive: 'gram+', Gram-negative: 'gram-' or Eukarya: 'euk' (default "euk") -plot string Plots output format. When long output selected, choose between 'png', 'eps' or 'none' to get just a tabular file. (default "png") -prefix string Output files prefix. (default "Input file prefix") -stdout Write the prediction summary to the STDOUT. -tmp string Specify temporary file directory. (default "System default tmpdir") -verbose Verbose output. Specify '-verbose=false' to avoid printing. (default true) -version Prints version.
- CWL Inputs:
- cwl
#!/usr/bin/env cwl-runner #signalp-singularity.cwl cwlVersion: v1.0 class: CommandLineTool label: SignalP Analysis inputs: fasta_file: type: File inputBinding: prefix: -fasta separate: true stdout_name: type: string outputs: output: type: stdout stdout: $(inputs.stdout_name) baseCommand: signalp arguments: - -stdout hints: - class: DockerRequirement dockerPull: my.registry/signalp-5 - yml:
# signalp5.yml fasta_file: class: File path: test_fasta.fa stdout_name: test_fasta.fa.tsv
- Running with docker:
- Command:
cwltool --debug signalp-singularity.cwl signalp5.yml - Output:
INFO /home/user/.local/bin/cwltool 3.1.20211001174446 INFO Resolved 'signalp-singularity.cwl' to 'file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/signalp-singularity.cwl' DEBUG [job signalp-singularity.cwl] initializing from file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/signalp-singularity.cwl DEBUG [job signalp-singularity.cwl] { "fasta_file": { "class": "File", "location": "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "size": 210, "basename": "test_fasta.fa", "nameroot": "test_fasta", "nameext": ".fa" }, "stdout_name": "test_fasta.fa.tsv" } DEBUG [job signalp-singularity.cwl] path mappings is { "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa": [ "/mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "/var/lib/cwl/stge0af255e-7eed-4ae1-b339-dc6a2e4b8a02/test_fasta.fa", "File", true ] } DEBUG [job signalp-singularity.cwl] command line bindings is [ { "position": [ -1000000, 0 ], "datum": "signalp" }, { "position": [ 0, 0 ], "datum": "-stdout" }, { "prefix": "-fasta", "separate": true, "position": [ 0, "fasta_file" ], "datum": { "class": "File", "location": "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "size": 210, "basename": "test_fasta.fa", "nameroot": "test_fasta", "nameext": ".fa", "path": "/var/lib/cwl/stge0af255e-7eed-4ae1-b339-dc6a2e4b8a02/test_fasta.fa", "dirname": "/var/lib/cwl/stge0af255e-7eed-4ae1-b339-dc6a2e4b8a02" } } ] DEBUG [job signalp-singularity.cwl] initial work dir {} INFO [job signalp-singularity.cwl] /tmp/jdx0o577$ docker \ run \ -i \ --mount=type=bind,source=/tmp/jdx0o577,target=/THQZwP \ --mount=type=bind,source=/tmp/88j9ac31,target=/tmp \ --mount=type=bind,source=/mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa,target=/var/lib/cwl/stge0af255e-7eed-4ae1-b339-dc6a2e4b8a02/test_fasta.fa,readonly \ --workdir=/THQZwP \ --read-only=true \ --log-driver=none \ --user=10134:10134 \ --rm \ --cidfile=/tmp/p5sfpp2a/20211112185415-691653.cid \ --env=TMPDIR=/tmp \ --env=HOME=/THQZwP \ my.registry/signalp-5 \ signalp \ -stdout \ -fasta \ /var/lib/cwl/stge0af255e-7eed-4ae1-b339-dc6a2e4b8a02/test_fasta.fa > /tmp/jdx0o577/test_fasta.fa.tsv INFO [job signalp-singularity.cwl] Max memory used: 309MiB INFO [job signalp-singularity.cwl] completed success DEBUG [job signalp-singularity.cwl] outputs { "output": { "location": "file:///tmp/jdx0o577/test_fasta.fa.tsv", "basename": "test_fasta.fa.tsv", "nameroot": "test_fasta.fa", "nameext": ".tsv", "class": "File", "checksum": "sha1$5a4154061e1200574652da07b5da948c4a1c2f0d", "size": 136, "http://commonwl.org/cwltool#generation": 0 } } DEBUG [job signalp-singularity.cwl] Removing input staging directory /tmp/k9im_71s DEBUG [job signalp-singularity.cwl] Removing temporary directory /tmp/88j9ac31 DEBUG Moving /tmp/jdx0o577/test_fasta.fa.tsv to /mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa.tsv { "output": { "location": "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa.tsv", "basename": "test_fasta.fa.tsv", "class": "File", "checksum": "sha1$5a4154061e1200574652da07b5da948c4a1c2f0d", "size": 136, "path": "/mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa.tsv" } } INFO Final process status is success
- Trying with Singularity:
- Command:
cwltool --debug --singularity signalp-singularity.cwl signalp5.yml - Output:
INFO /home/user/.local/bin/cwltool 3.1.20211001174446 INFO Resolved 'signalp-singularity.cwl' to 'file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/signalp-singularity.cwl' DEBUG [job signalp-singularity.cwl] initializing from file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/signalp-singularity.cwl DEBUG [job signalp-singularity.cwl] { "fasta_file": { "class": "File", "location": "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "size": 210, "basename": "test_fasta.fa", "nameroot": "test_fasta", "nameext": ".fa" }, "stdout_name": "test_fasta.fa.tsv" } DEBUG [job signalp-singularity.cwl] path mappings is { "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa": [ "/mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "/var/lib/cwl/stgeea7a54a-2697-48cd-8553-4977923afcc4/test_fasta.fa", "File", true ] } DEBUG [job signalp-singularity.cwl] command line bindings is [ { "position": [ -1000000, 0 ], "datum": "signalp" }, { "position": [ 0, 0 ], "datum": "-stdout" }, { "prefix": "-fasta", "separate": true, "position": [ 0, "fasta_file" ], "datum": { "class": "File", "location": "file:///mnt/bioinformatics/workingDirectories/user/Misc/CWL_Testing/signalp/test_fasta.fa", "size": 210, "basename": "test_fasta.fa", "nameroot": "test_fasta", "nameext": ".fa", "path": "/var/lib/cwl/stgeea7a54a-2697-48cd-8553-4977923afcc4/test_fasta.fa", "dirname": "/var/lib/cwl/stgeea7a54a-2697-48cd-8553-4977923afcc4" } } ] DEBUG Singularity version: singularity-ce version 3.8.0. DEBUG Singularity error Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/cwltool/job.py", line 774, in run self.get_from_requirements( File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 249, in get_from_requirements if not self.get_image(cast(Dict[str, str], r), pull_image, force_pull): File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 113, in get_image if is_version_3_or_newer(): File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 43, in is_version_3_or_newer return int(get_version()[0]) >= 3 ValueError: invalid literal for int() with base 10: 's' ERROR Workflow error: Singularity is not available for this tool, try --no-container to disable Singularity, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: invalid literal for int() with base 10: 's' Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/cwltool/job.py", line 774, in run self.get_from_requirements( File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 249, in get_from_requirements if not self.get_image(cast(Dict[str, str], r), pull_image, force_pull): File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 113, in get_image if is_version_3_or_newer(): File "/home/user/.local/lib/python3.8/site-packages/cwltool/singularity.py", line 43, in is_version_3_or_newer return int(get_version()[0]) >= 3 ValueError: invalid literal for int() with base 10: 's' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/cwltool/main.py", line 1247, in main (out, status) = real_executor( File "/home/user/.local/lib/python3.8/site-packages/cwltool/executors.py", line 60, in __call__ return self.execute(process, job_order_object, runtime_context, logger) File "/home/user/.local/lib/python3.8/site-packages/cwltool/executors.py", line 143, in execute self.run_jobs(process, job_order_object, logger, runtime_context) File "/home/user/.local/lib/python3.8/site-packages/cwltool/executors.py", line 250, in run_jobs job.run(runtime_context) File "/home/user/.local/lib/python3.8/site-packages/cwltool/job.py", line 825, in run raise WorkflowException( cwltool.errors.WorkflowException: Singularity is not available for this tool, try --no-container to disable Singularity, or install a user space Docker replacement like uDocker with --user-space-docker-cmd.: invalid literal for int() with base 10: 's'