Build and debug samtools_mpileup.cwl

CWL Questions
1

I am developing CWL tool for samtools_mpileup. People are invited to debug the tool.

(env3) root@ubuntu:/home/ngsap2# cwltool samtoolsMpileupDockerNew.cwl samtools.yaml 
INFO /home/ngsap2/env3/bin/cwltool 3.0.20200324120055
INFO Resolved 'samtoolsMpileupDockerNew.cwl' to 'file:///home/ngsap2/samtoolsMpileupDockerNew.cwl'
INFO [job samtoolsMpileupDocker] /tmp/xyvcttw2$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/xyvcttw2,target=/eFRfYZ \
    --mount=type=bind,source=/tmp/v05f2htg,target=/tmp \
    --mount=type=bind,source=/home/ngsap2/covid057Sorted.bam,target=/var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam,readonly \
    --mount=type=bind,source=/home/ngsap2/NormalizeFasta.fasta,target=/eFRfYZ/NormalizeFasta.fasta,readonly \
    --workdir=/eFRfYZ \
    --read-only=true \
    --user=0:0 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/eFRfYZ \
    --cidfile=/tmp/dcp6f4_t/20200525193121-275407.cid \
    quay.io/biocontainers/samtools:1.2-0 \
    'samtools mpileup -B -f /eFRfYZ/NormalizeFasta.fasta /var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam -o ncov.mpileup'
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"samtools mpileup -B -f /eFRfYZ/NormalizeFasta.fasta /var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam -o ncov.mpileup\": stat samtools mpileup -B -f /eFRfYZ/NormalizeFasta.fasta /var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam -o ncov.mpileup: no such file or directory": unknown.
ERRO[0001] error waiting for container: context canceled 
INFO [job samtoolsMpileupDocker] Max memory used: 0MiB
ERROR [job samtoolsMpileupDocker] Job error:
("Error collecting output for parameter 'rawMpileup':\nsamtoolsMpileupDockerNew.cwl:33:5: Did not find output file with glob pattern: '['ncov.mpileup']'", {})
WARNING [job samtoolsMpileupDocker] completed permanentFail
{}
WARNING Final process status is permanentFail
2

Welcome @Ambarish_Kumar!

Can you also share a link or post your CWL source?

As a reminder, command lines are a series of strings. When you type echo Hello world at the command line prompt, that is broken up into three pieces where there are spaces: "echo", "hello", "world". The first piece is used to find the program to execute on the $PATH, in this example it is echo and the program found is /bin/echo. Then /bin/echo is given the entire list of pieces and then it is up to /bin/echo to parse the command line and do something with it. Sometimes we want to instruct our shell (like BASH) to send an argument that has a space in it, like filename or a git commit message. Then we have to work around the “break the command line on spaces” processing by using quote marks or escape \ the spaces. For example: git commit -m 'hello world' becomes "git", "commit", "-m", "hello world", none of the quotation marks (single or double) get sent to /usr/bin/git in this example.

Now lets look at the output from running your CWL description:

'samtools mpileup -B -f /eFRfYZ/NormalizeFasta.fasta /var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam -o ncov.mpileup'

Note how this whole section is inside single quotes. So the shell (or in this case, the Docker engine) will try to look for a program named samtools mpileup -B -f /eFRfYZ/NormalizeFasta.fasta /var/lib/cwl/stg76b7cba7-a2c0-4738-85b4-5e7253da3b8e/covid057Sorted.bam -o ncov.mpileup instead of looking for a program named samtools and passing the rest of the options broken up at the spaces like normal.

3

https://seek4science.slack.com/files/U0141ADUY7P/F01471EA2AZ/samtoolsmpileupdockernew.cwl

4

https://seek4science.slack.com/files/U0141ADUY7P/F014EHUNJPN/samtools.yaml

5

Sir, above two links are for samtools_mpileup CWL tool and it’s input file.

6

Sir,
How to put both requirements block together?

requirements:
- class: ShellCommandRequirement
- class: InlineJavascriptRequirement

requirements:
  InitialWorkDirRequirement:
    listing: 
      - $(inputs.ReferenceGenome)
7

This one is the running samtools_mpileup.cwl

#!/usr/bin/env cwl-runner
# This tool description was generated automatically by wdl2cwl ver. 0.2

class: CommandLineTool
cwlVersion: v1.0

requirements:

  ShellCommandRequirement: {}

  InitialWorkDirRequirement:

    listing:

      - $(inputs.ReferenceGenome)

  InlineJavascriptRequirement: {}

hints:

  DockerRequirement:

    dockerPull: quay.io/biocontainers/samtools:1.2-0


inputs:
- id: inputBAM
  type: File
- id: ReferenceGenome
  type: File
- id: sampleName
  type: string


outputs:
- id: rawMpileup
  type: File
  outputBinding:
    glob: $(inputs.sampleName).mpileup

baseCommand: []
arguments:
- valueFrom: |-
    samtools index $(inputs.inputBAM.path) | samtools mpileup -B -f $(inputs.ReferenceGenome.path) $(inputs.inputBAM.path) > $(inputs.sampleName).mpileup
  shellQuote: false
id: samtoolsMpileupDocker
8

Here we have examples of non-map and mapped CWL. (More information is at Common Workflow Language (CWL) Command Line Tool Description, v1.1 )

I like the mapped form more, so I would combine this as

requirements:
  ShellCommandRequirement: {}
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing: 
      - $(inputs.ReferenceGenome)

Here is the combination using the non-mapped form

requirements:
- class: ShellCommandRequirement
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
  listing: 
    - $(inputs.ReferenceGenome)