Commandline tool for fastp generates empty report html/json

CWL Questions
1

Hi there,
I wrote a command line tool for fastp that runs in Docker. The trimming worked as expected and with --verbose it also shows that a report in HTML and JSON is generated. The trimmed fastq files are written as expected, but the .html and .json report are returned as empty files.

Here is my Dockerfile:

# Set the base image to micromamba
FROM mambaorg/micromamba

# copy env.yml
COPY env.yml /tmp/env.yml

# install with micromamba
RUN micromamba install -y -n base -f /tmp/env.yml &&\
    micromamba clean --all --yes

and my env.yml

name: base
channels:
  - bioconda
  - conda-forge
dependencies:
  - fastp=0.23.4

and more importantly, my cwl workflow:

cwlVersion: v1.2
class: CommandLineTool

requirements:
  InlineJavascriptRequirement: {}

hints:
  DockerRequirement:
    dockerImageId: fastp
    dockerFile: Dockerfile

baseCommand: fastp

inputs:
  in1:
    type: File
    inputBinding:
      prefix: '--in1'

  in2:
    type: File?
    inputBinding:
      prefix: '--in2'

  out1:
    type: string?
    inputBinding:
      prefix: '--out1'

  out2:
    type: string?
    inputBinding:
      prefix: '--out2'

  json:
    type: string?
    inputBinding:
      prefix: '--json'

  html:
    type: string?
    inputBinding:
      prefix: '--html'

  thread:
    type: int?
    inputBinding:
      prefix: '--thread'

  verbose:
    type: boolean?
    inputBinding:
      prefix: '--verbose'

outputs:
  fastq_fastpOut1:
    type: File
    outputBinding:
      glob: $(inputs.out1)

  fastq_fastpOut2:
    type: File?
    outputBinding:
      glob: $(inputs.out2)

  json_fastpJsonReport:
    type: File
    outputBinding:
      glob: $(inputs.json)

  html_fastpHtmlReport:
    type: File
    outputBinding:
      glob: $(inputs.html)

This resulted in the empty files:

^[[1;30mDEBUG^[[0m ^[[32m[job initialWorkflow.cwl] Removing input staging directory /tmp/wc24vpev^[[0m
^[[1;30mDEBUG^[[0m ^[[32m[job initialWorkflow.cwl] Removing temporary directory /tmp/5ibhgwjh^[[0m
^[[1;30mDEBUG^[[0m ^[[32mMoving /tmp/urk5ljrz/CLO1_1_fp.fq.gz to /vol/ar/CWLFASTPBUG/CLO1_1_fp.fq.gz^[[0m
^[[1;30mDEBUG^[[0m ^[[32mMoving /tmp/urk5ljrz/CLO1_2_fp.fq.gz to /vol/ar/CWLFASTPBUG/CLO1_2_fp.fq.gz^[[0m
^[[1;30mDEBUG^[[0m ^[[32mMoving /tmp/urk5ljrz/initialHTML.html to /vol/ar/CWLFASTPBUG/initialHTML.html^[[0m
^[[1;30mDEBUG^[[0m ^[[32mMoving /tmp/urk5ljrz/initialJSON.json to /vol/ar/CWLFASTPBUG/initialJSON.json^[[0m
^[[1;30mDEBUG^[[0m ^[[32mRemoving intermediate output directory /tmp/urk5ljrz^[[0m
{
    "fastq_fastpOut1": {
        "location": "file:///vol/ar/CWLFASTPBUG/CLO1_1_fp.fq.gz",
        "basename": "CLO1_1_fp.fq.gz",
        "class": "File",
        "checksum": "sha1$45f596c6be34cfabafba018c2dd4a8fc4334adaf",
        "size": 591503360,
        "path": "/vol/ar/CWLFASTPBUG/CLO1_1_fp.fq.gz"
    },
    "fastq_fastpOut2": {
        "location": "file:///vol/ar/CWLFASTPBUG/CLO1_2_fp.fq.gz",
        "basename": "CLO1_2_fp.fq.gz",
        "class": "File",
        "checksum": "sha1$a60487a31d46cceaad68a64b961101cb7f578e00",
        "size": 580161536,
        "path": "/vol/ar/CWLFASTPBUG/CLO1_2_fp.fq.gz"
    },
    "html_fastpHtmlReport": {
        "location": "file:///vol/ar/CWLFASTPBUG/initialHTML.html",
        "basename": "initialHTML.html",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "path": "/vol/ar/CWLFASTPBUG/initialHTML.html"
    },
    "json_fastpJsonReport": {
        "location": "file:///vol/ar/CWLFASTPBUG/initialJSON.json",
        "basename": "initialJSON.json",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "path": "/vol/ar/CWLFASTPBUG/initialJSON.json"
    }
}^[[1;30mINFO^[[0m Final process status is success

I thought this was a problem with Docker so i ran the tool outside of Docker in my local conda environment, and the same problem occurred.
I then continued and manually created and empty html and json file, staged them alongside the inputs and I received the correctly written html and json reports!

cwlVersion: v1.2
class: CommandLineTool

requirements:
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - $(inputs.in1)
      - $(inputs.in2)
      - $(inputs.html)
      - $(inputs.json)

baseCommand: fastp

inputs:
  in1:
    type: File
    inputBinding:
      prefix: '--in1'

  in2:
    type: File?
    inputBinding:
      prefix: '--in2'

  out1:
    type: string?
    inputBinding:
      prefix: '--out1'

  out2:
    type: string?
    inputBinding:
      prefix: '--out2'

  json:
    type: File
    inputBinding:
      prefix: '--json'

  html:
    type: File
    inputBinding:
      prefix: '--html'
 thread:
    type: int?
    inputBinding:
      prefix: '--thread'

  verbose:
    type: boolean?
    inputBinding:
      prefix: '--verbose'

outputs:
  fastq_fastpOut1:
    type: File
    outputBinding:
      glob: $(inputs.out1)

  fastq_fastpOut2:
    type: File?
    outputBinding:
      glob: $(inputs.out2)

  json_fastpJsonReport:
    type: File
    outputBinding:
      glob: $(inputs.json.basename)

  html_fastpHtmlReport:
    type: File
    outputBinding:
      glob: $(inputs.html.basename)

Not quite what i wanted, because I had to create the files manually beforehand, but it is progress. I then went and tried this in Docker, but again the files where empty!

Again outside of Docker, I tried to Create the File on Runtime like so:

cwlVersion: v1.2
class: CommandLineTool

requirements:
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - $(inputs.in1)
      - $(inputs.in2)
      - entryname: stageFileNoDockerHTML.html
        entry: ''
      - entryname: stageFileNoDockerJSON.json
        entry: ''


baseCommand: [fastp, --html, stageFileNoDockerHTML.html, --json, stageFileNoDockerJSON.json]

inputs:
  in1:
    type: File
    inputBinding:
      prefix: '--in1'

  in2:
    type: File?
    inputBinding:
      prefix: '--in2'

  out1:
    type: string?
    inputBinding:
      prefix: '--out1'

  out2:
    type: string?
    inputBinding:
      prefix: '--out2'

  thread:
    type: int?
    inputBinding:
      prefix: '--thread'

  verbose:
    type: boolean?
    inputBinding:
      prefix: '--verbose'

outputs:
  fastq_fastpOut1:
    type: File
    outputBinding:
      glob: $(inputs.out1)

  fastq_fastpOut2:
    type: File?
    outputBinding:
      glob: $(inputs.out2)

  json_fastpJsonReport:
    type: File
    outputBinding:
      glob: stageFileNoDockerJSON.json

  html_fastpHtmlReport:
    type: File
    outputBinding:
      glob: stageFileNoDockerHTML.html

But again, the files where empty.

I also tried a different version of fastp (0.20.0) with no success.
I also tried running the tool from the bio-cwl-tools Github, which had the same problem!

Now I wonder what the problem could be? Is this a reproducible problem for others? Or am I doing something wrong, when it comes to cwl? (I am fairly new to CWL).
CWL tools version is 3.1.20230719185429,btw.
I would be thankful for any ideas,
Best wishes,
Walja

2

My tmp-dir was running full, and messages were not hinting towards it. So was an easy fix, and everything works as intended.

1 Like