Wget CommandLineTool works with v1.0 but not with v1.2

CWL Questions
1

I just stumbled across something that I find interesting. The following CommandLineTool (source: DAT2-cwl/wget.cwl at main · pitagora-network/DAT2-cwl · GitHub) will work with v1.0, but not with v1.2:

cwlVersion: v1.0
class: CommandLineTool
doc: The non-interactive network downloader
requirements:
  DockerRequirement:
    dockerPull: alpine:3.9
  InlineJavascriptRequirement: {}
baseCommand: wget
inputs:
  url:
    doc: Download target URL
    type: string
    inputBinding: {}
  output_name:
    doc: Output file name (use `wget-stdout.txt` by default)
    type: string
    default: wget-stdout.txt
    inputBinding:
      prefix: --output-document=
      separate: false
      valueFrom: "$(inputs.use_remote_name ? inputs.url.split('/').pop() : self)"
  use_remote_name:
    doc: Use the basename of `url` parameter as an output file name. It is equivalent to `curl -O`.
    type: boolean
    default: true
outputs:
  downloaded:
    type: File
    outputBinding:
      glob: "$(inputs.use_remote_name ? inputs.url.split('/').pop() : inputs.output_name)"
  stderr: stderr
stderr: wget-stderr.log

This runs fine with cwlVersion: v1.0 but breaks with cwlVersion: v1.2 :person_shrugging:.

The error message is

WARNING [job download.pmc_oa_pdf_list] exited with status: 1
ERROR [job download.pmc_oa_pdf_list] Job error:
("Error collecting output for parameter 'downloaded': tools/generic/wget.cwl:30:7: Did not find output file with glob pattern: '['oa_non_comm_use_pdf.csv']'.", {})
WARNING [job download.pmc_oa_pdf_list] completed permanentFail
DEBUG [job download.pmc_oa_pdf_list] outputs {}
ERROR [step download.pmc_oa_pdf_list] Output is missing expected field file:///home/OBFUSCATED/src/my-project/workflow.cwl#download.pmc_oa_pdf_list/downloaded
DEBUG [step download.pmc_oa_pdf_list] produced output {}
WARNING [step download.pmc_oa_pdf_list] completed permanentFail
INFO [workflow ] completed permanentFail

Perhaps I mistakenly expected the versioning to be semantic, or I’ve made a newbie mistake. In any case, there’s nothing wrong (I think) with using v1.0 for this tool, but I’m wondering why this doesn’t work? Is there an overview/changelog to see the changes that 1.2 introduced as compared to 1.0?

Thanks!

2

Hello @sdruskat ; Please don’t increment the cwlVersion manually, but use the cwl-upgrader instead. And if you aren’t adding anything to your description that needs a newer CWL version, then you don’t have to increment the version number to match other files. Mixing CWL versions is supported.

In your case, running cwl-upgrader will add the missing networkAccess: true

BTW, if you need a remote file available over HTTP(S) then it is often better to put the URL directly in the location field of your input object, as many CWL-aware workflow platforms will cache that file on repeated runs

Finally, you can see the full changelog for CWL v1.1 at Common Workflow Language (CWL) Command Line Tool Description, v1.1

1 Like
3

Thanks for both hints, I’ll do that!

1 Like