Does a listing support arrays of arrays?

CWL Questions
1

Do I understand correctly that listing in the InitialWorkDirRequirement does not support arrays (-) of arrays? Unfortunately it seems we have implemented a CWL creator that creates the first piece of code, but that does not work if the inputs.Input is already an array of files.

requirements:
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
  listing:
  - $(inputs.Inputs)
label: Create Reference Genome TAR bundle
doc: Bundles all the files for a reference genome bundle into a single TAR file
inputs:
  Inputs:
    type:
      type: array
      items: File
      inputBinding:
        valueFrom: $(self.basename)
    inputBinding:
      position: 99

Is not working, but this is:

requirements:
- class: InlineJavascriptRequirement
- class: InitialWorkDirRequirement
  listing: $(inputs.Inputs)
label: Create Reference Genome TAR bundle
doc: Bundles all the files for a reference genome bundle into a single TAR file
inputs:
  Inputs:
    type:
      type: array
      items: File
      inputBinding:
        valueFrom: $(self.basename)
    inputBinding:
      position: 99
2

This post here seems to suggest this as well: https://www.biostars.org/p/318493/

3

Your first example works for me?
Here it is again as a complete CommandLineTool

cwlVersion: v1.0
class: CommandLineTool
requirements:
- class: InitialWorkDirRequirement
  listing:
    - $(inputs.Inputs)
label: Create Reference Genome TAR bundle
doc: Bundles all the files for a reference genome bundle into a single TAR file
inputs:
  Inputs:
    type:
      type: array
      items: File
      inputBinding:
    inputBinding:
      position: 99
outputs: []
baseCommand: ls

$ cwltool listing_array.cwl --Inputs README.rst --Inputs CONTRIBUTING.md 
INFO /home/michael/cwltool/env3.7/bin/cwltool 1.0.20191022103248
INFO Resolved 'listing_array.cwl' to 'file:///home/michael/cwltool/listing_array.cwl'
INFO [job listing_array.cwl] /tmp/6dwv047p$ ls
CONTRIBUTING.md  README.rst
INFO [job listing_array.cwl] completed success
{}
INFO Final process status is success
4

I’m getting this result with your code:

cwl-runner --debug /arrays.cwl /files.yml
/usr/bin/cwl-runner 1.0.20180302231433
Resolved '/arrays.cwl' to 'file:///arrays.cwl'
[job arrays.cwl] initializing from file:///arrays.cwl
[job arrays.cwl] {
    "Inputs": [
        {
            "class": "File",
            "location": "file:///aap/noot/mies/one.txt",
            "size": 14,
            "basename": "one.txt",
            "nameroot": "one",
            "nameext": ".txt"
        },
        {
            "class": "File",
            "location": "file:///tmp/two.txt",
            "size": 4,
            "basename": "two.txt",
            "nameroot": "two",
            "nameext": ".txt"
        }
    ]
}
Got workflow error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cwltool/executors.py", line 98, in run_jobs
    for r in jobiter:
  File "/usr/lib/python2.7/dist-packages/cwltool/command_line_tool.py", line 417, in job
    self.updatePathmap(builder.outdir, builder.pathmapper, l)
  File "/usr/lib/python2.7/dist-packages/cwltool/command_line_tool.py", line 241, in updatePathmap
    for sf in fn.get("secondaryFiles", []):
AttributeError: 'CommentedSeq' object has no attribute 'get'
Workflow error, try again with --debug for more information:
'CommentedSeq' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cwltool/main.py", line 588, in main
    **vars(args))
  File "/usr/lib/python2.7/dist-packages/cwltool/executors.py", line 29, in __call__
    return self.execute(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cwltool/executors.py", line 69, in execute
    self.run_jobs(t, job_order_object, logger, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cwltool/executors.py", line 113, in run_jobs
    raise WorkflowException(Text(e))
WorkflowException: 'CommentedSeq' object has no attribute 'get'
5

And here is the code that DOES work for me:

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
requirements:
- class: InitialWorkDirRequirement
  listing: $(inputs.Inputs)
label: Create Reference Genome TAR bundle
doc: Bundles all the files for a reference genome bundle into a single TAR file
inputs:
  Inputs:
    type:
      type: array
      items: File
      inputBinding:
    inputBinding:
      position: 99
outputs: []
baseCommand: ls

The only difference is the listing: $(inputs.Inputs)

6

/usr/bin/cwl-runner 1.0.20180302231433

I think that is the issue; your version of the CWL reference runner (cwltool) is a over a year old, and contains a bug.

2 Likes
7

Hmmm…I just did an apt-get install cwltool as instructed on the github page…

But I installed the latest version and now indeed it works!

Thanks and we can close this one…

8

Ah, looks like you’re on Ubuntu. You can request a backport of the newer version. Would be nice for the other Ubuntu users to get the latest features:

https://wiki.ubuntu.com/UbuntuBackports#Requesting_a_Backport

1 Like