Attempting to stage multiple files into a directory

CWL Questions
1

I am attempting to pass in multiple files as inputs and stage those specific files into a directory, which is provided as a named input to my tool. Specifically, this is for hisat2. I am unable to provide them up front in a directory due to other limitations not related to CWL or hisat2. Here is an example of how I’ve constructed the folder:

requirements:
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - entryname: "hisat2_index"
        entry: |
          ${
            return {
              "class": "Directory",
              "listing": [ inputs.hisat2_idx_1, inputs.hisat2_idx_2, inputs.hisat2_idx_3, inputs.hisat2_idx_4, inputs.hisat2_idx_5, inputs.hisat2_idx_6, inputs.hisat2_idx_7, inputs.hisat2_idx_8 ]
            };
          }

And here is how I am attempting to use it:

arguments:
  - prefix: -x
    valueFrom: hisat2_index

I have also tried:

arguments:
  - prefix: -x
    valueFrom: $(runtime.outdir)/hisat2_index

In both cases I receive the error (ERR): "hisat2_index" does not exist or (ERR): "/vBrQqO/hisat2_index" does not exist respectively. Is there something I’m missing?

2

I find it easier to specify the directory base name in the returned Directory object.

requirements:
  InlineJavascriptRequirement: {}
  InitialWorkDirRequirement:
    listing:
      - |2
          ${
            var asset = {
              basename: "hisat2_index",
              class: "Directory",
              listing: [
                inputs.hisat2_idx_1,
                inputs.hisat2_idx_2,
                inputs.hisat2_idx_3,
                inputs.hisat2_idx_4,
                inputs.hisat2_idx_5,
                inputs.hisat2_idx_6,
                inputs.hisat2_idx_7,
                inputs.hisat2_idx_8
              ],
              writable: false
            };
            return asset;
          }
3

How exactly do I reference that directory? When I use your changes and run with the --debug flag it appears that the directory is correctly being created:

DEBUG [job hisat2.cwl] initial work dir {
    "_:40d2c1d0-92eb-4686-a1be-67c81157d6c9": [
        "_:40d2c1d0-92eb-4686-a1be-67c81157d6c9",
        "/CkSeWc/hisat2_index",
        "Directory",
        true
    ],

and I do see it being populated with files:

"file:///my_input_path/hg38.1.ht2": [
        "/my_input_path/hg38.1.ht2",
        "/CkSeWc/hisat2_index/hg38.1.ht2",
        "File",
        true
    ],

However, when I try to use that directory like so:

arguments:
  - prefix: -x
    valueFrom: $(runtime.outdir)/hisat2_index

I get the error:

(ERR): "/CkSeWc/hisat2_index" does not exist
4

I’ve done some test runs keep the tmpdir for inspection and am having trouble making sense of the results. When I run with a container image I see it’s mounting the tmpdir like so:

--mount=type=bind,source=/tmp/fx8xwvn0,target=/LqxIIY

and when I ls /tmp/fx8xwvn0 I see that hisat2_index exists and is populated as expected. However, I am getting the error (ERR): "/LqxIIY/hisat2_index" does not exist. It seems to me that based on the mounting that that path should exist.

I’ve also tried running this without a container to attempt to diagnose if this is just a container specific issue. When I do that I see

INFO [job hisat2.cwl] /tmp/hh3_owur$ hisat2 \
    -x \
    /tmp/hh3_owur/hisat2_index \
    -k \
    1 \
    -1 \
    /tmp/sp2h1bwd/stg4906f7cd-d64d-4d88-bbaa-35df80ba6f2c/tmp_r1.fq \
    -2 \
    /tmp/sp2h1bwd/stgae1e9627-dafa-4298-9c13-c65f1da4c329/tmp_r2.fq \
    -S \
    aligned.sam
(ERR): "/tmp/hh3_owur/hisat2_index" does not exist

but I can run ls /tmp/hh3_owur/hisat2_index and see that it does exist and is populated with all of the files as expected.

5

I swapped to using this

requirements:
  InitialWorkDirRequirement:
    listing:
      - entryname: hisat2_index/hg38.1.ht2
        entry: $(inputs.hisat2_idx_1)
      - entryname: hisat2_index/hg38.2.ht2
        entry: $(inputs.hisat2_idx_2)
      - entryname: hisat2_index/hg38.3.ht2
        entry: $(inputs.hisat2_idx_3)
      - entryname: hisat2_index/hg38.4.ht2
        entry: $(inputs.hisat2_idx_4)
      - entryname: hisat2_index/hg38.5.ht2
        entry: $(inputs.hisat2_idx_5)
      - entryname: hisat2_index/hg38.6.ht2
        entry: $(inputs.hisat2_idx_6)
      - entryname: hisat2_index/hg38.7.ht2
        entry: $(inputs.hisat2_idx_7)
      - entryname: hisat2_index/hg38.8.ht2
        entry: $(inputs.hisat2_idx_8)

and it’s working now. I’m not entirely sure why, but at least it’s working now. Perhaps I had a typo somewhere.