# CWL - specifying Directory outputs as an input

**URL:** <https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107>\
**Category:** CWL Questions\
**Created:** [March 30, 2020, 10:45pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107 "2020-03-30T22:45:12Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [March 30, 2020, 10:45pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/1 "2020-03-30T22:45:12Z")

</div>

Hello,

I am working on a tool definition that requires specifying an input parameter for the outputs location.  
I have managed to write the following tool definition, that works as expected when testing with cwltool.

```auto
cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: umccr/alpine_pandas:1.0.1

requirements:
  InitialWorkDirRequirement:
    listing:
      - entry: $(inputs.outputDir)
        writable: true

baseCommand: []

inputs:
  script:
    type: File
    inputBinding:
      position: 0
  samplesheet:
    type: File
    inputBinding:
      position: 1
      prefix: -s
  metadata:
    type: File
    inputBinding:
      position: 2
      prefix: -t
  outputDir:
    type: Directory
    inputBinding:
      position: 4
      prefix: -o

outputs:
  splitSheets:
    type:
      type: array
      items: [File, Directory]
    outputBinding:
      glob: "*"

```

I am having an issue when trying to port this to an environment that does not allow writing to input files/directories. So, trying to find a workaround for this issue. I’m trying an alternate approach where the `outputDir` is passed as a string, i.e.

```auto
.
.
  outputDir:
    type: string
    inputBinding:
    position: 4
    prefix: -o

outputs:
  splitSheets:
    type:
      type: array
      items: [File, Directory]
    outputBinding:
      glob: "$(inputs.outputDir)"

```

But that produces the following error:

```auto
("Error collecting output for parameter 'splitSheets':\nsamplesheetPrep2.cwl:41:7: glob patterns must not start with '/'", {})

```

I have also tried specifying `inputs.outputDir` as a writable entry under `InitialWorkDirRequirement` and then specifying glob as

```auto
outputs:
  splitSheets:
    type:
      type: array
      items: [File, Directory]
    outputBinding:
      glob: "*"

```

This produces following error `OSError: [Errno 30] Read-only file system: '/Users'`

Wondering if there could be any other solution to this issue?  
Any help will be appreciated.

Cheers,  
Sehrish

---

<div class="post-metadata">

**Author:** ![kaushik-work](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/kaushik-work/32/11_2.png) [@kaushik-work](https://cwl.discourse.group/u/kaushik-work)\
**Post date:** [March 31, 2020, 2:01pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/2 "2020-03-31T14:01:00Z")

</div>

Hello,

If I understood the interface of your tool correctly, you just have to pass the input name as a string and then in the output glob that directory. So it will look like

```auto
cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: umccr/alpine_pandas:1.0.1

# Pass the working directory name here to the command line
baseCommand: []

inputs:
  script:
    type: File
    inputBinding:
      position: 0
  samplesheet:
    type: File
    inputBinding:
      position: 1
      prefix: -s
  metadata:
    type: File
    inputBinding:
      position: 2
      prefix: -t
  outputDir:
    type: string
    inputBinding:
      position: 4
      prefix: -o

outputs:
  splitSheets:
    type: Directory
    outputBinding:
      glob: "$(inputs.outputDir)"

```

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [March 31, 2020, 10:38pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/3 "2020-03-31T22:38:16Z")

</div>

Hi Kaushik,

Thanks for the response. I tried capturing `output` explicitly as a directory.

It still throws the glob error `("Error collecting output for parameter 'splitSheets':\nsamplesheetPrep2.cwl:43:7: glob patterns must not start with '/'", {})`

I think it does not like specifying `outputDir` as a string path that is starting with `/Users/....`

---

<div class="post-metadata">

**Author:** ![kaushik-work](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/kaushik-work/32/11_2.png) [@kaushik-work](https://cwl.discourse.group/u/kaushik-work)\
**Post date:** [April 1, 2020, 2:31am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/4 "2020-04-01T02:31:29Z")

</div>

Can the program take a relative path. Typically I enter a relative path, something like `output_dir`

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [April 1, 2020, 8:41am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/5 "2020-04-01T08:41:47Z")

</div>

I was still getting the following error when specifying a relative path for `outputDir`.

```auto
("Error collecting output for parameter 'splitSheets':\nsamplesheetPrep2.cwl:43:7: Did not find output file with glob pattern: '['cwl']'", {})

```

I had to update requirements as following to make it work - thanks to Michael F. for the hints

```auto
InitialWorkDirRequirement:
    listing:
    - '$({class: "Directory", basename: inputs.outputdir, listing: []})'

```

Hope this makes sense to others as well.  
Thank you.

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [April 1, 2020, 8:56am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/6 "2020-04-01T08:56:19Z")

</div>

> [@kaushik-work](#):
>
> Pass the working directory name here to the command line  
> baseCommand:

Sorry, I missed this bit - what did you mean by adding the working dir name before the base command?

---

<div class="post-metadata">

**Author:** ![kaushik-work](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/kaushik-work/32/11_2.png) [@kaushik-work](https://cwl.discourse.group/u/kaushik-work)\
**Post date:** [April 1, 2020, 11:10am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/7 "2020-04-01T11:10:47Z")

</div>

That was a reminder to pass the string to the command line, and that is done via the `inputBinding`.

---

<div class="post-metadata">

**Author:** ![kaushik-work](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/kaushik-work/32/11_2.png) [@kaushik-work](https://cwl.discourse.group/u/kaushik-work)\
**Post date:** [April 1, 2020, 11:38am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/8 "2020-04-01T11:38:00Z")

</div>

@skanwal Glad your problem was solved.

For completeness, here is an example where the command line program creates a directory and the directory is globbed as an output

`outdir.cwl`

```auto
cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: python:alpine3.11

baseCommand: [python]
arguments: [$(inputs.script.path), $(inputs.outdir)]

inputs:
  script: File
  outdir: string

outputs:
  mydir:
    type: Directory
    outputBinding:
      glob: "$(inputs.outdir)"

```

Where the input script is `script.py`

```auto
import pathlib
import sys

out = sys.argv[1]
pathlib.Path(out).mkdir()

```

This can be run as

`cwltool outdir.cwl --script script.py --outdir hello`

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [April 2, 2020, 6:52am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/9 "2020-04-02T06:52:24Z")

</div>

Hi Kaushik,

Thanks for the example, it makes sense.

I believe to capture the contents in the output directory, it’s necessary to pass `InitialWorkDirRequirement` with appropriate listing array?

---

<div class="post-metadata">

**Author:** ![illusional](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/illusional/32/53_2.png) [@illusional](https://cwl.discourse.group/u/illusional)\
**Post date:** [April 2, 2020, 6:56am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/10 "2020-04-02T06:56:26Z")

</div>

I’ll give some context to the solution, inside the [`InitialWorkDirRequirement`](https://www.commonwl.org/v1.0/CommandLineTool.html#InitialWorkDirRequirement), you return `listing` value which can be an `array<File | Directory | Dirent | string | Expression> | string | Expression`:

> May be an expression. If so, the expression return value must validate as `{type: array, items: [File, Directory]}`

Hence through the expression we can return a valid Directory object with attributes:

- `class`: “Directory” (duh),
- `basename`: Name of the folder (no leading slashes or anything)
- `listing`: empty list (`[]`), but you could return anything in here if you wanted the directory to have something in here.

Nb: wrap the expression in a string for it to be valid yaml.

This gives the tool:

```cwl
class: CommandLineTool
baseCommand: ls

requirements:
  InitialWorkDirRequirement:
    listing:
      - '$({class: "Directory", basename: inputs.outputdir, listing: []})'

inputs:
  outputdir: string

outputs:
  outdir:
    type: Directory
    outputBinding:
      glob: $(inputs.outputdir)
  outls: stdout

```

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [April 2, 2020, 10:03pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/11 "2020-04-02T22:03:14Z")

</div>

Hi Michael,

Thanks for adding to the discussion.

`listing` : empty list ( `[]` ), but you could return anything in here if you wanted the directory to have something in here.

Just another point, I was able to return files in the directory by using an empty list i.e. `listing: []`.

---

<div class="post-metadata">

**Author:** ![kaushik-work](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/kaushik-work/32/11_2.png) [@kaushik-work](https://cwl.discourse.group/u/kaushik-work)\
**Post date:** [April 3, 2020, 10:39am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/12 "2020-04-03T10:39:31Z")

</div>

Hi @skanwal, the `InitialWorkDirRequirement` is to stage or create files/directories during the job setup phase. It is separate from the output gathering stage. The example I provided does not have an `InitialWorkDirRequirement` because I don’t want to stage any inputs, merely gather the output directory, which the tool is creating.

---

<div class="post-metadata">

**Author:** ![skanwal](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/skanwal/32/55_2.png) [@skanwal](https://cwl.discourse.group/u/skanwal)\
**Post date:** [April 5, 2020, 2:23am UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/13 "2020-04-05T02:23:07Z")

</div>

Thanks a lot Kaushik. That makes absolute sense.

---

<div class="post-metadata">

**Author:** ![Jyotsana\_Mehra](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/jyotsana_mehra/32/556_2.png) [@Jyotsana\_Mehra](https://cwl.discourse.group/u/Jyotsana_Mehra)\
**Post date:** [May 15, 2024, 6:55pm UTC](https://cwl.discourse.group/t/cwl-specifying-directory-outputs-as-an-input/107/14 "2024-05-15T18:55:55Z")

</div>

**I am also facing the same error.**  
INFO [job varscan4.cwl] Max memory used: 230MiB  
ERROR [job varscan4.cwl] Job error:  
(“Error collecting output for parameter ‘vcf’: varscan4.cwl:39:7: Did not find output file with glob pattern: [‘output.vcf’].”, {})  
WARNING [job varscan4.cwl] completed permanentFail  
{}WARNING Final process status is permanentFail

**Here is the cwl script**

```auto
cwlVersion: v1.0
class: CommandLineTool
label: "VarScan Variant Calling"

requirements:
  - class: DockerRequirement
    dockerImageId: "kboltonlab/varscan2:1.1"

baseCommand: ["java", "-jar", "/opt/varscan/VarScan.v2.4.2.jar"]

arguments:
  - "pileup2cns"
  - "$(inputs.bam.path)"
  - "$(inputs.reference.path)"
  - "--output-vcf"
  - "1"

inputs:
  bam:
    type: File
    inputBinding:
      position: 3
  reference:
    type: File
    inputBinding:
      position: 2
    secondaryFiles: [.fai]

outputs:
  vcf:
    type: File
    outputBinding:
      glob: "output.vcf"

```

**and this is the json file with the input params**

```json
{
  "bam": {
    "class": "File",
    "path": "/home/ec2-user/healthomics_SI/varscan/SRRoutput.pileup",
    "secondaryFiles": [
      {
        "class": "File",
        "path": "/home/ec2-user/healthomics_SI/varscan/ref_with_svs.bai"
      }
    ]
  },
  "reference": {
    "class": "File",
    "path": "/ngs/reference/Homo_sapiens_assembly38.fasta",
    "secondaryFiles": [
        {"class": "File", "path": "/ngs/reference/Homo_sapiens_assembly38.fasta.fai"},
    ]
  },
  "strand_filter": 0,
  "min_coverage": 8,
  "min_var_freq": 0.1,
  "min_reads": 2,
  "p_value": 0.99,
  "sample_name": "save.vcf"
}

```