Secondary files cannot be collected by cwltool on Windows 10

CWL Questions
1

Hi,

I’m trying to make a workflow work on Windows 10 with cwltool but it seems there is an issue either with permissions on Windows (see below warning) or obtaining secondary files from output directory.

Here is a minimal tool to reproduce the error:

# tool.cwl
class: CommandLineTool
cwlVersion: v1.0

baseCommand:
  - touch
arguments:
  - $(inputs.filename + ".txt")
  - "&&"
  - touch
  - $(inputs.filename + ".txt.idx")
inputs:
  - id: filename
    type: string
outputs:
  - id: output_file
    type: File
    secondaryFiles: [.idx]
    outputBinding:
      glob: "*.txt"
hints:
  - class: DockerRequirement
    dockerPull: 'ubuntu:latest'
requirements:
  InlineJavascriptRequirement: {}
  ShellCommandRequirement: {}

And here is the input:

# input.yml
filename: myfile

This is how I run the tool on CMD:

cwltool tool.cwl input.yml

Here are the logs from cwltool:

INFO C:\Users\gungor\Desktop\Projeler\test-cwltool-for-secondary-files\env\Scripts\cwltool 3.0.20200530110633
INFO Resolved 'tool.cwl' to 'file:///C:/Users/gungor/Desktop/Projeler/test-cwltool-for-secondary-files/tool.cwl'
INFO [job tool.cwl] C:\Users\gungor\AppData\Local\Temp\z5pg3yy3$ docker \
    run \
    -i \
    --mount=type=bind,source=C:\Users\gungor\AppData\Local\Temp\z5pg3yy3,target=/vwbTiM \
    --mount=type=bind,source=C:\Users\gungor\AppData\Local\Temp\coktl1p4,target=/tmp \
    --workdir=/vwbTiM \
    --read-only=true \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/vwbTiM \
    --cidfile=C:\Users\gungor\AppData\Local\Temp\00vsij3x\20200604034605-489063.cid \
    ubuntu:latest \
    /bin/sh \
    -c \
    'touch' 'myfile.txt' && 'touch' 'myfile.txt.idx'
WARNING Ignored error with docker stats: [Errno 13] Permission denied: 'C:\\Users\\gungor\\AppData\\Local\\Temp\\funug1r7'
ERROR [job tool.cwl] Job error:
("Error collecting output for parameter 'output_file':\nC:Users\\gungor\\Desktop\\Projeler\\test-cwltool-for-secondary-files\\tool.cwl:17:5: Output file path /myfile.tool.cwl:17:5: Output file path /myfile.txt.idx must be within designated output directory (/vwbTiM) or an input file pass through.", {})
WARNING [job tool.cwl] completed permanentFail
{}
WARNING Final process status is permanentFail

The warning that concerns me is following, it did not go away even after I enabled file sharing from Docker settings:

WARNING Ignored error with docker stats: [Errno 13] Permission denied: 'C:\\Users\\gungor\\AppData\\Local\\Temp\\funug1r7'

Can anyone point me to the right direction so I can solve this issue?

Thanks!

2

I solved the error after digging the code a little further. The issue was that the path separator to determine secondary file path was hardcoded as / which was failing on Windows 10. I just replaced it with os.sep and it started working. I’m still not sure about the permission warning though. I’d appreciate any comment on it.

1 Like
3

Hello @gungorbudak and thanks for posting. We don’t properly test the CWL reference runner on Windows, however we have a good experience running it under WSL 2; can you try the directions for installing cwltool under WSL 2 ?