Specifying staging or temporary inside CWL script

CWL Questions
1

Hi,
I am using cwltool --tmpdir-prefix <mydir> --tmp-outdir-prefix <mydir> to specify a temp / staging directory other than the system /tmp (since the size of the intermediate files produced can be huge in my case). How do I access this dir in the cwl script, so a commandline tool could use this instead of /tmp?

I tried the following

 - position: 3
    prefix: '--tempdir'
    valueFrom: $(runtime.tmpdir)

BUT this still gives me values of the form /tmp/<random_staging_dir> , instead of <mydir>/<random_staging_dir>

I am using

$ cwltool --version
cwltool 3.0.20200807132242

Thanks in advance!

2

Welcome @ashraf_borg!

Does your CWL CommandLineTool use a DockerRequirement? If so, then that /tmp path is a path inside the docker container. This should be mapped to a path in mydir; if not, that’s a bug in cwltool.

3

No It doesn’t have a DockerRequirement .
It has the following directives:

class: CommandLineTool
cwlVersion: v1.0
...
...
requirements:
  - class: ShellCommandRequirement
  - class: ResourceRequirement
    ramMin: 15000
    coresMin: 4
  - class: InlineJavascriptRequirement
  - class: ScatterFeatureRequirement

However, I just noticed for this particular CWL script even the input file is NOT staged in <mydir> (even though for other CWL in the pipeline, which incidentally do use DockerRequirement, it is).

This is the debug output I see from the cwltool

INFO [job interproscan_31] /data/tmp/cwlwork/qhx8vxkd$ /bin/sh \
    -c \
    '/ .... /interproscan-5.36-75.0/interproscan.sh' '--disable-precalc' '--goterms' '--pathways' '--tempdir' **'/tmp/tmpj84_o8fo'** '--input' **'/tmp/tmps1as2dii/stg31a69ccf-73ff-4742-ba77-6c945f9537a8/18300001_18600000.faa'** '--applications' 'PfamA,TIGRFAM,PRINTS,PrositePatterns,Gene3d' '--formats' 'TSV'
4

@mrc So is this a bug, if there is NO DockerRequirement in the CWL script?

5

Correct. Until this is fixed, you can work around this by setting TMPDIR, TEMP, or TMP environment variables as documented in https://docs.python.org/3.6/library/tempfile.html#tempfile.gettempdir

TMPDIR=<mydir> cwltool --tmpdir-prefix <mydir> --tmp-outdir-prefix <mydir>

I presume that you opened the issue at https://github.com/common-workflow-language/cwltool/issues/1373 ? Thanks for doing that!

1 Like