Collecting logs in a folder: scatter+input array to `ExpressionTool`

CWL Questions
1

Hello,
Following this, I’m using an ExpressionTool to put the logs of my workflow tasks in a folder.
I’m scattering through the inputs of my tasks, processing multiple files at once. So task1 and task2 have the scatter keyword.
An example of my workflow:

inputs:
  evt: File[]
steps:
  task1:
    run: task1.cwl
    scatter: evt
    in:
      evt: evt
    out: [out, log]

  task1:
    run: task2.cwl
    scatter: evt
    in:
      evt: task1/out
    out: [out, log]

Now, I can manage to do what I want by appending a step group_logs in my workflow using this kind of syntax:

group_logs:
 run: group_files.cwl
 in:
   file1: task1/log
   file2: task2/log
 out: [out]

where group_files.cwl is:

cwlVersion: v1.2
class: ExpressionTool
requirements:
  InlineJavascriptRequirement: {}
inputs:
  file1: File[]
  file2: File[]
outputs:
  out: Directory
expression: |
  ${
    return {"out": {
      "class": "Directory", 
      "basename": "logs",
      "listing": [...inputs.file1, ...inputs.file2]
    } };
  }

and it works well.

But I’d like to be more flexible by using a single File[] as input to the ExpressionTool:

group_logs:
  run: group_files_array.cwl
  in: 
    files: [task1/log, task2/log]
  out: [out]

defining this analogous ExpressionTool group_files_array.cwl:

cwlVersion: v1.2
class: ExpressionTool
requirements:
  InlineJavascriptRequirement: {}
inputs:
  files: File[]
outputs:
  out: Directory
expression: |
  ${
    return {"out": {
      "class": "Directory", 
      "basename": "logs",
      "listing": [...inputs.files]
    } };
  }

Unfortunately this fails with:

Source 'log' of type {"type": "array", "items": "File"} is
incompatible with sink 'files' of type {"type": "array", "items": "File"}
source has linkMerge method merge_nested

I can’t understand what’s the problem, any help?
Probably I can’t see where to put an eventual: linkMerge: merge_flattened.

Thanks a lot!

2

I’ve found this to be useful for gathering outputs into subdirectories. It’s reasonably versatile. Perhaps it will be useful to you as well…

make-subdirs.cwl:

#!/usr/bin/env cwl-runner

######-------------------------------------------------------------------------------######
# This ExpressionTool returns a directory containing the files/directories provided.
# The dir_files input is extremely flexible. It can contain a single file or directory,
# or lists of files and/or directories, any of which may be or contain null values. Nested
# lists are unpacked recursively. Directories are added as themselves rather than their
# individual file listings.
# Author: Alex Tate
######-------------------------------------------------------------------------------######

cwlVersion: v1.2
class: ExpressionTool

requirements:
  - class: InlineJavascriptRequirement
  - class: InitialWorkDirRequirement
    listing: $(inputs.dir_files)

inputs:
  dir_files: Any?
  dir_name: {type: string?, default: "default_dirname"}

outputs:
  subdir: Directory

expression: |
  ${
    function add_array(files){
      var listing = [];
      for (var i in files){
        var item = files[i]
        if (item == null) continue;
        if (Array.isArray(item)) listing.push.apply(listing, add_array(item));
        if (["File", "Directory"].includes(item["class"])) listing.push(item);
      }
      return listing;
    }

    return {"subdir": {
      "class": "Directory",
      "basename": inputs.dir_name,
      "listing": add_array([inputs.dir_files])
    }}
  }

Usage:

  group_logs:
    run: make-subdir.cwl
    in:
      dir_files:
        source: [ task1/log, task2/log ]
      dir_name: logs
    out: [ subdir ]
1 Like