Hi all,
I am in the process of creating a pipeline that runs several steps. For each step the pipeline collects a log file. At the end of the pipeline I have a step (named merge_logs) that collect all the logs and puts the files in a single folder.
My problem is that if some of the steps fail the merge_logs step is not run. Even if I mark the inputs as being potentially null the step is never executed.
Here there is a simplified example that elucidates the problem:
cwlVersion: v1.1
class: Workflow
requirements:
SubworkflowFeatureRequirement: {}
InlineJavascriptRequirement: {}
StepInputExpressionRequirement: {}
MultipleInputFeatureRequirement: {}
inputs: []
outputs:
mmm:
type: Directory
outputSource: merge_output/output
steps:
step1:
in: []
out: [output]
run:
class: CommandLineTool
requirements:
- class: ShellCommandRequirement
arguments:
- shellQuote: false
valueFrom: |
echo "step1" > step1.txt
false
inputs: []
outputs:
output:
type: File?
outputBinding:
glob: "step1.txt"
step2:
in: []
out: [output]
run:
class: CommandLineTool
baseCommand: [echo]
stdout: "step2.txt"
inputs: []
outputs:
output:
type: stdout
arguments:
- valueFrom: "step2"
merge_output:
in:
# in1: step1/output
# in2: step2/output
files:
source:
- step1/output
- step2/output
linkMerge: merge_flattened
out: [output]
run:
class: ExpressionTool
inputs:
# in1: File?
# in1: File?
files:
type:
- type: array
items:
- "null"
- "File"
- "null"
outputs:
output:
type: Directory
expression: |
${
return {
output: {
class: "Directory",
basename: "files",
listing: inputs.files
// listing: [inputs.in1, inputs.in2]
}
};
}
I’ve tried both the uncommented and the commented versions but they work the same.
I’ve tried another route: create a common folder where to place the logs. Here is the workflow:
cwlVersion: v1.1
class: Workflow
requirements:
SubworkflowFeatureRequirement: {}
InlineJavascriptRequirement: {}
StepInputExpressionRequirement: {}
MultipleInputFeatureRequirement: {}
InitialWorkDirRequirement:
listing: |
${
return [
{
class: "Directory",
basename: "logs",
writable: true,
listing: []
}
];
}
inputs: []
outputs:
mmm:
type: Directory
outputSource: merge_output/output
steps:
step1:
in: []
out: [output]
run:
class: CommandLineTool
requirements:
- class: ShellCommandRequirement
arguments:
- shellQuote: false
valueFrom: |
echo "step1" > logs/step1.txt
false
inputs: []
outputs:
output:
type: File?
outputBinding:
glob: "step1.txt"
step2:
in: []
out: [output]
run:
class: CommandLineTool
baseCommand: [echo]
stdout: "step2.txt"
inputs: []
outputs:
output:
type: stdout
arguments:
- valueFrom: "step2"
merge_output:
in: []
out: [output]
run:
class: ExpressionTool
inputs: []
outputs:
output:
type: Directory
expression: |
${
return {
output: {
class: "Directory",
basename: "files",
listing: ???
}
};
}
In this case however I don’t know how to access the logs folder either in the merge_output step (what I put in place of the ??? placeholder?) or in a workflow output?
I like the former approach much better because it doesn’t need to adapt the steps or to make the steps aware of the final folder structure.
Do you have any suggestion?
Thank you very much!