Sub Step 3 really ought to be part of the Scattered Subworkflow so that it can asynchronously handle the outputs of Sub Step 2 without waiting for ALL of Sub Step 2’s scatter jobs to complete first. However, Sub Step 3 also requires an input from the Aux Step.
If I absorb Sub Step 3 into the subworkflow, then the entire subworkflow must wait for the Aux Step to complete before starting. This is because the Aux Step output must be listed as a WorkflowInputParameter to satisfy the WorkflowStepInput of Sub Step 3.
If I remove Sub Step 3 from the subworkflow (as seen above) and instead scatter it on its own, then Aux Step is able to run in parallel with the subworkflow (yay!), but Sub Step 3 must wait for the entire subworkflow to complete before starting (boo).
What I want: to be able to run Sub Step 3 in the subworkflow so that it can handle Sub Step 2’s outputs asynchronously and in parallel, while also allowing Aux Step to complete in parallel while Sub Steps 1 and 2 are executed. Is this possible?
A very clever executor could infer that the result of “aux step” isn’t used until sub step 3, and start running the other steps. Unfortunately, I don’t know of any CWL executors which are currently that clever.
I could envision a CWL hint on a workflow step requests deferred execution, where the step is only executed when its outputs are needed. However this wouldn’t do exactly what you want, because it would schedule the “aux step” when the first scatter got to step 3, instead of running in parallel with sub step 1.
You want something more like a promise, where the computation starts immediately, but the value isn’t delivered until later. That’s also something that could be indicated as a hint, or even build a clever executor around it (in the case of cwltool, this is kind of how it already works within a single workflow graph, but it doesn’t cross over into scatters / nested workflows).
Approaching the problem from a different direction, at least one executor (Arvados) can recognize when two job submissions are the exactly the same, and have them share the same underlying execution. In that situation, you could move “aux step” into the scatter, which would result in the “aux step” being submitted N times, but only actually executed once.
Tetron, thank you as always for your thoughtful response. I’m always happy to see notifications from you because I know I’ll be getting good info and good leads. Glad I wasn’t missing something obvious this time Thank you for the tip on Arvados, I’m excited to check it out
Thank you for the tip on Arvados, I’m excited to check it out