Inter steps output dependencies difficulties

CWL Questions
1

I have the following workflow (take geometry, create orbiting camera, render image sequence, create movie from sequence)

My renderScene step depends on output from both step “orbitCamera” and “blenderScene” but when the renderScene step runs, the output from orbitCamera has already been cleared.

I know there are flags in cwl-runner to keep all the tmp files and such but I was hoping that the dependency should tell cwl-runner to keep things around or have I misunderstood the nature of input/output dependencies in CWL ?

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: Workflow
inputs:
  - id: ttcamScript
    type: File
  - id: sceneGenScript
    type: File
  - id: inputAlembicMeshWF
    type: File
  - id: alembicCameraWF
    type: string
outputs:
  - id: animation
    type: File
    outputBinding:
      glob: "*.mp4"
steps:
  orbitCamera:
    run : create-orbiting-camera.cwl
    in:
      blenderScript: ttcamScript
      inputAlembicMesh: inputAlembicMeshWF
      alembicCamera: alembicCameraWF
    out: [outputAlembicCamera]
  blenderScene:
    run : create-render-scene-file.cwl
    in:
      blenderScript: sceneGenScript
      inputAlembicMesh: inputAlembicMeshWF
      alembicCamera: orbitCamera/outputAlembicCamera
    out: [outputBlenderScene]
  renderScene:
    run : render-scene-file.cwl
    in:
      blenderScene: blenderScene/outputBlenderScene
      alembicCamera: orbitCamera/outputAlembicCamera
    out: [renderedImages]
2

Yes, it is absolutely supposed to preserve files between steps. Maybe you are not capturing the outputs properly? Perhaps you could paste the code from your “orbitCamera” tools.

Separately, this is incorrect at the Workflow level:

You need to connect the output from the renderScene step instead:

outputs:
  - id: animation
    type: File
    outputSource: renderScene/renderedImages
3 
#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
requirements:
  - class: InlineJavascriptRequirement
baseCommand: blender_ttcam
arguments: ["--script", $(inputs.blenderScript.path), "--input", $(inputs.inputAlembicMesh.path), "--output", $(inputs.alembicCamera)]
inputs:
  - id: blenderScript
    type: File
  - id: inputAlembicMesh
    type: File
  - id: alembicCamera
    type: string
successCodes:
  - 0
outputs:
  - id: outputAlembicCamera
    type: File
    outputBinding:
      glob: $(inputs.alembicCamera)
4

Are you using cwltool ? Could you paste the log? You could also try cwltool --debug.

5

I am currently using cwl-runner but happy to learn how to run it with cwltool (since I use cwltool to validate the files)

Current way to run

cwl-runner --leave-tmpdir --leave-outputs create-turntable.cwl --ttcamScript ../blender_ttcam_script.py --sceneGenScript ../blender_build_scene_script.py --inputAlembicMeshWF ../monkey.abc --alembicCameraWF nicholas.abc
6

cwl-runner is the generic name, it could be any runner. You might be using cwltool already, just as a different name. What does it say if you do cwl-runner --version ?

7 
$ cwl-runner --version
/data1/nyue/projects/workflow-languages_git/CWL/dev_venv/bin/cwl-runner 3.0.20210124104916
8

Do you want to run your workflow with --debug and paste the log somewhere?

9

I have got past the problem for now.

Thank you for the note on using --debug to get additional log.

Cheers

1 Like