Tee for stdout/stderr capture

CWL Questions
1

Hello friends,

I’ve been using the stdout and stderr fields to produce log files from CommandLineTools and to keep the terminal output clean when running workflows.

I have experienced two drawbacks with their use:

  1. Sometimes I’d like to redirect to a log file while still printing the output to the terminal, particularly for errors communicated via stderr
  2. If a workflow step with captured output fails, and the log file is an input for a downstream step (for example, a step which organizes outputs into subdirectories), then the log file is lost along with all of the helpful information it contained

In both of these scenarios a user might be left with a workflow error and no explanation or means of troubleshooting the issue. I realize that I could pass --leave-tmpdir to cwltool so that the log files aren’t discarded on error, but this isn’t always a realistic option:

  • The workflow might produce huge volumes of intermediate data so it would be inconsiderate to the user to maintain --leave-tmpdir just incase there’s an error
  • The failing step might follow a time-intensive step that would make it inconvenient to restart a workflow with --leave-tmpdir

I believe having behavior similar to Unix’s tee command, where output is both redirected and printed to terminal, would solve these issues. As far as I can tell this isn’t possible with the cwltool runner.

Has anyone else encountered these issues and found a good solution?

Best,
Alex

1 Like
2

I have proposed the addition of stdout-tee and stderr-tee to the CommandLineTool specification on Matrix