Viewing logs in real time

CWL Questions
1

Hey.

For container based tools, I notice that I need to capture stdout/stderr as outputs.

For example, my tools fit this similar pattern:

cwlVersion: v1.2
class: CommandLineTool
baseCommand: [echo]
requirements:
  DockerRequirement:
    dockerPull: busybox
id: "echo"
inputs: 
  message: 
    type: string
    inputBinding:
      position: 1
outputs: 
  echoOut:
    type: stdout

I notice that once a workflow finishes (in both cwltool and toil-cwl-runner), the outputs get written out in stdout. I am able to parse stdout of the workflow run to get the log files.

Is it possible to view logs in real time as a workflow runs? We are looking to support long running simulations and this is a requirement for our users.

2

Is the stdout / stderr being written to a file in real time? If so, you could use a standard ELK stack to find the log files being written and read from them. Linking them back to a running pipeline instance might be tricky though