When I am using cwltool to execute a CWL tool and an error occurs, nothing is written to standard output or standard error. I have been able to work around this by commenting out the offending portion of the CWL document and rerunning the tool to debug. Is there a better way to debug? Is there a way to force stdout and stderr to be written even when an error occurs?
Do you mean the stdout or stderr of the tool? That should part of the logging already, there is nothing suppressing the logs. Unless they are being redirected by the tool definition, in which case it should still record output even if the tool failed.
Thanks for your reply. Below is a simple tool that I created. I deliberately added a bug in the add.py script (sys.argv[4] rather than sys.argv[3]).
cwlVersion: v1.2
class: CommandLineTool
label: Adds two numbers
doc: |-
This tool adds two integers and saves the sum to a file.
requirements:
ShellCommandRequirement: {}
InlineJavascriptRequirement: {}
DockerRequirement:
dockerImageId: add_tool
dockerFile: |-
FROM python:3.9-slim-buster
NetworkAccess:
class: NetworkAccess
networkAccess: true
InitialWorkDirRequirement:
listing:
- entryname: add.py
entry: |-
import sys
number1 = int(sys.argv[1])
number2 = int(sys.argv[2])
out_file_path = sys.argv[4]
with open(out_file_path, 'w') as out_file:
total = number1 + number2
out_file.write(str(total))
inputs:
number1:
type: int
doc: |-
An integer
number2:
type: int
doc: |-
A second integer
output_file:
type: string
doc: |-
Name of the output file that will be created.
#Output_File=edam:format_1964
arguments:
- shellQuote: false
valueFrom: |-
# This could be done with a bash command, but we're using Python for consistency with the other tool.
python add.py $(inputs.number1) $(inputs.number2) $(inputs.output_file)
outputs:
output_file:
type: File
outputBinding:
glob: "$(inputs.output_file)"
doc: |-
Output file matching the name specified in the "output_file" input.
format: edam:format_1964
standard_output:
type: stdout
format: edam:format_1964
standard_error:
type: stderr
format: edam:format_1964
stdout: stdout.txt
stderr: stderr.txt
s:author:
- class: s:Person
s:name: Stephen Piccolo
s:identifier: https://orcid.org/0000-0003-2001-5640
s:dateCreated: "2020-07-14"
s:license: https://spdx.org/licenses/Apache-2.0
$namespaces:
s: https://schema.org/
edam: http://edamontology.org/
$schemas:
- https://schema.org/version/latest/schemaorg-current-http.rdf
- http://edamontology.org/EDAM_1.23.owl
Here is my input-object file:
number1: 99
number2: -83
output_file: result.txt
When I run the tool using cwltool, I get the following output:
**INFO** **[job add_tool.cwl] Max memory used: 0MiB**
**ERROR** **[job add_tool.cwl] Job error:**
**("Error collecting output for parameter 'output_file':\nadd_tool.cwl:53:7: Did not find output file with glob pattern: '['result.txt']'", {})**
**WARNING** **[job add_tool.cwl] completed permanentFail**
**{}**
**WARNING** **Final process status is permanentFail**
When this happens, it does not generate stdout.txt or stderr.txt or result.txt. So I cannot see the error message from my Python script unless I comment out the “output_file” output.
Any thoughts? There must be a better way for me to approach this.
I think what you really want is a cwltool feature that logs the stdout/stderr of each step, so that you don’t have to capture the stdout and stderr in the CWL itself. Right now you can capture the stderr of the entire cwltool run (cwltool 2> stderr.txt) but of course that mixes all the logging together which may be hard to read.
When creating this ticket, I was focusing on a tool rather than a workflow. So at a minimum, it would be great if any stdout/stderr files that have been specified are created even on a permanentFail condition. In my example, my Python script throws an error, but I don’t see that error when a permanentFail occurs, even in cwltool’s outputs. So to debug it, I have to remove the cause of the permanentFail (in this case, the fact that the output files were not created) to see the error thrown by my Python script.
But of course, I could be missing something obvious.