Problem with when statements in the workflow with NoneType input for scatter

CWL Questions
1

I have 2 following steps in CWL workflow that are executed conditionally (using when statement) and output from one of them is input for the second one, and second one contains scatter. There is a fragment of code here:

steps:
  list_fastqs_from_dirs_with_ngs_data:
    run: ../shared_tools/collections2files/collections2files.cwl
    when: $(inputs.run_step === true)
    in:
      dirs: dirs_with_ngs_data
      run_step: run_sort_fastq
    out: [files]
  
  sort_fastq:
    run: ./sort_fastq/sort_fastq.cwl
    when: $(inputs.run_sort_fastq === true)
    in:
      fastq_file: list_fastqs_from_dirs_with_ngs_data/files
      run_sort_fastq: run_sort_fastq
    scatter: fastq_file
    out: [sorted_fastq]

When I run the pipeline and run_sort_fastq === true pipeline is completed.
But when run_sort_fastq === false pipeline fails, because scatter is empty:

File "/usr/share/python3/dist/python3-arvados-cwl-runner/lib/python3.7/site-packages/cwltool/workflow_job.py", line 732, in <listcomp>
shortname(s) for s in scatter if len(cast(Sized, inputobj[s])) == 0
TypeError: object of type 'NoneType' has no len()

It should not matter that scatter is empty, because sort_fastq step is not executed.
I fixed the problem, using dummy input in sort_fastq with the same input type than list_fastqs_from_dirs_with_ngs_data/files that is always not empty. The code with solution looks like that:

      fastq_file:
        source: [list_fastqs_from_dirs_with_ngs_data/files, blank_input]
        pickValue: first_non_null

and works.
Is there any better solution to fix this problem?
Thanks

2

Thank you @Monika_Krzyzanowska for sharing your issue. Sounds like it is an error with your CWL runner. I see that you are using Arvados, so I suggest opening an issue with them using your support contract or at https://dev.arvados.org/projects/arvados/issues/new

I would be curious if cwltool has this bug as well.

3

Hi @mrc. Thank you for quick respond. I was also curious if the same problem appears in cwltool and I checked. The problem is the same:

ERROR Unhandled exception
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/lib/python3.9/site-packages/cwltool/workflow_job.py", line 730, in try_make_job
    emptyscatter = [
  File "/usr/local/Caskroom/miniconda/base/lib/python3.9/site-packages/cwltool/workflow_job.py", line 731, in <listcomp>
    shortname(s) for s in scatter if len(cast(Sized, inputobj[s])) == 0
TypeError: object of type 'NoneType' has no len()
4

You are welcome.

Looks like there is a bug in the cwltool side; can you open an issue at Sign in to GitHub ยท GitHub ?

Thanks!