Validation ERROR: "While scanning a simple key could not find expected ':'", inside valueFrom field

CWL Questions
1

The context for this problem is that I am trying to create an input record for one step in a workflow. Depending on how certain parameters for the workflow were set, this record should either pull values from a prior step or (if the prior step wasn’t run) pull the value from the user-input parameters in the yml for the workflow. For my non-record inputs this is fairly straightforward to work with, as the following works to define the input to the variable:

      round_offset:
        source: [stagedSorted/round_offset, round_offset]
        pickValue: first_non_null

However, it does not seem to be possible to generate a record as output from the stagedSorted step without setting up a custom datatype (which I would then have to use everywhere this record is used). Because this same pickValue doesn’t work when one item is a complete record and the other needs to be generated with an expression, I used the following valueFrom, thinking it would let me create the appropriate record in-line.

      aux_tilesets:
        source: [aux_tilesets, stagedSorted/aux_names, stagedSorted/aux_file_formats, stagedSorted/aux_file_vars, stagedSorted/aux_cache_read_order, channel_count, stagedSorted/aux_channel_slope, stagedSorted/aux_channel_intercept]
        linkMerge: merge_flattened
        valueFrom: |
        ${
          if(!self[1]){
            return {
                aux_names: self[0].aux_names,
                aux_file_formats: self[0].aux_file_formats,
                aux_file_vars: self[0].aux_file_vars,
                aux_cache_read_order: self[0].aux_cache_read_order,
                aux_channel_count: self[0].aux_channel_count,
                aux_channel_slope: self[0].aux_channel_slope,
                aux_channel_intercept: self[0].aux_channel_intercept
            };
          } else {
            return {
                aux_names: self[1],
                aux_file_formats: self[2],
                aux_file_vars: self[3],
                aux_cache_read_order: self[4],
                aux_channel_count: self[5],
                aux_channel_slope: self[6],
                aux_channel_intercept: self[7]
            };
          };
        }

This looks similar to other times when I have manipulated records with expressions, but this time I get the following error when I try to validate the cwl:

ERROR Tool definition failed validation:
pipeline.cwl:409:9:  while scanning a simple key
pipeline.cwl:412:26:   could not find expected ':'

(The line numbers refer to the first return in the if statement)

The only things that are different this time than the other times I’ve made a record with an expression are the linkMerge and the if statement-- is what I’m trying to do even possible, or is there some other way I could get the same result?

2

I think you need to indent the ${ by a couple of spaces.

1 Like
3

I was getting the same error; then I indented the ${ by a couple of spaces and now it gives me other error as follows:

ERROR Tool definition failed validation:
Try_wf/one2.cwl:21:7: while parsing a block mapping
Try_wf/one2.cwl:27:7:   expected <block end>, but found '}'

My Workflow is as follows:

cwlVersion: v1.2
class: Workflow

requirements:
 InlineJavascriptRequirement: {}

inputs: 
  xfile: string
  msg: int

steps:
  listdir:
    run: 'one.cwl'
    in:
      num: xfile
    out: [a]

  filter_files:
    run: 'two.cwl'
    in:
      num: msg
      infile: |
        ${ 
          if (inputs.num === 0) { 
            return [{ listdir/a }]; }
          else { return [ null] ; } 
      }
      
    out: [ ab ]

outputs:
  abcd:
    type: File
    outputSource: filter_files/ab

Can we use the if else statement like this in CWL for deciding the inputs? If yes, where am I doing something wrong causing this error?

4

The trailing } also needs to be intended

      infile: |
        ${ 
          if (inputs.num === 0) { 
            return [{ listdir/a }]; }
          else { return [ null] ; } 
        }
5

What’s going on here is that this a YAML text-block, when you have a field that starts with | followed by one or more lines of intended text, that is a multi-line string, the text block ends when there is a line which is indented less than the first line of the text block.

So whenever you see foo: | every line following it needs to be intended by at least one more column than the foo: field.

1 Like
6

After trying the solution, it says

ERROR Tool definition failed validation:
Try_wf/one2.cwl:12:1: checking field `steps`
Try_wf/one2.cwl:19:3:   checking object `Try_wf/one2.cwl#filter_files`
Try_wf/one2.cwl:21:5:     checking field `in`
Try_wf/one2.cwl:23:7:       checking object `Try_wf/one2.cwl#filter_files/infile`
                              Field `source` references unknown identifier `${

I don’t understand why?

7

Oh, I’m sorry, I wasn’t looking closely enough at the actual javascript expression.

You want to do something like this (assuming that “infile” needs to take an array of files):

      infile:
        source: listdir/a
        valueFrom: |
          ${ 
            if (inputs.num === 0) { 
              return [self]; 
            }
            else { return [null]; } 
          }

(edit: messed up the indentation myself)

8

What this is doing is taking the result from listdir/a and then evaluating an expression in valueFrom. In the expression the value from source is assigned to self, other values are assigned to inputs.

1 Like
9

After indenting it correctly it says:

ERROR Exception on step 'filter_files'
ERROR [step filter_files] Cannot make job: Invalid job input record:
the `infile` field is not valid because
  item is invalid because
    is not a dict

And one more question - can we add multiple sources here? If yes is it similar to the optional type?

      infile:
        source: [listdir/a, xyz/x]
        valueFrom: |
          ${ 
            if (inputs.num === 0) { 
              return [listdir/a]; 
            }
            else { return [xyz/x]; } 
          }

Is this the correct way of implementing it?

10

Yes, you can depend on multiple upstream values.

It’ll turn self into an array.

The “item is invalid because it is not a dict” error means infile is a file and not an array, so you don’t want to wrap it in [ ].

Try this:

      infile:
        source: [listdir/a, xyz/x]
        valueFrom: |
          ${ 
            if (inputs.num === 0) { 
              return self[0]; 
            }
            else { return self[1]; } 
          }
1 Like
11

Thank you so much, it solved my issue.
Actually, it is the sample script for the workaround for loop in my workflow :slight_smile: