Error trying to create input from array of records with secondary files

steve · February 11, 2021, 12:25am

I am trying to use an input like this;

{
"bams": [
        {
        "bam_file": { "class": "File", "path": "/path/to/sample2.bam" },
        "sample_id": "Sample24"
        },
        {
        "bam_file": { "class": "File", "path": "/path/to/sample.bam" },
        "sample_id": "Sample23"
        }
    ]
}

The input to my CWL looks like this;

inputs:
  bams:
    type:
      type: array
      items:
        type: record
        fields:
          bam_file: File
            secondaryFiles:
              - ^.bai
          sample_id: string

But I keep getting errors like this;

ERROR Tool definition failed validation:
cwl/fillout_workflow.cwl:26:27: mapping values are not allowed here

It seems like its the secondaryFiles part is causing it to break, but its required for the tool I am using.

Any ideas?

mrc · February 11, 2021, 2:04pm

inputs:
  bams:
    type:
      type: array
      items:
        type: record
        fields:
          bam_file:
            type: File
            secondaryFiles:
              - ^.bai
          sample_id: string

From Common Workflow Language (CWL) Command Line Tool Description, v1.2 ; if you need to do more than specify the type is File then you need to use the expanded form

steve · February 11, 2021, 10:49pm

I tried that but it did not work either;

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: Workflow
requirements:
  MultipleInputFeatureRequirement: {}
  ScatterFeatureRequirement: {}
  StepInputExpressionRequirement: {}
  InlineJavascriptRequirement: {}
  SubworkflowFeatureRequirement: {}

inputs:
  maf_file:
    type: File
  bams:
    type:
      type: array
      items:
        type: record
        fields:
          bam_file:
            type: File
            secondaryFiles:
              - ^.bai
          sample_id: string

I get error messages that look like this;

ERROR Tool definition failed validation:
cwl/fillout_workflow.cwl:3:1:     Object `cwl/fillout_workflow.cwl` is not valid because
                                  tried `Workflow` but
cwl/fillout_workflow.cwl:15:1:       the `inputs` field is not valid because
cwl/fillout_workflow.cwl:19:3:         item is invalid because
cwl/fillout_workflow.cwl:20:5:           the `type` field is not valid because
                                          - tried InputRecordSchema but
cwl/fillout_workflow.cwl:21:7:               * the `type` field is not valid because
                                              the value 'array' is not a valid Record_symbol,
                                              expected 'record'
cwl/fillout_workflow.cwl:22:7:               * invalid field `items`, expected one of: 'fields',
                                            'type', 'label'
cwl/fillout_workflow.cwl:20:5:             - tried InputEnumSchema but
                                            * missing required field `symbols`
cwl/fillout_workflow.cwl:21:7:               * the `type` field is not valid because
                                              the value 'array' is not a valid Enum_symbol,
                                              expected 'enum'
cwl/fillout_workflow.cwl:22:7:               * invalid field `items`, expected one of: 'symbols',
                                            'type', 'label', 'inputBinding'
cwl/fillout_workflow.cwl:20:5:             - tried InputArraySchema but
cwl/fillout_workflow.cwl:22:7:               the `items` field is not valid because
                                              - tried InputRecordSchema but
cwl/fillout_workflow.cwl:24:9:                   the `fields` field is not valid because
                                                  tried array of <InputRecordField> but
cwl/fillout_workflow.cwl:25:11:                     item is invalid because
cwl/fillout_workflow.cwl:27:13:                       invalid field `secondaryFiles`, expected one
                                                      of: 'name', 'doc', 'type', 'inputBinding', 'label'
cwl/fillout_workflow.cwl:22:7:                 - tried InputEnumSchema but
                                                * missing required field `symbols`
cwl/fillout_workflow.cwl:23:9:                   * the `type` field is not valid because
                                                  the value 'record' is not a valid Enum_symbol,
                                                  expected 'enum'
cwl/fillout_workflow.cwl:24:9:                   * invalid field `fields`, expected one of:
                                                'symbols', 'type', 'label', 'inputBinding'
cwl/fillout_workflow.cwl:22:7:                 - tried InputArraySchema but
                                                * missing required field `items`
cwl/fillout_workflow.cwl:23:9:                   * the `type` field is not valid because
                                                  the value 'record' is not a valid Array_symbol,
                                                  expected 'array'
cwl/fillout_workflow.cwl:24:9:                   * invalid field `fields`, expected one of: 'items',
                                                'type', 'label', 'inputBinding'

steve · February 12, 2021, 12:14am

Looks like I needed to use cwlVersion: v1.1 instead to get it to work, thanks

steve · February 19, 2021, 6:00pm

Now, how do I set it up if I want an array of files with secondaryFiles?

I tried this;

inputs:
  vcf_gz_files:
    type:
      type: array
      items:
        type: File
        secondaryFiles:
          - ^.bai
  bam_files:
    type:
      type: array
      items:
        type: File
        secondaryFiles:
          - ^.bai

But get these errors

ERROR Tool definition failed validation:
cwl/igv-reports.cwl:16:1: Object `cwl/igv-reports.cwl` is not valid because
                            tried `CommandLineTool` but
cwl/igv-reports.cwl:37:1:     the `inputs` field is not valid because
cwl/igv-reports.cwl:45:3:       item is invalid because
cwl/igv-reports.cwl:46:5:         the `type` field is not valid because
                                    - tried CommandInputRecordSchema but
cwl/igv-reports.cwl:47:7:             * the `type` field is not valid because
                                        the value 'array' is not a valid Record_symbol, expected
                                        'record'
cwl/igv-reports.cwl:48:7:             * invalid field `items`, expected one of: 'fields', 'type',
                                      'label'
cwl/igv-reports.cwl:46:5:           - tried CommandInputEnumSchema but
                                      * missing required field `symbols`
cwl/igv-reports.cwl:47:7:             * the `type` field is not valid because
                                        the value 'array' is not a valid Enum_symbol, expected
                                        'enum'
cwl/igv-reports.cwl:48:7:             * invalid field `items`, expected one of: 'symbols', 'type',
                                      'label', 'inputBinding'
cwl/igv-reports.cwl:46:5:           - tried CommandInputArraySchema but
cwl/igv-reports.cwl:48:7:             the `items` field is not valid because
                                        - tried CommandInputRecordSchema but
cwl/igv-reports.cwl:49:9:                 * the `type` field is not valid because
                                            the value 'File' is not a valid Record_symbol, expected
                                            'record'
cwl/igv-reports.cwl:50:9:                 * invalid field `secondaryFiles`, expected one of:
                                          'fields', 'type', 'label'
cwl/igv-reports.cwl:48:7:               - tried CommandInputEnumSchema but
                                          * missing required field `symbols`
cwl/igv-reports.cwl:49:9:                 * the `type` field is not valid because
                                            the value 'File' is not a valid Enum_symbol, expected
                                            'enum'
cwl/igv-reports.cwl:50:9:                 * invalid field `secondaryFiles`, expected one of:
                                          'symbols', 'type', 'label', 'inputBinding'
cwl/igv-reports.cwl:48:7:               - tried CommandInputArraySchema but
                                          * missing required field `items`
cwl/igv-reports.cwl:49:9:                 * the `type` field is not valid because
                                            the value 'File' is not a valid Array_symbol, expected
                                            'array'
cwl/igv-reports.cwl:50:9:                 * invalid field `secondaryFiles`, expected one of:
                                          'items', 'type', 'label', 'inputBinding'

tried with both cwlVersion: v1.0 and cwlVersion: v1.1

mrc · March 5, 2021, 2:31pm

Try This

inputs:
  vcf_gz_files:
    type:
      type: array
      items:
        type: File
    secondaryFiles:
      - ^.bai
  bam_files:
    type:
      type: array
      items:
        type: File
    secondaryFiles:
      - ^.bai

Or more succinctly

inputs:
  vcf_gz_files:
    type: File[]
    secondaryFiles:
      - ^.bai
  bam_files:
    type:
      type: File[]
    secondaryFiles:
      - ^.bai

BJWiley233 · July 12, 2021, 4:38pm

I am getting similar error:

`type` field is not valid because: Expected one of ('record',) 
invalid field
`items`, expected one of: `fields`,
`type`, `label`, `doc`, `name`,
`inputBinding`

With input:

custom_annotations:
        type:
            - "null"
            - type: array
              items: ../types/vep_custom_annotation.yml#vep_custom_annotation
              label: "custom type, check types directory for input format"
              inputBinding:
                  valueFrom: |
                       ${
                           return [self.annotation.check_existing ? '--check_existing' : '',
                             '--custom',
                             [self.annotation.file.path,
                             self.annotation.name,
                             self.annotation.data_format,
                             self.method,
                             self.force_report_coordinates ? 1 : 0,
                             self.annotation.vcf_fields ? self.annotation.vcf_fields : ''
                             ].filter(String).join(',')
                           ].filter(String)
                       }
                  position: 6