CWL draft-3 document upgrade

Hi everyone
I am new to CWL, I am trying to run the EBI-mgnify pipeline5 workflow with the example data provided in the GitHub.
I have installed the cwltool and I ran the command “cwltool raw-reads–v.5-cond1.yml raw-reads-wf–v.5-cond.cwl”
It showed Error as follows
INFO /home/plankton/anaconda3/bin/cwltool 3.1
INFO Resolved ‘raw-reads–v.5-cond1.yml’ to ‘file:///home/plankton/pipeline-v5-master/workflows/raw-reads–v.5-cond1.yml’
ERROR Tool definition failed validation: No cwlVersion found. Use the following syntax in your CWL document to declare the version: cwlVersion: .
Note: if this is a CWL draft-3 (pre v1.0) document then it will need to be upgraded first using cwl-upgrader · PyPI . ‘sbg:draft-2’ documents can be upgraded using sevenbridges-cwl-draft2-upgrader · PyPI.

I have tried to update the document but it’s not working.
Please help me in this regard

Thank you for your valuable time and suggestion

Thank you

1 Like

Hi @balamurugan_Sadaiapp ,

I didn’t know what workflow you were talking about, but after some searching on GitHub, I found this one: pipeline-v5/raw-reads-wf--v.5-cond.cwl at 981aafc0df341a2536ed5059695bb7c33af4d3ee · EBI-Metagenomics/pipeline-v5 · GitHub

Using the development version of cwltool to test it with cwltool raw-reads-wf--v.5-cond.cwl resulted in the following error.

INFO /home/kinow/Development/python/workspace/cwltool/venv/bin/cwltool 3.1.20220428075913
INFO Resolved 'raw-reads-wf--v.5-cond.cwl' to 'file:///tmp/pipeline-v5/workflows/raw-reads-wf--v.5-cond.cwl'
Cache entry deserialization failed, entry ignored
ERROR Tool definition failed validation:
Version 'v1.2.0-dev2' is a development or deprecated version.
 Update your document to a stable version (v1.0, v1.1, v1.2) or use --enable-dev to enable support for development and deprecated versions.

That CWL uses a development version in cwlVersion. So after enabling --enable-dev it started running the workflow with apparently no error (left it running while writing this, but stopped the process as I didn’t know how long it’d take).

cwltool --enable-dev raw-reads-wf--v.5-cond.cwl 
INFO /home/kinow/Development/python/workspace/cwltool/venv/bin/cwltool 3.1.20220428075913
INFO Resolved 'raw-reads-wf--v.5-cond.cwl' to 'file:///tmp/pipeline-v5/workflows/raw-reads-wf--v.5-cond.cwl'
...
...

Give it a try and see if the --enable-dev works. If not, share where others can find the workflow you are having issues to run, and also what’s your cwltool version (I think it’s 3.1? But the cwltool --version may contain more details).

Cheers
Bruno

Hi Bruno
Thank you for your suggestion, with --enable-dev, it shows following errors
cwltool --enable-dev raw-reads-wf–v.5-cond.cwl

INFO /home/plankton/anaconda3/bin/cwltool 3.1
INFO Resolved ‘raw-reads-wf–v.5-cond.cwl’ to ‘file:///home/plankton/pipeline-v5-master/workflows/raw-reads-wf–v.5-cond.cwl’
WARNING conditionals/raw-reads/raw-reads-2.cwl:242:9: JSHINT: (function(){return (( self.filter(file => !!file.basename.match(/^..faa.$/)).pop() ));})()
conditionals/raw-reads/raw-reads-2.cwl:242:9: JSHINT: ^
conditionals/raw-reads/raw-reads-2.cwl:242:9: JSHINT: W119: ‘arrow function syntax (=>)’ is only available in ES6. CWL only supports ES5.1
WARNING conditionals/raw-reads/raw-reads-2.cwl:239:9: JSHINT: (function(){return (( self.filter(file => !!file.basename.match(/^..ffn.$/)).pop() ));})()
conditionals/raw-reads/raw-reads-2.cwl:239:9: JSHINT: ^
conditionals/raw-reads/raw-reads-2.cwl:239:9: JSHINT: W119: ‘arrow function syntax (=>)’ is only available in ES6. CWL only supports ES5.1
WARNING conditionals/raw-reads/raw-reads-2.cwl:195:10: JSHINT: (function(){return (( self.filter(file => !!file.basename.match(/^..faa.$/)).pop() ));})()
conditionals/raw-reads/raw-reads-2.cwl:195:10: JSHINT: ^
conditionals/raw-reads/raw-reads-2.cwl:195:10: JSHINT: W119: ‘arrow function syntax (=>)’ is only available in ES6. CWL only supports ES5.1
WARNING Workflow checker warning:
subworkflows/final_chunking.cwl:24:5: Source ‘LSU’ of type [“null”, “File”] may be incompatible
subworkflows/final_chunking.cwl:72:7: with sink ‘infile’ of type {“type”: “array”, “items”:
“File”}
source has linkMerge method merge_nested
subworkflows/final_chunking.cwl:25:5: Source ‘SSU’ of type [“null”, “File”] may be incompatible
subworkflows/final_chunking.cwl:72:7: with sink ‘infile’ of type {“type”: “array”, “items”:
“File”}
source has linkMerge method merge_nested
WARNING …/utils/return_directory.cwl:28:1: JSHINT: for (const item in process) {
…/utils/return_directory.cwl:28:1: JSHINT: ^
…/utils/return_directory.cwl:28:1: JSHINT: W104: ‘const’ is available in ES. CWL only supports ES5.1
WARNING …/utils/return_directory.cwl:28:1: JSHINT: for (const item in process) {
…/utils/return_directory.cwl:28:1: JSHINT: ^
…/utils/return_directory.cwl:28:1: JSHINT: W104: ‘const’ is available in ES. CWL only supports ES5.1
WARNING …/utils/return_directory.cwl:28:1: JSHINT: for (const item in process) {
…/utils/return_directory.cwl:28:1: JSHINT: ^
…/utils/return_directory.cwl:28:1: JSHINT: W104: ‘const’ is available in ES. CWL only supports ES5.1
Could not load extension schema https://schema.org/version/latest/schema.rdf: Error fetching https://schema.org/version/latest/schema.rdf: 404 Client Error: Not Found for url: https://schema.org/version/latest/schema.rdf
Warning: Field $schemas contains undefined reference to https://schema.org/version/latest/schema.rdf
WARNING …/utils/return_directory.cwl:28:1: JSHINT: for (const item in process) {
…/utils/return_directory.cwl:28:1: JSHINT: ^
…/utils/return_directory.cwl:28:1: JSHINT: W104: ‘const’ is available in ES. CWL only supports ES5.1
WARNING Workflow checker warning:
subworkflows/classify-otu-visualise.cwl:86:12: Source ‘result’ of type “File” may be incompatible
subworkflows/classify-otu-visualise.cwl:104:7: with sink ‘file_list’ of type [“null”, {“type”:
“array”, “items”: [“null”, “File”]}]
source has linkMerge method merge_nested
Source is from conditional step, but pickValue is
not used
subworkflows/classify-otu-visualise.cwl:96:12: Source ‘result’ of type “File” may be incompatible
subworkflows/classify-otu-visualise.cwl:104:7: with sink ‘file_list’ of type [“null”, {“type”:
“array”, “items”: [“null”, “File”]}]
source has linkMerge method merge_nested
Source is from conditional step, but pickValue is
not used
subworkflows/classify-otu-visualise.cwl:17:5: Source ‘mapseq_ref’ of type “File” may be
incompatible
subworkflows/classify-otu-visualise.cwl:43:7: with sink ‘database’ of type “File”
Could not load extension schema https://schema.org/version/latest/schema.rdf: Error fetching https://schema.org/version/latest/schema.rdf: 404 Client Error: Not Found for url: https://schema.org/version/latest/schema.rdf
Warning: Field $schemas contains undefined reference to https://schema.org/version/latest/schema.rdf
WARNING …/utils/return_directory.cwl:28:1: JSHINT: for (const item in process) {
…/utils/return_directory.cwl:28:1: JSHINT: ^
…/utils/return_directory.cwl:28:1: JSHINT: W104: ‘const’ is available in ES. CWL only supports ES5.1
WARNING Workflow checker warning:
subworkflows/classify-otu-visualise.cwl:17:5: Source ‘mapseq_ref’ of type “File” may be
incompatible
subworkflows/classify-otu-visualise.cwl:43:7: with sink ‘database’ of type “File”
subworkflows/classify-otu-visualise.cwl:86:12: Source ‘result’ of type “File” may be incompatible
subworkflows/classify-otu-visualise.cwl:104:7: with sink ‘file_list’ of type [“null”, {“type”:
“array”, “items”: [“null”, “File”]}]
source has linkMerge method merge_nested
Source is from conditional step, but pickValue is
not used
subworkflows/classify-otu-visualise.cwl:96:12: Source ‘result’ of type “File” may be incompatible
subworkflows/classify-otu-visualise.cwl:104:7: with sink ‘file_list’ of type [“null”, {“type”:
“array”, “items”: [“null”, “File”]}]
source has linkMerge method merge_nested
Source is from conditional step, but pickValue is
not used
WARNING Workflow checker warning:
subworkflows/rna_prediction-sub-wf.cwl:179:21: Source ‘number_lines_mapseq’ of type [“null”, “int”]
may be incompatible
subworkflows/rna_prediction-sub-wf.cwl:78:5: with sink ‘number_LSU_mapseq’ of type “int”
Source is from conditional step and may produce
null
subworkflows/rna_prediction-sub-wf.cwl:179:21: Source ‘number_lines_mapseq’ of type [“null”, “int”]
may be incompatible
subworkflows/rna_prediction-sub-wf.cwl:78:5: with sink ‘number_LSU_mapseq’ of type “int”
subworkflows/rna_prediction-sub-wf.cwl:157:21: Source ‘number_lines_mapseq’ of type [“null”, “int”]
may be incompatible
subworkflows/rna_prediction-sub-wf.cwl:81:5: with sink ‘number_SSU_mapseq’ of type “int”
Source is from conditional step and may produce
null
subworkflows/rna_prediction-sub-wf.cwl:157:21: Source ‘number_lines_mapseq’ of type [“null”, “int”]
may be incompatible
subworkflows/rna_prediction-sub-wf.cwl:81:5: with sink ‘number_SSU_mapseq’ of type “int”
WARNING Workflow checker warning:
conditionals/raw-reads/raw-reads-2.cwl:246:9: Source ‘nucleotide_fasta_chunks’ of type [“null”,
{“type”: “array”, “items”: “File”}] may be
incompatible
conditionals/raw-reads/raw-reads-2.cwl:77:5: with sink ‘chunking_nucleotides’ of type {“type”:
“array”, “items”: “File”}
conditionals/raw-reads/raw-reads-2.cwl:247:9: Source ‘protein_fasta_chunks’ of type [“null”,
{“type”: “array”, “items”: “File”}] may be
incompatible
conditionals/raw-reads/raw-reads-2.cwl:80:5: with sink ‘chunking_proteins’ of type {“type”:
“array”, “items”: “File”}
WARNING Workflow checker warning:
subworkflows/seqprep-subwf.cwl:13:5: Source ‘forward_reads’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:47:7: with sink ‘sequences’ of type “File”
subworkflows/seqprep-subwf.cwl:96:12: Source ‘unzipped_file’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:103:7: with sink ‘sequences’ of type “File”
Source is from conditional step and may produce null
subworkflows/seqprep-subwf.cwl:96:12: Source ‘unzipped_file’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:103:7: with sink ‘sequences’ of type “File”
subworkflows/seqprep-subwf.cwl:76:12: Source ‘merged_reads’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:83:7: with sink ‘target_reads’ of type “File”
Source is from conditional step and may produce null
subworkflows/seqprep-subwf.cwl:76:12: Source ‘merged_reads’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:83:7: with sink ‘target_reads’ of type “File”
subworkflows/seqprep-subwf.cwl:15:5: Source ‘single_reads’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:94:7: with sink ‘target_reads’ of type “File”
subworkflows/seqprep-subwf.cwl:13:5: Source ‘forward_reads’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:57:7: with sink ‘fastq1’ of type “File”
subworkflows/seqprep-subwf.cwl:96:12: Source ‘unzipped_file’ of type [“null”, “File”] may be
incompatible
subworkflows/seqprep-subwf.cwl:20:5: with sink ‘unzipped_single_reads’ of type “File”
pickValue is: first_non_null
WARNING Workflow checker warning:
conditionals/raw-reads/raw-reads-1.cwl:68:12: Source ‘hashsum’ of type [“null”, {“type”: “array”,
“items”: “File”}] may be incompatible
conditionals/raw-reads/raw-reads-1.cwl:28:5: with sink ‘input_files_hashsum_paired’ of type
[“null”, {“type”: “array”, “items”: “File”}]
pickValue is used but only a single input source is
declared
WARNING Workflow checker warning:

raw-reads-wf–v.5-cond.cwl:219:7: Source is from conditional step and may produce null

raw-reads-wf–v.5-cond.cwl:211:7: Source is from conditional step and may produce null
raw-reads-wf–v.5-cond.cwl:198:9: Source ‘compressed_files’ of type [“null”, {“type”: “array”,
“items”: “File”}] may be incompatible
raw-reads-wf–v.5-cond.cwl:102:5: with sink ‘compressed_files’ of type {“type”: “array”,
“items”: “File”}
pickValue is used but only a single input source is declared
raw-reads-wf–v.5-cond.cwl:198:9: Source ‘compressed_files’ of type [“null”, {“type”: “array”,
“items”: “File”}] may be incompatible
raw-reads-wf–v.5-cond.cwl:102:5: with sink ‘compressed_files’ of type {“type”: “array”,
“items”: “File”}
Source is from conditional step and may produce null
raw-reads-wf–v.5-cond.cwl:198:9: Source ‘compressed_files’ of type [“null”, {“type”: “array”,
“items”: “File”}] may be incompatible
raw-reads-wf–v.5-cond.cwl:102:5: with sink ‘compressed_files’ of type {“type”: “array”,
“items”: “File”}
pickValue is: all_non_null
usage: raw-reads-wf–v.5-cond.cwl [-h] --5.8s_pattern 5.8S_PATTERN
–5s_pattern 5S_PATTERN --CGC_config
CGC_CONFIG --CGC_postfixes CGC_POSTFIXES
[–EggNOG_data_dir EGGNOG_DATA_DIR]
[–EggNOG_db EGGNOG_DB]
[–EggNOG_diamond_db EGGNOG_DIAMOND_DB]
–HMM_gathering_bit_score
–HMM_name_database HMM_NAME_DATABASE
–HMM_omit_alignment
–InterProScan_applications
INTERPROSCAN_APPLICATIONS
–InterProScan_databases
INTERPROSCAN_DATABASES
–InterProScan_outputFormat
INTERPROSCAN_OUTPUTFORMAT --cgc_chunk_size
CGC_CHUNK_SIZE
[–forward_reads FORWARD_READS]
–func_ann_names_hmmer FUNC_ANN_NAMES_HMMER
–func_ann_names_ips FUNC_ANN_NAMES_IPS
–go_config GO_CONFIG --hmmsearch_header
HMMSEARCH_HEADER --ips_header IPS_HEADER
–ko_file KO_FILE --lsu_db LSU_DB
–lsu_label LSU_LABEL --lsu_otus LSU_OTUS
–lsu_tax LSU_TAX --other_ncRNA_models
OTHER_NCRNA_MODELS --protein_chunk_size_IPS
PROTEIN_CHUNK_SIZE_IPS
–protein_chunk_size_hmm
PROTEIN_CHUNK_SIZE_HMM --qc_min_length
QC_MIN_LENGTH
[–reverse_reads REVERSE_READS]
–rfam_model_clans RFAM_MODEL_CLANS
–rfam_models RFAM_MODELS
[–single_reads SINGLE_READS] --ssu_db
SSU_DB --ssu_label SSU_LABEL --ssu_otus
SSU_OTUS --ssu_tax SSU_TAX
[job_order]
raw-reads-wf–v.5-cond.cwl: error: the following arguments are required: --5.8s_pattern, --5s_pattern, --CGC_config, --CGC_postfixes, --HMM_gathering_bit_score, --HMM_name_database, --HMM_omit_alignment, --InterProScan_applications, --InterProScan_databases, --InterProScan_outputFormat, --cgc_chunk_size, --func_ann_names_hmmer, --func_ann_names_ips, --go_config, --hmmsearch_header, --ips_header, --ko_file, --lsu_db, --lsu_label, --lsu_otus, --lsu_tax, --other_ncRNA_models, --protein_chunk_size_IPS, --protein_chunk_size_hmm, --qc_min_length, --rfam_model_clans, --rfam_models, --ssu_db, --ssu_label, --ssu_otus, --ssu_tax.

Help me in this regard
Thank you

@balamurugan_Sadaiapp Can you share the output of conda list -f cwltool ? (you may need to add -n name_of_env or similar)

I would report those warnings to the authors of that workflow. However, they don’t appear to affect execution; you should need to add the inputs to the workflow in a json or yaml file. See the inp-job.yml example at Essential Input Parameters – Common Workflow Language User Guide, but use the names of the inputs listed at https://github.com/EBI-Metagenomics/pipeline-v5/blob/981aafc0df341a2536ed5059695bb7c33af4d3ee/workflows/raw-reads-wf--v.5-cond.cwl#L15