Conformance tests are failing for a certain CWL runner

CWL Questions
1

Hi,
I am Afi.
I went through this conformance_test.md

I got this(below) after running the ./run_test.sh

Beginning lines

--- Running CWL Conformance Tests v1.2 on /Users/afimaamedufie/.pyenv/shims/cwl-runner ---
/Users/afimaamedufie/.pyenv/versions/3.10.2/bin/cwl-runner 3.1.20221018083734
URI prefix 'tests/colon' of 'tests/colon:test:job.yaml' not recognized, are you missing a $namespaces section?
URI prefix 'tests/colon' of 'tests/colon:test.cwl' not recognized, are you missing a $namespaces section?
Test [1/366] General test of command line generation
Test [2/366] Test nested prefixes with arrays
Test [3/366] Test nested command line bindings

End Lines

Test [363/366] stdout_chained_commands: Test that chaining two echo calls causes the workflow tool to emit the output to stdout. This is to confirm that the workflow tool will **not** create an expression such as `echo a && echo b > out.txt`, but instead will produce the correct `echo a && echo b`, and capture the output correctly.
Test [364/366] multiple-input-feature-requirement: MultipleInputFeatureRequirement on workflow outputs.
Test [365/366] js-input-record: A test case for JS with an input record.
Test [366/366] schemadef_types_with_import: Test SchemaDefRequirement with a workflow, with the `$import` under types. It is similar to schemadef-wf, but the `$import` is different.
361 tests passed, 5 failures, 0 unsupported features

1 tool tests failed

Any help?

2

Hello @AfiMaameDufie

There should be a specific error message showing the failed test. Did you modify the conformance tests at all?

Your other GitHub comment mentioned you are running MacOS. Which version? And what is the output of docker version?

Finally, what is your version of cwltest? pip show cwltest

3

Hi @mrc

I did not modify any of the tests, I was following the guide and reached the ./run_test.sh part

MacOS Version: Moneterey (12.6)

(cwltest_env) afimaamedufie@Afis-MacBook-Pro cwl % pip show cwltest
Name: cwltest
Version: 2.2.20220521103021
Summary: Common Workflow Language testing framework
Home-page: https://github.com/common-workflow-language/cwltest
Author: Common workflow language working group
Author-email: common-workflow-language@googlegroups.com
License: Apache 2.0
Location: /Users/afimaamedufie/cwl/cwltest_env/lib/python3.10/site-packages
Requires: defusedxml, junit-xml, schema-salad

Docker

Client:
 Cloud integration: v1.0.29
 Version:           20.10.17
4

Thanks. Okay, all of that is recent enough.

There should be a specific error message showing the failed test within the output of ./run_test.sh. If you can share that, then I can see if I can reproduce the error.

5

Oh okay.

From the top :

Test [182/367] Test default value on step input parameter, no ExpressionTool
Test 182 failed: /Users/afimaamedufie/.pyenv/shims/cwl-runner --outdir=/var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp0noyp1vc --quiet tests/count-lines9-wf-noET.cwl tests/empty.json
Test default value on step input parameter, no ExpressionTool
Compare failure expected: {
    "wc_output": {
        "checksum": "sha1$3596ea087bfdaf52380eae441077572ed289d657",
        "class": "File",
        "size": 3
    }
}
got: {
    "wc_output": {
        "basename": "output",
        "checksum": "sha1$9aefd466231cf30b4e9ae5234d58d192af7bcac4",
        "class": "File",
        "location": "file:///var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp0noyp1vc/output",
        "path": "/var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp0noyp1vc/output",
        "size": 9
    }
}
caused by: failed comparison for key 'wc_output': expected: {
    "checksum": "sha1$3596ea087bfdaf52380eae441077572ed289d657",
    "class": "File",
    "size": 3
}
got: {
    "basename": "output",
    "checksum": "sha1$9aefd466231cf30b4e9ae5234d58d192af7bcac4",
    "class": "File",
    "location": "file:///var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp0noyp1vc/output",
    "path": "/var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp0noyp1vc/output",
    "size": 9
}
caused by: field 'checksum' failed comparison: expected: "sha1$3596ea087bfdaf52380eae441077572ed289d657"
got: "sha1$9aefd466231cf30b4e9ae5234d58d192af7bcac4"

They were mostly errors related to checksum.

6

Okay, I’ll see if I can find a Mac to examine this. In the mean time, this shouldn’t stop you from writing your own conformance tests.

The reason is possibly due to https://github.com/common-workflow-language/cwl-v1.2/blob/f0f9bc2ae01c7336f94037a1447ed5c2db08edf1/tests/wc-tool.cwl not using a docker container, and the MacOS wc command is behaving differently.

You could try running export CWLTOOL_OPTIONS=--default-container=docker.io/debian:stable-slim first and then re-run the tests as you did before.

1 Like
7

I was able to reproduce your issue today. I can get all the conformances tests to pass on MacOS if I set CWLTOOL_OPTIONS="--relax-path-checks --default-container=docker.io/debian:stable-slim

1 Like
8

Oh okay. Will try that out and revert.
Thank you.

9

So I tried that out and most run. However, I saw a new error with test 124.

I did the export before running the test and my docker was running.

Test [124/367] Test file literal as input without Docker
Test 124 failed: /Users/afimaamedufie/.pyenv/shims/cwl-runner --outdir=/var/folders/by/hbl2rsdx46v3qf_lvkw__7xm0000gn/T/tmp7ufdu_nc --quiet tests/cat3-nodocker.cwl tests/file-literal.yml
Test file literal as input without Docker
Does not support required feature
Error response from daemon: Head "https://registry-1.docker.io/v2/library/debian/manifests/stable-slim": Get "https://auth.docker.io/token?account=afigbadago&scope=repository%3Alibrary%2Fdebian%3Apull&service=registry.docker.io": net/http: TLS handshake timeout
ERROR Workflow or tool uses unsupported feature:
Docker is required to run this tool: Command '['docker', 'pull', 'docker.io/debian:stable-slim']' returned non-zero exit status 1.