On-the-fly document resolving

CWL Questions
1

Hi!

First of all, congratulations for your great work :slight_smile: It is the first time I write here, but I have een using cwltool for a while and it is great

I am trying to make a small REST API for running geospatial processing, using cwltool as an imported library. Making it simple, you can register steps using CommandLineTools and docker images: the CWL is stored in a database, recovered when required, saved into a temporary file and executed. It works out of the box.

However, I have some issues running a workflow. For example, let’s imagine the following workflow:

{
	"cwlVersion": "v1.0",
	"class": "Workflow",
	"inputs": {
		"start_time": "string",
		"end_time": "string",
		"aoi": "string"
	},
	"outputs": {
		"output": {
			"type": "File",
			"outputSource": "step2/output"
		}
	},
	"steps": {
		"step1": {
			"run": "step1_cwl",
			"in": {
				"start_time": "start_time",
				"end_time": "end_time",
				"aoi": "aoi"
			},
			"out": [
				"output"
			]
		},
		"step2": {
			"run": "step2_cwl",
			"in": {
				"aoi": "step2/output"
			},
			"out": [
				"output"
			]
		}
	}
}

In this case, both step1_cwl and step2_cwl are CommandLineTools CWLs registered before using the REST API. As I said, they work as individual steps, but they are not available as .cwl files, and therefore, when I try to load the workflow with load_document, they are not resolved and an error is raised.

My first attempt was to use the loading context’s resolver, but I realized it only works for finding the root file.

Is there any callback or similar for loading those files (into memory or into temporary files) as the document is being parsed?

Thanks in advance, and sorry for my eEnglish (it is not my mother tongue)

Luis

2

I have been checking and maybe it is better to parse the document with cwl-utils recursively, instead of patching load_document() or similar. What do you think?

Thanks!!

Luis