CWL for HPC: are we there yet?

Presenter: Iacopo Colonnelli, University of Torino @GlassOfWhiskey


  • Session 1: :earth_americas: :earth_africa: (Americas-EMEA) Monday, February 27th 09:00 - 13:00 US EST / 14:00 - 18:00 UTC
  • Session 2: :earth_africa: :earth_asia: (EMEA-APAC) Wednesday, March 1st 07:00 - 11:00 UTC / 16:00 - 20:00 Japanese Standard Time

Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWLโ€™s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.

CWL4HPC Matrix channel:


Please leave your questions for the presenter below!

As an alternative to YouTube, this presentations is also available on ConfTube

1 Like

Hi Iacopo!

Iโ€™ve started playing with the latest version of cwltool to test loops and compare with other cyclic WMSโ€™s like Cylc, ecFlow, StackStorm, etc. I was impressed for being able to convert a Cylc 8 simple workflow to CWL + Loops: at 4455ad542a8a638317948136223415763e78bcbe ยท kinow/ ยท GitHub

In Cylc (and also in ecFlow) in some workflows the cycles/loops have to be started in parallel. This is to maximize the use of resources in the HPC server, and because some of the steps in the workflow may be independent, meaning that the whole loop/cycle, or parts of it, can โ€œrun-aheadโ€ (see Cylcโ€™s docs about the runahead-limiting).

Q. Would it be possible with CWL + Loops to have something like:

  • A number of maximum runahead loop iterations / cycles?
1 Like