CWL for HPC: are we there yet?

CWLCon 2023
1

Presenter: Iacopo Colonnelli, University of Torino @GlassOfWhiskey

Schedule:

  • Session 1: :earth_americas: :earth_africa: (Americas-EMEA) Monday, February 27th 09:00 - 13:00 US EST / 14:00 - 18:00 UTC
  • Session 2: :earth_africa: :earth_asia: (EMEA-APAC) Wednesday, March 1st 07:00 - 11:00 UTC / 16:00 - 20:00 Japanese Standard Time

Modern HPC applications are becoming so heterogeneous and complex that a modular approach to their design, deployment and orchestration is now necessary. This talk explores the benefits of using a vendor-agnostic workflow language (CWL) coupled with a hybrid workflow management system (StreamFlow) in the HPC ecosystem. Also, it will examine the requirements needed to model HPC applications effectively, the CWL’s readiness to meet such requirements, and the proposals made to improve the language where needed. Four real use cases will drive the discussion: the ACROSS Project (G.A. n. 955648), where CWL is the primary interface to model three HPC workflows, and the EUPEX Project (G.A. n. 101033975), where StreamFlow will be used for the rapid prototyping of a seismic engineering HPC application for a Modular Supercomputing Architecture (MSA) system.

CWL4HPC Matrix channel: https://matrix.to/#/#cwl4hpc:matrix.org

Slides: https://datacloud.di.unito.it/index.php/s/CMCd5LiZeXsxwEg

Please leave your questions for the presenter below!

As an alternative to YouTube, this presentations is also available on ConfTube

2 Likes
2

Hi Iacopo!

I’ve started playing with the latest version of cwltool to test loops and compare with other cyclic WMS’s like Cylc, ecFlow, StackStorm, etc. I was impressed for being able to convert a Cylc 8 simple workflow to CWL + Loops: kinoshita.eti.br/cylc1.md at 4455ad542a8a638317948136223415763e78bcbe · kinow/kinoshita.eti.br · GitHub

In Cylc (and also in ecFlow) in some workflows the cycles/loops have to be started in parallel. This is to maximize the use of resources in the HPC server, and because some of the steps in the workflow may be independent, meaning that the whole loop/cycle, or parts of it, can “run-ahead” (see Cylc’s docs about the runahead-limiting).

Q. Would it be possible with CWL + Loops to have something like:

  • A number of maximum runahead loop iterations / cycles?
2 Likes