Hello all,
A few months ago I completed my master’s thesis “CWL workflows with MPI in bare-metal, containers, cloud, and HPC”, supervised by Professor Pablo Quesada, and Michael Crusoe. Adam Novak from Toil & Iacopo Colonnelli from StreamFlow also provided immense help in guiding me how to launch the CWL workflows with their runners, and troubleshooting issues found.
We ran the CWL conformance tests using CWL v1.0, v1.1, and v1.2 with cwltool, Toil, and StreamFlow. The tests were executed on the following HPCs: CESGA FinisTerrae III, BSC MareNostrum5, and CSC LUMI. We also ran a simple MPI workflow with & without Singularity with cwltool & Toil. And finally a more complex MPI workflow with cwltool running the MPI model FALL3D. The MPI tests included a test with baseCommand: mpirun, and with baseCommand: some-app and the MPIRequirement extension instead controlling the launch of the application with the MPI implementation available.
In general we were able to run several CWL conformance tests and workflows on these HPC clusters successfully. We also identified several improvements for cwltool, StreamFlow, Toil, and also for other code used like cwltest and cwl-utils.
After the thesis defense, I got an HPC project to run the tests on RIKEN Fugaku (thank you Mohamed Wahib for the project hours). I ran the CWL v1.0 conformance tests successfully on a login node, and within a PJM job allocation using a newer version of cwltool. A simple MPI workflow ran successfully too on Fugaku’s login nodes, but they failed on PJM due to an error pulling and building containers with cwltool. I am still investigating the cause for that, and after that I still need to run the MPI tests with StreamFlow and Toil.
Today I finished running the conformance tests successfully on JSC JUPITER Booster (thank you Damian Alvarez for the project hours) with cwltool and CWL v1.0, v1.1, and v1.2, using login nodes and a Slurm job allocation. I will launch the tests with Toil and StreamFlow now, and then will start the MPI tests (the simple MPI workflow first, and then will try to compile FALL3D to run a larger allocation).
You can find the thesis logs & results in the Git repository GitHub - kinow/cwl-mpi: Repository for research artefacts produced in the thesis “CWL workflows with MPI in bare-metal, containers, cloud, and HPC” · GitHub .
If you have a HPC where you can provide ~100/300 hours, I can run the tests there or give you the instructions from my thesis to try to reproduce the runs there yourself. For JUPITER, so far I believe I used <2 hours to run all the conformance tests with cwltool. So even though I normally ask for 100/300 hours, all tests may be run successfully in well under 100 hours if everything goes well.
The master’s thesis is an activity linked to our working group, CWL4HPC: CWL4HPC Working Group | Common Workflow Language (CWL) . We have recurrent meetings on Element/Matrix every 1st Monday of each month, and on the 3rd Monday whenever we have agenda items.
Cheers, Bruno