Maturing CWL’s MPIRequirement extension
- MPI (the message passing interface - https://www.mpi-forum.org) is the most widely-used standard for distributed memory parallelism in high-performance computing.
cwltool reference runner supports an extension (
MPIRequirement) for running MPI-parallelised tools within a workflow. This is necessary as typically applications have to be started with an MPI-library specific launcher tool which deals with starting the required number of processes across one or more nodes and putting them in communication with each other.
- The requirement is ONLY implemented in the reference runner
- The requirement only allows a
CommandLineTool to set its number of processes. Other configuration (such as process placement) is global across the executed workflow which could be problematic.
A project could involve implementing support for
MPIRequirement within another CWL runner, such as Toil (may require help from a Toil developer!)
Another enhancement would be to give the runner a better understanding of the execution hardware (threads per core, cores per NUMA region, NUMA regions per socket, sockets per node) and to expose this to a tool. A related idea would be to give some awareness of OpenMP (the commonest way of doing shared-memory parallelism in HPC), such that the CWL author can say “run my tool over all the hardware it’s got, one MPI process per NUMA region, one OpenMP thread per core”. E.g.:
processes: $(runtime.number_of_nodes * runtime.numa_regions_per_node)
It is probable (although I haven’t asked yet) that we can make a small amount of HPC time available for development and testing.
I could co-mentor.