Call for Outreachy project ideas

GSoC/Outreachy
1

Hi everyone,

Curii, using grant funds from the Chan-Zuckerberg Initiative, is sponsoring two Outreachy interns to work on projects for the CWL community for the December 2021 - March 2022 internship period.

Previously, interns worked on a WDL-CWL translator and on CWL Viewer.

This is a call for project ideas from the community for the next internship period. Please reply with your ideas, and also if you would be able to co-mentor the project. Projects for this period are due September 29th.

Thanks,
Peter

2

Automatic completion of workflow composition in CWL
A tool to help workflow writers in CWL.

3

Maturing CWL’s MPIRequirement extension

  • MPI (the message passing interface - https://www.mpi-forum.org) is the most widely-used standard for distributed memory parallelism in high-performance computing.
  • The cwltool reference runner supports an extension (MPIRequirement) for running MPI-parallelised tools within a workflow. This is necessary as typically applications have to be started with an MPI-library specific launcher tool which deals with starting the required number of processes across one or more nodes and putting them in communication with each other.
  • The requirement is ONLY implemented in the reference runner
  • The requirement only allows a CommandLineTool to set its number of processes. Other configuration (such as process placement) is global across the executed workflow which could be problematic.

A project could involve implementing support for MPIRequirement within another CWL runner, such as Toil (may require help from a Toil developer!)

Another enhancement would be to give the runner a better understanding of the execution hardware (threads per core, cores per NUMA region, NUMA regions per socket, sockets per node) and to expose this to a tool. A related idea would be to give some awareness of OpenMP (the commonest way of doing shared-memory parallelism in HPC), such that the CWL author can say “run my tool over all the hardware it’s got, one MPI process per NUMA region, one OpenMP thread per core”. E.g.:

MPIRequirement:
  processes: $(runtime.number_of_nodes * runtime.numa_regions_per_node)
OMPRequirement:
    NUM_THREADS: $(runtime.cores_per_numa_region)

It is probable (although I haven’t asked yet) that we can make a small amount of HPC time available for development and testing.

I could co-mentor.

2 Likes
4

Hi Rupika, would GitHub - rabix/benten: A language server for Common Workflow Language be helpful here?

5

Hi Rupert, thank you for volunteering to co-mentoring a CWL MPI project. I’m going to need your help.

Could you create an account at https://www.outreachy.org/ and then go to Common Workflow Language: Call for Outreachy mentors and volunteers to submit a project proposal?

6

Toil on Slurm would be a good candidate for maturing MPIRequirement, I think, because of the straightforward way that it maps workflow steps to Slurm jobs. I am not an expert but adding MPI parameters could happen in toil.batchSystems.slurm.prepareSbatch. The tricky part would be convincing Toil to propagate the MPRRequirement down to that point, the same way it does for mem and cpus-per-task.

@rupertnash do you have a specific use case for MPI in mind? I might be able to help with testing using our HPC platform

1 Like
7

Hi David - yes Toil + MPIReq would be good. I have glanced at the Toil source but didn’t get very far.

Unfortunately I’ve not been given the OK to offer an intern simple access to one of our machines (I’d have to do a more involved process) so I just withdrew the project :’( I guess I could propose this as a student project internally (they already have HPC access)

8

If you could give an external intern access to your cluster for testing, we can revive Rupe’s Outreachy proposal!

9

Wearing my “publishing workflows” hat, it would be nice if the CWL Viewer was updated to allow iframes/easily embedable workflows. It was previously mentioned to me that this was a likely small addition for the future.

10

Currently Toil’s use of Slurm just goes through the command line and looks straightforward to me.

11

Some progress here: API for SVG sketch of CWL workflow and API for PNG sketch of CWL workflow

1 Like
12

I can confirm that we can give external access to the DAS-6: Distributed ASCI Supercomputer 6 slurm cluster for this purpose.