Various tips for over thousand of jobs at HPC environment

Presenter: Manabu Ishii

Schedule: Session 3 (Amsterdam / Online hybrid) Wednesday 15 May, APAC-EMEA Friendly time

Discusses practical tips for using Toil, a workflow engine, in various High-Performance Computing (HPC) environments. This talk will cover the seamless integration of Toil with existing Common Workflow Language (CWL) definitions without needing modifications. Key points include the creation of a singularity cache to avoid delays in image creation from the internet if connectivity issues arise. This talk emphasizes the importance of logging for troubleshooting and outlines use of grid engines and SLURM batch systems, which work efficiently without modifications. This talk will cover several command line tips to extend Toil’s functionality, including memory management techniques essential for resource-intensive applications such as Java-based tools. The talk aims to provide the audience with practical insights into optimizing Toil workflows in HPC environments to ensure smooth and efficient operations.

Slide: CWLCon2024 Toil on HPC TIPS_20240515 - Google Slides

Please leave your questions for the presenter below!

As an alternative to YouTube, this presentations is also available on ConfTube

Release 6.1.0 · DataBiosphere/toil Release 6.1.0 · DataBiosphere/toil · GitHub

I found

  • Toil will no longer warn about a missing XDG_RUNTIME_DIR (#4769)

I talk WORKAROUND about this, but 6.1.0 fix it , I hope !!!

Thanks developers

1 Like

Toil debugging

Already merged

Allow debugging jobs by name (and status improvements) by adamnovak · Pull Request #4840 · DataBiosphere/toil


toil/docs/running/debugging.rst at master · DataBiosphere/toil

1 Like