# Various tips for over thousand of jobs at HPC environment

**URL:** <https://cwl.discourse.group/t/various-tips-for-over-thousand-of-jobs-at-hpc-environment/879>\
**Category:** CWLCon 2024\
**Created:** [April 25, 2024, 10:07am UTC](https://cwl.discourse.group/t/various-tips-for-over-thousand-of-jobs-at-hpc-environment/879 "2024-04-25T10:07:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![manabuishii](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/manabuishii/32/513_2.png) [@manabuishii](https://cwl.discourse.group/u/manabuishii)\
**Post date:** [April 25, 2024, 10:07am UTC](https://cwl.discourse.group/t/various-tips-for-over-thousand-of-jobs-at-hpc-environment/879/1 "2024-04-25T10:07:07Z")

</div>

Presenter: [Manabu Ishii](https://orcid.org/0000-0002-5843-4712)

Schedule: [Session 3 (Amsterdam / Online hybrid) Wednesday 15 May, APAC-EMEA Friendly time](https://www.commonwl.org/conferences/2024/#session3)

> Discusses practical tips for using Toil, a workflow engine, in various High-Performance Computing (HPC) environments. This talk will cover the seamless integration of Toil with existing Common Workflow Language (CWL) definitions without needing modifications. Key points include the creation of a singularity cache to avoid delays in image creation from the internet if connectivity issues arise. This talk emphasizes the importance of logging for troubleshooting and outlines use of grid engines and SLURM batch systems, which work efficiently without modifications. This talk will cover several command line tips to extend Toil’s functionality, including memory management techniques essential for resource-intensive applications such as Java-based tools. The talk aims to provide the audience with practical insights into optimizing Toil workflows in HPC environments to ensure smooth and efficient operations.

Slide: [CWLCon2024 Toil on HPC TIPS\_20240515 - Google Slides](https://docs.google.com/presentation/d/1G7Qv6d2DwxCniRRyJTPWL4xZkwcrphl8v6hbYoWylbo/edit?usp=sharing)

**Please leave your questions for the presenter below!**

[![](https://global.discourse-cdn.com/free1/uploads/cwl1/original/1X/1e993f83f090f132c860a5f47454a9a437753c73.jpeg "Various tips for over thousand of jobs at HPC environment") ](https://www.youtube.com/watch?v=qSJ05lfBvm4)

As an alternative to YouTube, this presentations is also available on ConfTube

https://conf.tube/videos/embed/a865e290-f57f-4265-a70e-7f87ea2f029b

---

<div class="post-metadata">

**Author:** ![manabuishii](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/manabuishii/32/513_2.png) [@manabuishii](https://cwl.discourse.group/u/manabuishii)\
**Post date:** [May 10, 2024, 8:08am UTC](https://cwl.discourse.group/t/various-tips-for-over-thousand-of-jobs-at-hpc-environment/879/2 "2024-05-10T08:08:28Z")

</div>

Release 6.1.0 · DataBiosphere/toil [Release 6.1.0 · DataBiosphere/toil · GitHub](https://github.com/DataBiosphere/toil/releases/tag/releases%2F6.1.0)

I found

- Toil will no longer warn about a missing XDG\_RUNTIME\_DIR ([#4769](https://github.com/DataBiosphere/toil/pull/4769))

I talk WORKAROUND about this, but 6.1.0 fix it , I hope !!!

Thanks developers

---

<div class="post-metadata">

**Author:** ![manabuishii](https://yyz2.discourse-cdn.com/free1/user_avatar/cwl.discourse.group/manabuishii/32/513_2.png) [@manabuishii](https://cwl.discourse.group/u/manabuishii)\
**Post date:** [May 10, 2024, 4:10pm UTC](https://cwl.discourse.group/t/various-tips-for-over-thousand-of-jobs-at-hpc-environment/879/3 "2024-05-10T16:10:04Z")

</div>

Toil debugging

Already merged

[Allow debugging jobs by name (and status improvements) by adamnovak · Pull Request #4840 · DataBiosphere/toil](https://github.com/DataBiosphere/toil/pull/4840)

Document

[toil/docs/running/debugging.rst at master · DataBiosphere/toil](https://github.com/DataBiosphere/toil/blob/115be6700e284e90d034b40ac4eea60dff06d526/docs/running/debugging.rst)