This course is organised in collaboration with MAX CoE, EuroCC Czechia and EuroCC Germany.
Annotation
Many scientific workflows consist of huge numbers of small tasks, which traditional HPC schedulers such as SLURM and PBS handle poorly. This tutorial, presented by the core developers of HyperQueue, introduces this open-source scheduler and meta-scheduler for large pipelines.
Starting from the basics, we show how to run millions of tasks on SLURM- or PBS-managed clusters. HyperQueue requests allocations automatically and distributes tasks into them with minimal overhead. It also works on systems without any batch scheduler. It installs as a single binary without admin rights and is already preinstalled on many supercomputers.
Participants will learn to submit and monitor tasks and to define complex resource requests for heterogeneous clusters. We conclude with best practices and tips drawn from our experience supporting real-world pipelines.
Benefits for the attendees, what will they learn:
-
Recognise when the “many small tasks” pattern overwhelms SLURM or PBS, and how a meta-scheduler solves it.
-
Install HyperQueue on their own machine or use an existing installation on a supercomputer.
-
Submit, monitor, and manage large numbers of tasks, including task dependencies.
-
Let HyperQueue request SLURM/PBS allocations automatically instead of submitting jobs by hand.
-
Define complex resource requests (e.g. GPUs, CPU counts, custom resources) and use heterogeneous clusters effectively.
- Keep scheduling overhead low when scaling to millions of tasks.
- Apply best practices and avoid common pitfalls, with tips from real-world pipelines shared by the HyperQueue developers.
Level
beginner, intermediate
Prerequisites
No prior HyperQueue experience is required, only basic familiarity with the command line and HPC clusters.
Language
English
Tutor
Ada Böhm holds a Ph.D. in Computer Science and is a researcher at the IT4Innovations National Supercomputing Center, Czech Republic. She works on task scheduling and distributed computing for HPC and is a core author of HyperQueue. Her work also covers Rust systems programming, and infrastructure for quantum computing. She has a background in formal methods and automata theory.
Acknowledgements
EuroCC 3 has received funding from the European High-Performance Computing Joint Undertaking (JU) under Grant Agreement No. 101306701. The JU receives support from European Union‘s Digital Europe Programme and Germany, Albania, Austria, Belgium, Bosnia and Herzegovina, Bulgaria, Croatia, Cyprus, Czechia, Denmark, Estonia, Finland, France, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Montenegro, Netherlands, North Macedonia, Norway, Poland, Portugal, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Türkiye, Kosovo. The EuroCC 3 project received funding from the Ministry of Education, Youth and Sports of the Czech Republic.
MaX - Materials design at the Exascale has received funding from the European High Performance Computing Joint Undertaking and Participating Countries in Project (Czechia, France, Germany, Italy, Slovenia and Spain) under grant agreement no. 101093374.
.png)
This course was supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).
All presentations and educational materials of this course are provided under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.