[ONLINE] Containerisation, CI/CD, and Benchmarking Solutions for HPC

24 Mar 2025, 09:00 → 26 Mar 2025, 17:00 Europe/Prague

ONLINE

This course is organised in cooperation with CoE HiDALGO2 and NCC Czechia.

Annotation

This training is designed for individuals working on containerisation, automation, and performance optimisation in HPC environments. It provides a structured approach to developing, deploying, and benchmarking HPC applications using modern DevOps methodologies. Participants will learn to create containerised environments, automate workflows with CI/CD pipelines, and evaluate application performance using benchmarking tools.

The course covers both fundamental and advanced topics, including Docker, Singularity/Apptainer, GitHub Actions, GitLab Runners, JacamarCI, and benchmarking with reframe-hpc and feelpp.benchmarking. Practical, hands-on exercises will help attendees apply their knowledge to real HPC environments, improving workflow reproducibility, efficiency, and scalability.

This training is ideal for HPC administrators, software developers, DevOps engineers, researchers, and scientists who want to optimise HPC workflows and benchmark applications at scale.

Benefits for the attendees: what will they learn

1. Containerize HPC Applications - Create, manage, and optimise container images (Docker, Singularity/Apptainer).
2.  Implement CI/CD Workflows for HPC - Automate builds, tests, and deployments using GitHub Actions, GitLab Runners, and JacamarCI .
3. Design & Automate Benchmarking Pipelines - Use feelpp.benchmarking and reframe-hpc to evaluate performance, scaling, and resource usage.
4. Optimise & Scale HPC Workloads - Leverage HPC environment modules, schedulers, and best practices for high performance and reproducibility.

Target Audience

• Researchers, Scientists, and Engineers aiming to streamline HPC workflows.

• HPC Administrators or DevOps professionals seeking automation strategies on clusters.

• Software Developers looking to improve reproducibility and performance for scientific workloads.

Level

Intermediate

Language

English

Prerequisites

Essential :

• Bash/Python scripting
• Git (basic commands)
• YAML for CI configurations

 

Recommended :

• Docker/Singularity/Apptainer experience
• HPC environment usage (modules, job submission)

 

Tutors

Christophe Prud'homme - Professor in Applied Mathematics since September 2012 at the University of Strasbourg, France and from 2006 to 2012 in Grenoble. He founded Cemosis and is a manager and contributor to Feel++, an open-source framework for solving PDEs and complex problems. Feel++ is a building block for the Urban Building pilot. His expertise lies in Modeling, Simulation, Optimisation, and High-Performance Computing. He has been involved in numerous courses and tutorials on scientific computing, HPC, and open-source software development. He designed the CI/CD for the Urban Building pilot with automated deployment on EuroHPC systems.

Javier Cladellas is a Junior Research Software Engineer at Cemosis. He has been working as a developer at Cemosis since 2023. In particular, he has focused on developing a sensor measurement database for data validation and assimilation for the UB pilot in the European project Hidalgo2. Javier is also the main developer of the feelpp.benchmarking framework, conceived for benchmarking applications on HPC systems.

Sameer Haroon has been a Research Engineer at HLRS since 2023. He has a background in Systems Engineering and Domain-Specific Language development, and he is an expert in infrastructure-as-code (IaC) provisioning and maintenance of Web Services at HLRS.

Vincent Chabannes has been a Research Engineer at Cemosis in the Institute of Mathematical Research (IRMA) of Strasbourg since 2016. He is in charge of the computing infrastructure department and is one of the main developers of the Feel++ and Ktirio software. Vincent obtained his PhD in applied mathematics in 2013. The thesis focused on developing algorithms for the numerical simulation of blood flows, with particular emphasis on fluid-structure interaction modelling and high-performance computing.

Acknowledgements

Co-funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) and Poland, Germany, Spain, Hungary, France under grant agreement number: 101093457.

 

This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 101101903. The JU receives support from the Digital Europe Programme and Germany, Bulgaria, Austria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Ireland, Italy, Lithuania, Latvia, Poland, Portugal, Romania, Slovenia, Spain, Sweden, France, Netherlands, Belgium, Luxembourg, Slovakia, Norway, Türkiye, Republic of North Macedonia, Iceland, Montenegro, Serbia. This project has received funding from the Ministry of Education, Youth and Sports of the Czech Republic.

 

This course was supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).

 

All presentations and educational materials of this course are provided under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Course materials

Video recording

Monday, 24 March

09:00 → 09:30 Welcome, course objectives, and participant introductions

• Course objectives, key takeaways
• Participant introductions (optional)

09:30 → 10:45 Overview of containerization for HPC (Docker & Singularity/Apptainer) and CI/CD basics

HPC-specific constraints (security, MPI, parallel file systems)
GitHub Actions basics (workflows, runners, triggers)

10:45 → 11:00 Coffee Break

11:00 → 12:30 Hands-on: Building and pushing container images for HPC applications

• Configuring a GitHub Actions workflow file
• Running build/test stages for a sample HPC application
• Using VS Code & GitHub Classroom setup

12:30 → 13:30 Lunch Break

13:30 → 15:00 Introduction to CI/CD fundamentals with GitHub Actions

15:00 → 15:15 Coffee Break

15:15 → 16:30 Hands-on: Setting up a simple CI pipeline with GitHub Actions and handling containers

• Example HPC code (e.g., PDE solver)
• Testing on local environment (Docker) vs. Karolina (Singularity)

16:30 → 17:00 Q&A and wrap-up

• Review of Day 1
• Common pitfalls, best practices

Tuesday, 25 March

09:00 → 09:30 Recap of Day 1 and Q&A

• Open Q&A on Slack/Zoom

09:30 → 11:00 Morning: Advanced CI/CD techniques using GitHub Actions

• GitLab pipelines/runners for HPC
• JacamarCI fundamentals (orchestration for HPC)

11:00 → 11:15 Coffee Break

11:15 → 12:30 Hands-on: Creating a multi-stage CI pipeline with GitHub Actions

• GitLab CI file setup for HPC jobs (build, test, deploy)
• Integration with HPC scheduler (Slurm)
• Debugging pipeline runs in Slack channel

12:30 → 13:30 Lunch Break

13:30 → 15:00 Afternoon: Introduction to reframe-hpc and initial benchmark setups

• HPC performance considerations (CPU vs. GPU, network, memory)
• Tools: feelpp.benchmarking, reframe-hpc
• Automated performance data collection & reporting

15:00 → 15:15 Coffee Break

15:15 → 16:30 Hands-on: Building a benchmarking pipeline with reframe-hpc

• Designing a sample HPC benchmarking job
• Collecting and interpreting metrics (scaling, time-to-solution)

16:30 → 17:00 Q&A and discussion

• Review of Day 2
• CI/CD design in HPC: best practices

Wednesday, 26 March

09:00 → 09:30 Recap of Day 2 and Q&A

• Q&A via Slack/Zoom

09:30 → 11:00 Morning: CI/CD with GitLab Runner and integration with job schedulers

• Using reframe-hpc to manage multiple tests (configurations, node counts)
• Integrating feelpp.benchmarking for advanced analytics

11:00 → 11:15 Coffee Break

11:15 → 12:30 Hands-on: Building a CI/CD pipeline using GitLab Runner

• Multi-run orchestration and results collection
• Creating automated reports

12:30 → 13:30 Lunch Break

13:30 → 15:00 Afternoon: Introduction to feelpp.benchmarking for advanced performance analysis

• Environment modules, container HPC images
• Large data handling (parallel file systems)
• Performance tuning (profiling tools, specialized libraries)

15:00 → 15:15 Coffee Break

15:15 → 16:30 Hands-on: Designing multi-run benchmarking tests and generating automated reports

• Open discussion on HPC container orchestration
• Next steps, further resources

16:30 → 17:00 Final Q&A, open discussion, and closing remarks

• Course feedback
• Certificates (if applicable)