This HPC training continues to expand your skills and knowledge in using productivity tools and technologies, to be able to quickly build up an efficient HPC user environment, from scratch, without admin rights. We shall demonstrate tools and procedures tailored to the Salomon and Anselm clusters, however they are easily replicable on any HPC system. The topics include:
GIT - a version control system for coordinating work among multiple developers
GIT (the stupid content tracker) is the world's most used Version Control System. Originally designed for development of Linux kernel, it has evolved to universal tool for managing changes in code, configuration and documents, especially if those changes are not done by a single person. We will help you understand how GIT works internally and introduce you to basic GIT commands, that should cover 99% of daily GIT usage.
KVM - a virtualization infrastructure for the Linux kernel that converts it into a hypervisor
KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It provides the ability to run complex software stacks, including MS Windows, on powerful, linux-based supercomputer nodes with very low overhead costs. Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc. In this lesson we will introduce the QEMU machine emulator using KVM virtualization, learn how to set up the QEMU/KVM on the Salomon supercomputer, create a QEMU system image, install the operating system into the image, set up the networking and data access and execute the calculations via the QEMU/KVM. We will also discuss visualized Ethernet networks in the context of QEMU and VDE2.
Docker & Singularity - technologies for paravirtualization
Docker for HPC? Yes, Singularity! Docker provides the ability to package and run an application in a loosely isolated environment called a container. The application with all settings and libraries can be packed together and then run on any other computer. It is perfect for developers, hosting providers, server farms, you name it! But, is it so good for supercomputers too? Let's look at why there are some problems with using Docker in shared environment and why Singularity is different. Singularity is developed with HPC in mind and is directly intended for use on HPC clusters with the direct support of technology such as OpenMPI. We will show you how to use Singularity, convert Docker images, create new containers and everything else you need to know about running a Singularity container in an HPC environment.
EasyBuild - install and build custom programs
In the supercomputer environment, pre-configured modules of various programs can be used. But how do I get my own program when I am a developer, or I need a program that is not installed? What if I need a different version of the program than the available modules? There is the possibility of sending a request for installation to the support, but what if I need it immediately? There are more options to manually build a program, but not everyone can do it because the user cannot install on the system but only in his own storage, and you need to know the compiler switches and system settings. You can also build a program using the tools that compile for us and create environment setting modules for that. In the lecture, we shall demonstrate how to manually compile a program, how to modify the environment so that the program can be used. We shall show you the EasyBuild tool to build a program and introduce new releases, such as creating Docker or Singularity rules and installing programs using EasyBuild directly into the Singularity image.
intermediate - advanced
Purpose of the course (benefits for the attendees)
The participants will broaden their range of techniques for efficient use of HPC by mastering modern technologies for code management and execution.
About the tutors
The tutors are core members of the Supercomputing Services division of IT4Innovations.
Branislav Jansík obtained his PhD in computational chemistry at the Royal Institute of Technology, Sweden in 2004. He took a postdoctoral position at IPCF, Consiglio Niazionale delle Ricerche, Italy, to carry on development and applications of high performance computational methods for molecular optical properties. From 2006 he worked on the development of highly parallel optimization methods in the domain of electronic structure theory at Aarhus University, Denmark. In 2012 he joined IT4Innovations, the Czech national supercomputing center, as the head of Supercomputing Services. He has published over 35 papers and co-authored the DALTON electronic structure theory code.
David Hrbáč obtained his Master's Degree in Measurement and Control Engineering at VŠB - Technical University of Ostrava in 1997. Since 1994 he has worked for many IT companies as a system architect and CIO. In 2013 he joined IT4Innovations.
Lubomír Prda obtained his Master's Degree in Information and Communication Technologies at VŠB - Technical University of Ostrava in 2010. Before joining the IT4Innovations team as an HPC specialist in 2016, he worked at the Tieto Corporation as a network engineer, and later moved to identity and access management for the company's nordic and international customers. Lubomír's focus is to manage and maintain the centre's back-end IT infrastructure and services.
This work was supported by the PRACE-5IP project – the European Union's Horizon 2020 research and innovation programme under grant agreement No. 730913.