[HYBRID] Data Parallelism: How to Train Deep Learning Models on Multiple GPUs

Annotation

Modern deep learning challenges leverage increasingly larger datasets and more complex models. As a result, significant computational power is required to train models effectively and efficiently. Learning to distribute data across multiple GPUs during training makes possible an incredible wealth of new applications that utilize deep learning.

Effectively using systems with multiple GPUs also reduces training time, allowing for faster application development and much faster iteration cycles. Teams who can train with multiple GPUs have an edge, building models trained on more data in shorter periods and with greater engineer productivity.

This workshop teaches you techniques for data-parallel deep learning training on multiple GPUs to shorten the training time required for data-intensive applications. Working with deep learning tools, frameworks, and workflows to perform neural network training, you’ll learn how to decrease model training time by distributing data to multiple GPUs while retaining the accuracy of training on a single GPU.

Learning Objectives

By participating in this workshop, you’ll:

• Perform data-parallel deep learning training with multiple GPUs
• Achieve maximum throughput when training for the best use of multiple GPUs
• Distribute training to multiple GPUs using PyTorch Distributed Data-Parallel (DDP)
• Understand and utilize algorithmic considerations specific to multi-GPU training performance and accuracy

Level

Advanced

Language

English

Hardware requirements

Desktop or laptop computer capable of running the latest version of Chrome or Firefox. Each participant will be provided with dedicated access to a fully configured, GPU-accelerated workstation in the cloud.

Tools, libraries, and frameworks

PyTorch, PyTorch Distributed Data-Parallel, NVIDIA Collective Communications Library (NCCL)

Tutor

Georg Zitzlsberger is a research specialist for Machine and Deep Learning at IT4Innovations. For over four years, he has been certified by NVIDIA as a University Ambassador of the NVIDIA Deep Learning Institute (DLI) program. This certification allows him to offer NVIDIA DLI courses to users of IT4Innovations' HPC services. In addition, in collaboration with Bayncore, he was a trainer for Intel HPC and AI workshops and conferences across Europe. He has been contributing to these events, which are held for audiences from industry and academia, for five years. Recently, he also received instructor certifications from Intel for oneAPI-related courses.

Certificate

Upon completing the assessment, participants will receive an NVIDIA DLI certificate from recognising their subject matter competency and supporting professional career growth.

Practicalities

This training will be a hybrid event. Technical details about joining will be sent to the accepted registrants before the event. If you are coming to IT4Innovations to attend personally, please bring your laptop.

Please note that the training is held using Zoom. We advise all participants to download the Zoom application to enjoy full functionality. The link to the Zoom meeting will be sent only to registered participants.

The capacity is limited to 30 participants combined online and onsite.

Price Information

Full price per person: 10 000 CZK

Subsidized price: 1 500 CZK

Terms of subsidy and cancellation

Registration