Strategies for Parallelizing LLMs Masterclass

Introduction
1 Introduction & What Is This Course About
2 Course Structure
3 DEMO – What You’ll Build in This Course

Course Source Code and Resources
4 Get Course Slides
5 Get Source Code

Strategies for Parallelizing LLMS – Deep Dive
6 What is Parallelism and Why it Matters
7 Understanding the Single GPU Strategy
8 Understanding the Parallel Strategy and Advantages
9 Parallelism vs Single GPU – Summary

IT Fundamental Concepts
10 IT Fundamentals – Introduction
11 What is a Computer – CPU and RAM Overview
12 Data Storage and File Systems
13 OS File System Structure
14 LAN Introduction
15 What is the Internet
16 Internet Communication Deep Dive
17 Understanding Servers and Clients
18 GPUs – Overview

GPU Architecture for LLM Training Deep Dive
19 GPU Architecture for LLM Training
20 Why this Architecture Excels

Deep and Machine Learning – Deep Dive
21 Machine and Deep Learning Introduction
22 Deep and Machine Learning – Overview and Breakdown
23 Deep Learning Key Aspects
24 Deep Neural Networks – Deep Dive
25 The Single Neuron Computation – Deep Dive
26 Weights
27 Activation Functions – Deep Dive
28 Deep Learning – Summary
29 Machine Learning Introduction – ML vs DL
30 Learning Types and Full ML & DL Analogy Example
31 DL and ML Comparative Capabilities – Summary

Large Language Models – Fundamentals of AI and LLMs
32 Introduction
33 The Transformer Architecture Fundamentals
34 The Self-Attention Mechanism – Analogy
35 The Transformer Architecture Animation
36 The Transformer Library – Deep dive

Parallel Computing Fundamentals & Parallelism in LLM Training
37 Parallel Computing Introduction – Key Concepts
38 Parallel Computing Fundamentals and Scaling Laws – Deep Dive

Types of Parallelism in LLM Training – Data – Model and Hybrid Parallelism
39 Types of Parallelism in LLM Training
40 Data Parallelism – How It Works
41 Data Parallelism Advantages for LLM Training
42 Real-world Example – Data Parallelism in GPT-3 Training
43 Model Parallelism and Tensor Parallelism and Layer Parallelism – Deep Dive
44 LLM Relevance and Implementaion
45 Model vs Data Parallelism
46 Key Differences Highlighted – Data vs Model Parallelism
47 Data vs Model Parallelism
48 Hybrid Parallelism – Animation
49 Hybrid Parallelism – What is It and Motivation

Types of Parallelism – Pipeline and Tensor Parallelism
50 Pipeline Parallelism Overview
51 Pipeline Parallelism Key Concepts and How it Works – Step by Step
52 Pipeline Bubbles Key Concepts
53 Pipeline Schedules Key Concepts
54 Activation Recomputation – Overview and Introduction
55 Neural Network and Activation and Backward and Forward Passes – Full Dive
56 Understanding Activation Recomputation vs Standard Training – Deep Dive
57 Demo – Activation Recomputation Visualization
58 Activation Recomputation vs Standard Approach
59 Benefits of Activation Recomputation and Implementation Strategies
60 Pipeline Parallelism Implementation Frameworks and Key Takeaways

Tensor Parallelism – Deep Dive
61 What is Tensor Parallelism and Why – Benefits
62 Tensor Parallel Pizza Making Analogy
63 Tensors and Partitioning Strategies – Deep Dive
64 Tensor Communication Patterns – Deep Dive
65 Device Mesh Communication Pattern – Deep Dive
66 How Components Work Together in Distributed LLM Training
67 Understanding Tensor Parallelism with LEGO Bricks Animation Demo
68 Putting it All Together – All Strategies in LLM Training

HANDS-ON Strategies for Parallelism – Data Parallelism Deep Dive
69 Strategies for Parallelizing LLMs – Hands- on Introduction
70 Pytorch – LLM Training Library Overview
71 The Transformers Library – Overview
72 Numpy Overview
73 TorchVision and TorchDistributed Overview
74 DeepSpeed and Megatron-LM – Overview
75 Datasets and Why this Toolkit
76 HANDS-On Data Parallelism – Training a Small Model – MNIST Dataset
77 Testing Pseudo Data Parallelism Trained Model
78 HANDS-ON Data Parallelism – Colab – Full Demo
79 Data Parallelism – Simulated Parallelism on GPU Takeaways

HANDS-ON Data Parallelism w WikiText Dataset & DeepSpeed Mem. Optimizatization
80 Hands-on Data Parallelism – Wikitext-2 Dataset
81 DeepSpeed – Full Dive
82 Hands-on Data Parallelism with DeepSpeed Optimization

Running TRUE Parallelism on Multiple GPU Systems – Runpod.io
83 Setup Runpod.io Environment Overview
84 Runpod SSH Setup
85 Setting up Runpod Parallelism in JupyterNotebook
86 HANDS-ON – Parallelism with IMDB Dataset – Deep Dive – True Parallelism
87 Runpod Cleanup

Fault Tolerance and Scalability & Advanced Checkpointing Strategies – Deep Dive
88 Fault Tolerance Introduction & Types of Failures in Distributed LLM Training
89 Strategies for Fault Tolerance
90 Checkpointing in LLM Training – Animation
91 Basic Checkpointing in LLM Taining
92 Incremental Checkpointing in LLM Training
93 Asynchronous Checkpointing in LLM Training
94 Multi-level Checkpointing in LLM Training – Animation
95 Checkpoint Storage Considerations – Deep Dive
96 Implementing a Hybrid Approach – Performance, Failure, Optimizations – Full Dive
97 Checkpoint Storage Strategy – Summary

Advanced Topics and Emerging Trends
98 Advanced Topics and Emerging Trends

Wrap up and Next Steps
99 Course Summary and Next Steps