Building Large Language Models: Complete 102-Minute Lecture Notes
Master Thesis: A frontier chatbot (ChatGPT, Claude, Gemini) is built in two distinct regimes—Pre-training (→ GPT-3) compresses trillions of cleaned web tokens into weights governed by predictable scaling laws and GPU memory-bandwidth limits; Post-training (→ ChatGPT) turns that text completer into an assistant via SFT, RLHF/DPO, and length-controlled evaluation.
Source Verification Note: Although the viral X post labels this 102-minute stream a “Claude AI automation course,” scanning all 35 timepoints confirms it is Yann Dubois’s Stanford CS229 lecture on how frontier LLMs work. Read below for every equation, dataset number, and empirical proof from the full recording.
PROPORTIONAL 102-MINUTE LECTURE TIMELINE (CLICK ANY CHAPTER TO JUMP)00:00 ──────────────────────────► 01:42:37
■ Crimson: Part I · Pre-training → GPT-3 (59% of lecture)■ Teal: Part II · Post-training & Eval → ChatGPT (34%)■ Navy: Part III · GPU Memory Wall (7%)
EXECUTIVE SYNTHESIS
Six Core Arguments You Walk Away With From the Full 102 Minutes
THESIS 01 · SLIDES 25–2615T FINEWEB
Clean Data Is the Real Moat, Not Architecture
Raw Common Crawl is 250B dirty pages (>1 PB). Filtering, deduplication, and domain mixing drive practical quality—kept secret for competitive advantage and copyright liability.
THESIS 02 · SLIDES 28–333d TUNE → 27d RUN
Large Models Don’t Overfit; Tune Small, Extrapolate Big
Given 10K GPUs for 30 days, tune small models in <3 days, fit scaling recipes, and spend ~27 days training one optimal frontier run.
THESIS 03 · SLIDE 34FLOPS = 6NP
Frontier Compute Follows Four Lines of Arithmetic
Llama 3 405B (15.6T tokens, ~40 tok/param) = 3.8×10²⁵ FLOPs → ~70 days on 16K H100s at 400 TFLOPS (~30M real GPU-hrs).
THESIS 04 · SLIDES 45–50SFT → PPO → DPO
SFT Only Clones Humans; DPO Matches PPO Without RL
RLHF uses a Bradley-Terry reward model + PPO; DPO optimizes preferred vs. rejected log-ratios directly to reach the same global optimum.
THESIS 05 · SLIDES 53–5922.9% → 64.3%
Humans & LLM Judges Reward Length Over Truth
RLHF inflates length 4.1× (59 → 243 tokens). Simply prompting GPT-4 to be verbose nearly triples its AlpacaEval win rate—requiring Length-Controlled evals.
THESIS 06 · SLIDE 640.2% → 39% TIME
GPUs Starve on HBM Memory Bandwidth, Not FLOPs
In 20 years GPU FLOPs grew 60,000× vs. 100× DRAM bandwidth. Normalization & activations are 0.2% of FLOPs yet eat 39% of Transformer runtime.
INTERACTIVE LECTURE LAB
Explore the Lecture’s Equations & Empirical Proofs Live
Autoregressive Language Modeling & Byte-Pair Encoding (BPE)
Every frontier LLM starts as a pure probability engine trained with Cross-Entropy loss to predict the next subword token—with zero mathematical approximation via the chain rule.
Exact Autoregressive Factorization (Slide 7): Given context "I saw a cat on a", embeddings feed a neural network to emit vector h, classifying over the entire vocabulary for the next token.
P(x₁, x₂, …, x_L) = ∏_{i=1..L} P(x_i | x₁, …, x_{i-1}) | Objective: Cross-Entropy Loss over BPE vocabulary
Why BPE Tokenizers (Slide 15): Word-level vocabularies fail on typos/code; character-level sequences explode attention cost. BPE initializes 1 token per character and iteratively merges the most frequent adjacent pairs until reaching target vocab size.
Figure 1 · Slide 2 (00:10)Opening framing: explaining how ChatGPT, Claude, Gemini, and Llama all emerge from the same two-stage autoregressive + post-training recipe.
Pre-Training Evaluation: What Perplexity Actually Tells You
Perplexity intuitively measures how many candidate tokens the model is “hesitating” between—dead as a public leaderboard, yet indispensable internally while training runs.
Between 2017 and 2023, test perplexity plunged from hesitating across ~70 tokens (Grave et al. 2016 LSTM) to ~20 (GPT-2, 2019) and <15 (SparseGPT 175B, 2023). Because perplexity moves smoothly with compute and data, labs still monitor it continuously during pre-training diagnostics.
Raw web crawls (Common Crawl: 250B pages, >1 PB) are noisy and skewed; cleaning, deduplicating, and domain-balancing data matters more than architecture tweaks.
Open Corpus Scaling:C4 (150B tokens / 800GB) → The Pile (280B) → Dolma (3T) → FineWeb (15T tokens). Frontier labs guard their mixtures strictly for two explicit reasons: competitive dynamics and copyright liability.
Figure 2 · Slide 26 (00:39)Explicitly calls data “~the key”, documents why mixtures remain secret, and highlights four active research frontiers: processing speed, domain ratios, synthetic data, and multimodal corpora.
Empirically, larger models do not overfit—power laws let teams tune tiny cheap models and extrapolate optimal settings before spending millions on a single frontier run.
The 10K-GPU Month Playbook (Slide 29): Never burn 30 days tuning 30 huge runs. Tune small models in <3 days, extrapolate learning rate & batch recipes via Chinchilla IsoFLOP curves (Slide 31), and reserve ~27 days for the final large run—embodying The Bitter Lesson (Slide 33) that general compute-scaling methods defeat hand-crafted heuristics.
Figure 3 · Slide 29 (00:45)The practical reason scaling laws revolutionized AI engineering: turning hyperparameter tuning from an unaffordable gamble into predictable extrapolation.
Four lines of arithmetic reproduce Meta’s Llama 3 405B training budget within ~15%—and explain why labs deliberately over-train past Chinchilla compute-optimality.
FLOPs = 6 × 15.6e12 tokens × 405e9 params = 3.8×10²⁵ FLOPs (~2× below US AI Executive Order 10²⁶ threshold) Time = 3.8e25 / (400e12 TFLOPS × 3600s) = 26M GPU-hrs ÷ (16,000 H100s × 24h) ≈ 70 days (Paper: ~30M GPU-hrs)
Llama 3 405B trained at ~40 tokens/param (~2× Chinchilla training-optimal) specifically so the resulting model is smaller and cheaper across billions of inference calls.
Figure 4 · Slide 34 (00:57)Exact back-of-the-envelope compute derivation for Llama 3 405B—try varying tokens & parameters in the interactive calculator above.
Turning GPT-3 Into ChatGPT: SFT → RLHF (PPO) → DPO
Language modeling ≠ assisting users. SFT clones human demonstrations, RLHF optimizes pairwise preferences via a Bradley-Terry reward model + PPO, and DPO hits the exact same optimum without RL.
SFT Ceiling (Slide 45): SFT is behavior cloning of humans—it teaches chat format but cannot reliably exceed annotators or penalize confident hallucinations.
RLHF/PPO vs. DPO (Slides 47 & 50): PPO fits a continuous Reward Model R(x,y) via Bradley-Terry logistic loss and optimizes via RL. DPO [Rafailov+ 2023] eliminates the separate reward model and RL sampling loop—directly increasing preferred y_w vs. rejected y_l log-probability ratios against π_ref with the same global minima as RLHF/PPO.
The Human Data Bottleneck, LLM Judges & Verbosity Hacking
Human annotators and GPT-4 judges share one massive blind spot: both systematically reward longer, prettier responses over concise accuracy.
RLHF Length Explosion (Slide 53 · [Singhal+ 2024]): Mean output length jumps 4.1× from 59 tokens (SFT) to 243 tokens (RLHF). Replacing human raters with LLM preferences (Slide 54) matches ~65% human mode agreement at 10×–100× lower cost per 1,000 pairs.
Chatbot Arena & Length-Controlled Evals (Slides 57 & 59): Blinded pairwise Elo (300K+ votes) plus Length-Controlled AlpacaEval defend against verbosity gaming—where making gpt4_1106_preview verbose inflates raw win rate from 22.9% to 64.3% (plotted in Empirical Proof 1 above).
§8 · 01:36 – 01:42 · SLIDE 64 · PART III: SYSTEMS 101
Systems 101: Why GPUs Starve on HBM Memory Bandwidth
Hardware FLOPs grew 60,000× over 20 years while DRAM bandwidth grew 100× and interconnects 30×—making tensor data movement the dominant bottleneck in modern Transformers.
Per DataMovement [Ivanov+ 2020], Matmul accounts for 99.80% of FLOPs in just 61.0% of runtime, whereas LayerNorm/Softmax (0.17% FLOPs → 25.5% runtime) and element-wise activations (0.03% FLOPs → 13.5% runtime) consume 39% of wall-clock time waiting on GPU memory—explaining why FlashAttention, operator fusion, and BF16/FP8 are non-negotiable.
Figure 6 · Slide 64 (01:39)20-year FLOPs vs. memory bandwidth divergence chart and BERT Transformer PyTorch runtime table.
END-TO-END SYNTHESIS
Pre-Training (→ GPT-3) vs. Post-Training (→ ChatGPT) Blueprint
Dimension
Stage I · Pre-Training (Base Model)
Stage II · Post-Training (Assistant)
Core Job
Compress world knowledge, language & code into weights
Steer into helpful, safe assistant behavior
Data Scale
Trillions of cleaned web tokens (15.6T Llama 3 / 15T FineWeb)
10K–1M+ instruction & preference pairs (human or GPT-4 rated)
Algorithm
Autoregressive Cross-Entropy over BPE tokens
SFT → RLHF (Bradley-Terry RM + PPO) or DPO
Compute Budget
~95%+ of total spend (3.8×10²⁵ FLOPs · ~70d on 16K H100s)
Tiny fraction of FLOPs, maximum impact on UX & safety
Main Failure Mode
Dirty web text, suboptimal mix, HBM memory-bandwidth stalls