John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.
We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.
Topics:
- Implicit vs. explicit world models, and the case against JEPA-style objectives
- Compact belief states: why compression beats a growing KV cache
- Does research still matter in the age of scale? The Kimi K3 argument
- Agent-driven research: 2,000 pre-training experiments in 90 days
- The invention of CAPTCHA
- Optimizers from SGD and Vowpal Wabbit to Muon
Chapters
- 00:00 Why world models: the sample-complexity gap
- 09:48 The case against JEPA; a transformer-style implicit world model
- 15:52 Compact belief states: epicycles vs. heliocentrism
- 23:41 Does research still matter? The Kimi K3 argument
- 27:35 Open vs. closed models
- 35:57 Recursive self-improvement and agent-driven research
- 42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility
- 54:54 The invention of CAPTCHA
- 1:00:53 Optimizers: from Vowpal Wabbit to Muon
Music
- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
