Send us Fan Mail
When leading frontier AI researchers estimate a greater than 10% chance of human extinction from artificial intelligence within this decade (p(doom) > 0.10), standard corporate risk management breaks down. Host Will breaks down the decision-theoretic mathematics of existential risk, audits AI failure probabilities against regulatory safety thresholds in aviation, nuclear power, and pharmaceuticals, and outlines three concrete institutional mechanisms needed to solve the Multipolar Prisoner's Dilemma driving the AI race.
Episode Timestamps & Highlights
- - Cold Open & Intro: Unpacking Anthropic's internal p(doom) disclosures and the systemic failure to evaluate unbounded risks.
- - Key Insight 1: The Math That Changes Everything
- Expected Utility Theory (EUT): Why standard cost-benefit calculations fail when accounting for 10^52 potential future human lives.
- Maxipok Principle: Nick Bostrom’s imperative to maximise the probability of an "OK outcome" when dealing with existential hazards.
- Probability Neglect: Cass Sunstein’s psychological research explaining why public discourse swings between sci-fi doom and total dismissal.
- - Key Insight 2: Cross-Industry Safety Audit
- Commercial Aviation: FAA standards mandate catastrophic failures be less than 1 in 1 billion per flight hour. A 10% ten-year failure probability would ground the global airliner fleet instantly.
- Nuclear Power: NRC rules limit Core Damage Frequency to under 1 in 10,000 per reactor-year. The annualised AI risk (~1%) is over 1,000 times higher than permissible reactor limits.
- Pharmaceuticals: FDA frameworks maintain zero tolerance for unmitigated 10% lethal risks in non-terminal populations. AI scaling imposes universal, non-consensual population exposure without an opt-out.
- - Key Insight 3: Inside the Labs & The Game-Theoretic Trap
- Technical Alignment Deficits: Outer alignment failures, inner/deceptive alignment, and black-box interpretability challenges.
- Multipolar Prisoner's Dilemma: Why individual lab speed running remains rational under competitive market dynamics despite recognised civilisational risks.
- - Actionable Policy Solutions
- 1. Statutory Compute Thresholds: Mandatory licensing for training runs exceeding 10^26 FLOPs.
- 2. Independent Third-Party Audits: State-funded evaluation bodies empowered to freeze unsafe models (e.g., Sanders-Casar Framework).
- 3. International Treaties: Binding global coordination modelled on the 1975 Asilomar rDNA Conference.
Key Quotes
- "Ten percent times negative infinity is not a manageable risk you can offset against an upside. It is negative infinity."
- "We are currently deploying AI systems at scale whose internal risk estimates are more than a thousand times worse than what the Nuclear Regulatory Commission accepts for a single reactor."
Up Next
Tune in for Episode 102 as we test AI safety against historical technology shifts—from 19th-century steam boiler explosions to the 1945 Manhattan Project—to determine if history offers a playbook or if agentic AI crosses an unprecedented "Novelty Boundary."
