Artificial General
Intelligence: a working paper.
A synthesis of the technical, governance, and philosophical literature as of 2026 — written so that you can argue with someone who has read Bostrom and someone who has read Bender, in the same dinner.
What we mean by AGI
The term "AGI" was popularized by Shane Legg, Ben Goertzel, and Marcus Hutter circa 2002. Earlier "strong AI" (Searle) and "HLMI" (high-level machine intelligence) capture overlapping ideas. Operational definitions cluster into three families:
(i) Task-coverage — passes a wide battery of human exams (SAT, MCAT, LSAT, codeforces, ARC), generalizes zero-shot. The OpenAI / Anthropic / DeepMind house style.
(ii) Economic — automates a large fraction (≥50%) of remote-work tasks at expert human reliability. The Metaculus / forecaster favourite.
(iii) Cognitive completeness — possesses long-horizon planning, online learning, agency, theory of mind, embodiment-readiness. Gary Marcus's preferred definition; harder to claim.
The disagreement is consequential: under (i), GPT-4-class models are arguably AGI now. Under (iii), nothing built since 1956 qualifies.
The scaling-law observation
Kaplan et al. (2020) and Hoffmann et al. (Chinchilla, 2022) showed test loss decreases as a power law in parameters, data, and compute. The frontier has spent ~10²⁵ FLOPs as of 2024 (GPT-4-era), with Llama-3.1, Claude 3.5, Gemini Ultra, and Grok-3 all in similar bands.
The scaling hypothesis says: keep going, capabilities continue to emerge. Critics (Bender, Marcus, Mitchell) argue that benchmarks measure pattern-completion not understanding, and that emergent capabilities may be artifacts of evaluation thresholds (Schaeffer et al., 2023).
Four candidate paths to AGI
What people who do this for a living say
| Source | Estimate (median) | Notes | Year |
|---|---|---|---|
| Metaculus (community) | ~2032 for "weak AGI", ~2040 for full | Pulled in dramatically since 2022 | 2024 |
| AI Impacts survey of researchers | 2047 for HLMI (50%) | 2,778 respondents; high variance | 2023 |
| Daniel Kokotajlo · Situational Awareness | 2027–28 plausible | Aggressive scaling extrapolation | 2024 |
| Yann LeCun | Decades; current LLMs are "an off-ramp" | Argues for JEPA-style world models | 2024 |
| Demis Hassabis | "5–10 years" for AGI | Repeated ~2024–25 | 2024 |
| Dario Amodei | 2026–27 plausible for "powerful AI" | Machines of Loving Grace, 2024 | 2024 |
| Ajeya Cotra · Open Philanthropy | ~2050 for transformative AI (median) | Bio-anchors model; updated downward | 2022 |
All estimates are conditional on no compute crackdown, no major war, no ceiling discovered. Variance across these sources is the most honest data point.
The alignment problem, in one paragraph
We can train models to maximize a reward signal. We cannot reliably train them to want what we actually want. As capability increases, the gap between "does what we said" and "does what we meant" becomes more dangerous. The technical research agenda — interpretability, scalable oversight, debate, RLHF, constitutional AI, model organisms of misalignment — is the field's response.
Five canonical concerns
- Specification gaming — clever solutions that satisfy the metric, not the goal
- Goal misgeneralization — same training, different deployment, wrong goal
- Mesa-optimization — learned models that themselves optimize
- Deceptive alignment — models that look aligned during training
- Power-seeking — instrumentally convergent subgoals (Omohundro, Bostrom)
Four scenarios for the 2030s
Who builds, who governs, who studies
| Lab / org | Stance | Notable output |
|---|---|---|
| OpenAI | Mission to build "safe" AGI; capped-profit | GPT-4, o1, Sora, Operator |
| Anthropic | Safety-first frontier lab | Claude, RSP, mechanistic interp |
| Google DeepMind | Research-driven, Hassabis-led | AlphaFold, Gemini, AlphaProof |
| Meta AI | Open-weights, LeCun-led | Llama series, JEPA |
| xAI | Frontier ramp + Twitter integration | Grok, Memphis cluster |
| UK AISI / US AISI | Government safety institutes (2023–) | Pre-deployment evals, model access |
| EU AI Act | Risk-tier regulation, in force 2024–27 | Foundation-model rules, Article 56 |
| FHI / GovAI / MIRI | Academic-adjacent x-risk research | Bostrom, Dafoe, Yudkowsky |
— I.J. Good, Speculations Concerning the First Ultraintelligent Machine, 1965
Good's paper is the foundational text. Vinge (1993) named the resulting threshold "the singularity". Bostrom (2014) gave it a 360-page treatment. Yudkowsky (2008–) gave it the more pessimistic gloss.
Five live policy debates
- Compute thresholds. EO 14110 (US, 2023, partly rescinded 2025): training runs > 10²⁶ FLOPs trigger reporting. Levers exist but evolve.
- Pre-deployment evals. AISI / METR / Apollo Research: dangerous-capability tests before frontier release.
- Open vs closed weights. Llama-style releases vs API-only frontier. Trade-off: democratization vs misuse uplift.
- Liability. If your AI agent transfers funds, who is liable? SB 1047 (CA, 2024) vetoed; the question doesn't go away.
- International coordination. Bletchley (2023), Seoul (2024), Paris (2025) summits. China's Interim Measures (2023). No binding regime yet.
forecast By 2030, compute reporting will be normalized in OECD countries; binding international AI treaty unlikely.
The case it isn't happening soon
Emily Bender & Alex Hanna argue that LLMs are "stochastic parrots" — fluent but ungrounded. The capability claims, they say, are marketing.
Gary Marcus has documented persistent failure modes: planning, robust reasoning, world-model maintenance. Current models still hallucinate, cannot reliably count, fail at simple physical inference.
Yann LeCun argues autoregressive LLMs cannot reach AGI; we need world-model architectures (JEPA, video prediction with abstract latents).
Melanie Mitchell emphasizes the abstraction gap revealed by ARC and similar benchmarks.
Sayash Kapoor & Arvind Narayanan ('AI Snake Oil') stress that "agentic AI" demos do not survive contact with messy real-world deployment.
The skeptics may be right. The scaling lab leaders may be right. They may both be right at different timescales.
Recommended source
Anthropic Talks · "Mechanistic Interpretability: A Field Guide"
A research-grade walk through what we can and cannot currently see inside a transformer. Pair with Chris Olah's "Zoom In" (2020).
youtube.com/@anthropic-ai →Lex Fridman × Demis Hassabis (#475)
Three-hour conversation covering AlphaFold, Gemini, scaling, alignment, and the 5–10-year claim. Slow but high-yield.
youtube.com/@lexfridman →Indicators · 2026–2030
- Frontier compute trajectory — does the 4×/yr trend continue past 10²⁶ FLOPs?
- Long-horizon agent benchmarks (METR, GAIA, OS-World) saturating?
- First mainstream AI-native scientific discovery (post-AlphaFold)
- Mechanistic interpretability scaling to frontier model sizes
- First major model-caused incident requiring regulatory recall
- National-level chip export regimes — Section 232 / EU equivalents
Cite as: The Deck Catalog, Vol 6, Paper 03, 2026. Critique welcome.