Walsh's AI-extinction ranking exposes governance mismatch: diffuse societal erosion outranks acute catastrophe

The pattern Toby Walsh surfaces in ranking five AI-caused extinction scenarios — diffuse societal breakdown at the top, acute superintelligence catastrophes at the bottom — is, under one framing, itself the central finding: the rate of AI capability deployment appears to be outrunning the rate at which labor-market, information-environment, political-community, and relational institutions can adapt. Walsh, a professor of AI and research group leader at the University of New South Wales, set out the ranking in an article republished by The Guardian on September 23 and originally published in The Conversation. He wrote that he ordered his five scenarios from “most vague to most precise” and that the list is “also ordered from least to most probable.” Within each scenario, he included what he called the “good news.”

The five scenarios do not function as five competing extinction risks in the conventional probabilistic sense. They function as five surfaces of one underlying dynamic, in which AI deployment pace outruns institutional-response pace. The ordering does not compare probabilities of independent events; it ranks proximity to the present deployment trajectory.

The four lower-probability scenarios share a structural feature: each requires one or more capability leaps that are not yet present. Walsh’s first — the “catch-22” at the center of “AI doomer” arguments — requires a superintelligence whose behavior is, by stipulation, beyond human forecast. Walsh quoted the limitation: “We’d have to be superintelligent to predict what a superintelligence would be able to do. It’s like asking your family dog to imagine thermonuclear war.” His second, the Oxford philosopher Nick Bostrom’s “paperclip maximizer” thought experiment, requires a superintelligence with both the goal architecture and the physical reach to convert available matter into a single product category. Walsh wrote that the scenario “confuses intelligence with power” and noted that converting the planet into paperclip factories would require planning permissions, public-opposition litigation, and active resistance visible in datacenter siting disputes. He added that “you could think of datacentres as a current embodiment of the theoretical paperclip scenario.”

The third and fourth scenarios — AI-designed bioweapons and AI penetration of nuclear command-and-control — sit closer to the present trajectory. Walsh cited Stanford research in which scientists “used a genetic language AI model to synthesize 16 new viruses by sending sequences to a mail-order lab” at a cost of “a couple of hundred thousand dollars at most.” He acknowledged biological constraints on virus-based extinction: “highly transmissible viruses tend to be less lethal” and “highly lethal viruses tend to be less transmissible.” He cited Covid, which “killed less than 1% of humanity,” and the 14th-century Black Death, in which the plague “killed more than one-third of Europe’s population” — adding that “even the plague would be much less deadly today due to our increased medical knowledge and better sanitation.” On the nuclear scenario, Walsh noted that humanity has “come close to nuclear war by mistake several times in the past 50 years,” cited the 2010 Stuxnet computer worm as evidence that air-gapped systems remain vulnerable, and wrote that AI can feed militaries false intelligence “which could lead to irreparable actions.” Nuclear risk, Walsh wrote, primarily operates through the famine caused by nuclear winter rather than direct blast.

The fifth and most-probable scenario — societal breakdown driven by AI-induced job losses, information-environment pollution, political-community fragmentation, and relational disruption through synthetic companionship — requires no capability leap. It requires only that the present trajectory continue. Walsh wrote that under this scenario, “Society might easily break. Slowly but surely, we’d stop being able to support human life at any scale.”

Examining the most-probable scenario under the deployment-versus-governance frame, the first sub-cause is that AI labor displacement is occurring at a rate that exceeds the rate at which labor-market institutions — credentialing, retraining, safety-net design — can be updated. A further sub-cause, extending beyond what the substrate establishes, is an incentive structure in which the actor deploying AI first captures market share, while the institutional costs of displacement are distributed across workers, information consumers, and political communities that do not contract for the deployment. Read under this frame, the misalignment is not technical but incentive-structural: the technology is functioning as designed, and the design is not under coordinated governance. A proposed intervention targeting only the surface — for instance, expanded retraining programs — would not prevent recurrence, because it does not alter the first-deployer incentive.

The forward-projection interaction between Walsh’s scenarios is the central integrated finding: Walsh’s lower-ranked scenarios compound rather than compete with his top-ranked one. If the fifth (societal breakdown) is underway, the institutional capacity to defend against the third (bioweapon) and fourth (nuclear) is plausibly itself reduced. Walsh noted that humanity has “come close to nuclear war by mistake several times in the past 50 years,” but the “by mistake” framing presupposes functioning command-and-control and human-in-the-loop judgment. A society “slowly breaking” toward inability “to support human life at any scale” would, on this reading, plausibly face degraded nuclear command-and-control. The Stanford bioweapon research caps single-vulnerability cost at “a couple of hundred thousand dollars at most,” but a society losing institutional coordination would also plausibly be losing the surveillance and response capacity that contains such events. The probability bands of Walsh’s lower-ranked scenarios appear, on this reading, to rise as a function of the trajectory of his top-ranked one — a divergence point the ranking framework itself cannot price, because it treats the five scenarios as competing rather than compounding. This compounding reading extends beyond what Walsh’s ordering explicitly states.

Walsh closed by cataloguing the “good news” embedded in each scenario: superintelligence remains distant, bioweapons face biological constraints, nuclear stockpiles have declined, and human pushback against unchecked AI infrastructure has begun. “There are some things to be worried about for sure,” Walsh wrote. “But not to be too worried, I hope.” The hedging in that closing line is itself analytically informative: Walsh’s probability ordering depends on the continuation of current institutional pushback at a rate that matches AI deployment. Under the deployment-versus-governance frame, that is the assumption most exposed to the dynamic his own ranking describes.

Analytical techniques used in this piece

This analysis applies the methods below. Each links to a short, plain-English explainer you can read and reuse.

Root-Cause Analysis

Traces a symptom back along its causal chain to the conditions that actually generated it.

Wicked Futures

Explores a long-horizon, deeply entangled future with no clean resolution.

Wicked Problems

Treats a problem as wicked — no stopping rule, no clean test of success, every attempt consequential.