A civil engineer who told you there was a ten-percent chance your child’s school would collapse this year would not conclude that the country should stop building schools. He would specify the failure mode, isolate the dangerous component, test the structure, and fix what the test revealed.
Anthropic scientist Evan Hubinger has put the odds of artificial intelligence causing human extinction within the next decade above ten percent. Anthropic co-founder Jack Clark has said a mandatory, third-party-verified “kill switch” may eventually need to be imposed across the industry. The proposal sounds prudent until you ask the tradesman’s question: who holds the switch, who certifies the hand holding it, and whose machinery gets shut down?
Most frontier laboratories already have ways to stop or restrict their systems. Clark’s proposal is not that the capability should exist. It is that Washington should make one particular control mechanism unavoidable.
That distinction is the whole story.
Anthropic chief executive Dario Amodei has proposed a three-point slowdown plan. Sam Altman and Elon Musk have publicly backed deceleration and additional safeguards. Microsoft has published an outline document on what it called “humanist AI.” The language differs. The structure does not. Every firm already leading the race is asking for rules that would slow the firms behind it, while preserving commercial advantage for the firms writing the rules.
This is not a safety movement. It is incumbent protection dressed in humanitarian clothing.
A kill switch can be sensible inside a particular system, under a defined threat model, with a defined operator, an auditable failure condition, and a clear procedure for appeal and restoration. A federal requirement for a third-party-controlled switch across an entire industry is something else. It creates a permission regime. It gives designated auditors, agencies, and politically connected contractors authority over infrastructure that American companies have spent billions building.
Answerable to whom?
Certified by whom?
Under what technical standard?
Those are not procedural details. They are the system.
A frontier model is not a light switch sitting in a cupboard. It is a distributed stack of trained weights, hardware, data pipelines, deployment interfaces, access controls, and operators. “Turn it off” can mean disabling a public endpoint, revoking a model licence, blocking compute, cutting access to weights, or taking physical data-centre capacity offline. Those are different interventions with different failure modes. A proposal that collapses them into the friendly phrase “kill switch” has not yet reached the engineering stage. It has reached the lobbying stage.
The same applies to third-party verification. An independent auditor is not independent merely because the word appears in the title. Someone funds the auditor. Someone appoints it. Someone defines the threshold at which a model becomes “too dangerous.” Someone decides whether the threshold was crossed. Someone receives the power to issue the order.
The frontier firms already possess the models, the compute, the lawyers, the policy staff, and the relationships with regulators. A rule that raises the cost of entry does not burden all firms equally. It burdens the firm that has not yet entered.
That is what regulation does when it is designed around incumbents rather than risks. It turns a technical safeguard into a moat.
Cory Doctorow’s analysis of enshittification begins with a useful distinction: the problem is not that executives suddenly became worse people. The problem is that competition, regulation, interoperability, and labour stopped constraining what they could do. Remove the friction and the same profit-seeking firm can move from serving users to extracting value from users, suppliers, and the public.
The AI safety coalition is proposing to reinstall one kind of friction for everyone except, in practice, the companies already equipped to absorb it.
A regulated industry is not automatically a cartel. Safety rules can prevent real harm. The trouble begins when the firms with the greatest commercial stake in the rules also supply the risk model, the technical vocabulary, the proposed enforcement mechanism, and the definition of compliance. That is regulatory capture: private incumbents using public rule-making to control the conditions under which competitors may exist.
The companies are not even hiding the constraint. Amodei’s proposal includes the qualification that deceleration must occur “without sacrificing commercial advantage.” That phrase should be printed on the front of every policy document in which it appears. It means the labs want to slow the open frontier while their own closed frontier continues shipping.
This is the old enclosure mechanism with GPUs attached.
The humanist outline, the slowdown plan, the mandatory kill switch, and the Washington rally are not separate events. They are the same institutional move repeated through different mouths: convert a contested commercial strategy into a moral emergency, then ask the state to enforce it.
The political reinforcement is unusually broad. Bernie Sanders has warned of the “existential threat” posed by uncontrolled AI growth. Steve Bannon and Sanders are scheduled to share a stage at a “Pro-Human Assembly” in Washington. Microsoft has supplied its own moral vocabulary. The firms supply the technical vocabulary. The result is a coalition that disagrees about nearly everything except the need to put someone in charge of the frontier.
That agreement does not prove the risk is imaginary. It proves the proposed remedy has become politically useful.
Donald Trump’s response has been crude, but his central objection is closer to the material question than the hand-wringing is. He has warned that restrictions on AI and data centres would benefit China and written that “WHOEVER WINS AI, WINS!” The slogan is not a policy. It is not even a complete analysis. But it identifies the strategic cost that the safety campaign keeps treating as an externality: a unilateral American slowdown changes the relative position of Chinese firms whether or not the American rule was written with China in mind.
Chinese state media has already supplied the accompanying interpretation. It reported that Washington’s anxiety over China’s rapid AI development was distorting American policy priorities, citing Fudan University professor Xin Qiang. When Beijing’s public commentators are pleased that American companies want their government to slow American development, the result does not automatically validate every American critic of the proposal. It does, however, establish the direction of the benefit.
The benefit runs somewhere.
The safety advocates will say that Chinese competition is no reason to build unsafe systems. They are correct. “China is doing it” is not a technical safety case, and national competition cannot excuse reckless deployment. A model that fails in a hospital, a financial market, a power grid, or a public-benefits system does not become safe because its competitor is Chinese.
But the reverse is also true. “Safety” is not a magic word that exempts a proposal from antitrust analysis. A safeguard must be judged by its mechanism, its operator, its scope, its error costs, its appeal process, and its distributional effects. If the same rule makes entry harder for small laboratories, open-source developers, foreign newcomers, and domestic challengers while imposing manageable costs on the largest firms, then the rule has a competition effect whether its authors acknowledge it or not.
The question is not whether the executives are frightened. The question is what their fear would purchase.
The companies asking Washington to impose deceleration are not neutral observers of the market. They are market participants with a direct financial interest in the speed, shape, and ownership of the next phase. Their safety arguments may contain genuine technical concerns. Their commercial incentives remain genuine as well.
That is not an ad hominem attack. It is the first page of the ledger.
The danger of the present debate is that it turns uncertainty into authority. No one can write a complete specification for “human extinction caused by AI” and then prove that a federal kill switch would reduce that probability by a known amount. The claim may be morally serious. It is not yet an engineering requirement.
In cryptographic protocol verification, the difference between “this appears to work” and “this has been shown to resist a defined adversary” is the entire discipline. You name the protocol. You name the threat model. You state the properties to be preserved. You test the system against the attacker you actually expect, not the attacker who makes the press release sound most impressive.
The AI safety coalition is asking the public to accept the conclusion before it has supplied the specification.
That is the technical version of a blank cheque.
The better policy is not to let the laboratories govern themselves and call the result safety. It is to publish the safety research, require evidence for concrete deployment claims, subject high-risk systems to narrow and auditable controls, preserve open scientific scrutiny, and refuse to write one firm’s risk model into federal law. It is to enforce competition rather than suspend it. It is to prevent proprietary systems from becoming the only systems permitted to exist.
Open standards matter here. Interoperability matters. Independent audits matter when the auditor’s power, funding, scope, and appeal process are visible. A narrowly defined emergency stop may matter when it is tied to a specific system and a specific failure. None of those propositions requires granting Anthropic, Microsoft, OpenAI, or any other incumbent the authority to design the market in their own image.
The frontier is not too dangerous to enter.
It is too valuable to let the current occupants fence it off.