US military nearly intercepted Chinese ship on erroneous AI report, CNN says

Newsom orders AI kill-switch; Sanders, Casar seek superintelligence ban

A CNN report published September 18 described how the United States “almost started a war” with China, based on an intelligence report that claimed a Chinese ship was transporting components of nuclear weapons. Military personnel made plans to intercept the Chinese vessel, complete with aircraft and soldiers ready to board the ship, according to the report. American officials investigated the intelligence further only moments before executing the plan and found that it had been generated with the use of a chatbot and contained erroneous information about the vessel’s contents. The CNN account has not been verified by other major outlets.

The episode is the centerpiece of an October 1 opinion essay in the Guardian by Timnit Gebru, executive director of the research group Dair, and Emily M. Bender, professor of linguistics at the University of Washington. The authors argue that the episode’s lesson is not about “rogue, superintelligent AI” but about “government reliance on error-prone systems that are marketed by their manufacturers as being close to superintelligent.” A military skirmish between China and the United States, they write, “could quickly escalate and spiral into world war three.”

Gebru and Bender write that the biggest international AI news of the preceding three weeks was the resignation of Anthropic engineer Jacob Coxon, who said OpenAI and Anthropic are “racing straight towards self-improving superintelligence and gambling with our lives.” Coxon’s description of a “terminator” scenario — a machine becoming much smarter than humanity and deciding to wipe us out — “captured the public, journalists’ and politicians’ attention,” the authors write, stirring “waves of fear-inducing, credulous reporting” and policy actions.

Those policy actions include a mandate from California Governor Gavin Newsom to create an “AI kill-switch”; an exploration of “AI kill-switches” by New York Governor Kathy Hochul; and legislation introduced by Senator Bernie Sanders and Congressman Greg Casar to “ban artificial superintelligence.” The authors contrast that response with the near-war episode, which, they write, did not galvanize lawmakers into announcing calls for regulation and did not prompt one media outlet after another to warn the public that the United States had almost triggered a war against a nuclear superpower.

Gebru and Bender describe the large language models that power chatbots as “stochastic parrots” — models designed to regurgitate the patterns of the data they are trained on. News coverage describing them as “powerful” “rogue” models that could render humanity extinct all on their own, they write, bolsters perceptions of tools based on these models being infallible enough to merit usage in stakes as high as warfare. CNN was not able to learn which chatbot product was used in the episode, they write, but the LLM powering the product was “likely fine-tuned to output text with the stylistic hallmarks of intelligence reports.” They dispute CNN’s framing of the chatbot as presenting “profound risks” because it is “powerful, new and relatively poorly understood technology.” Today’s LLMs and the chatbots they power are not poorly understood, nor are they powerful in the sense of being effective tools for this use case, they argue; they are systems for generating plausible-looking text, built out of enormous, haphazardly collected datasets and then “fine-tuned” — further trained — to be especially appealing to their intended users. Claims that system behavior is mysterious, they write, “arise from deliberate mystification on the part of those selling those systems, and a misapprehension of what their output is by those procuring and using it.”

The authors cite their 2021 paper “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” and write that both the basic functionality and the risks of such models have been “well-understood and well-documented for years.” Unlike the claims of OpenAI and Anthropic executives and those who repeat their talking points, they write, the harmful and well-documented impacts of these companies’ products have arisen “because people believe that the products are superintelligent, not because they actually are.” They cite “AI” scribes used by hospitals that erroneously classified patients as illicit drug users, and governments that, they write, killed children after relying on “intelligence analysis” systems that mistake schools for military facilities. The marketing campaigns of the companies selling this software warn of impending superintelligence, they write, directing attention away from the real risk toward those companies’ fantasies of doom.

The authors call for regulating usage of systems sold as “AI” — not because they are magical machines, they argue, but because they are error-prone products that should not be used in high-stakes scenarios. “Any automation used in military, medical or other life-and-death situations must be thoroughly evaluated within its use case and paired with usage norms that keep accountability with people who have actual power to make decisions,” they write. They quote former Federal Trade Commission chair Lina Khan’s reminder that “there is no exemption to the law when it comes to AI.” Existing laws, the authors write, can be enforced by federal agencies and other regulators to protect the public against the AI industry’s “unsafe and unscientific practices” of pushing unsound products while marketing them as “magic intelligence in the sky.”

Gebru and Bender argue that going along with company executives and describing their products as beings with agency allows executives to “evade accountability while deceiving the public.” They close by urging journalists and lawmakers to “hold companies accountable rather than parrot their marketing talking points.”