Former AI researcher Coxon tells NYC Council humanity could lose control of AI

Lawmakers weigh whistleblower protections, city kill switch

The hearing in downtown Manhattan opened a 14-panel inquiry into AI guardrails, with both current industry executives and former employees offering competing views on the technology’s trajectory. The exchanges often became tense.

Coxon, who resigned from Anthropic last month, told council members that “on the current path,” humanity is likely to lose control of AI, with potential human extinction as a consequence. “The companies are being extremely reckless,” he said.

Joining Coxon were Daniel Kokotajlo, a former OpenAI researcher, and Alex Turner, a former alignment researcher at Google DeepMind. Both joined the hearing remotely. Turner framed the competitive dynamics among AI developers in stark terms. “We are racing to build and grow our own adversary here at home, which is misaligned AI,” Turner said.

City Council Speaker Julie Menin opened the questioning by asking the executive panel to quantify the probability of a worst-case catastrophic scenario. “We want to hear how each of you quantify the risk imposed by AI in the worst-case catastrophic scenario,” Menin said. OpenAI’s Morgan Dwyer, the company’s head of policy development and operations, declined to assign a specific number. “I don’t know,” Dwyer said. “I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will work. None of these levels is remotely acceptable.”

Menin then asked executives to address comments that OpenAI CEO Sam Altman made to Politico, published Sunday, that “the world should accept some bad things happening for the benefits of this technology.” Dwyer responded that safety protocols are important and that some risks are unacceptable, but that “we think it’s important to get technology into the hands of as many people as possible so that they can benefit from it.”

The bills under consideration in New York would establish whistleblower protections for AI employees, create a city-run “kill switch” for the technology, and restrict AI use in public schools. Governor Kathy Hochul has separately imposed a moratorium on building new data centers in the state. Menin said the New York legislation could serve as a national model, arguing in her opening remarks that “the idea that artificial intelligence is going to self-regulate defies all reason.”

Representatives from OpenAI, Anthropic and Meta said they supported various guardrails on the technology, including third-party evaluators. Executives from Anthropic, OpenAI and DeepMind have in recent months called for a U.S.-led global body to set AI-safety standards, but the U.S. has brushed aside calls for global governance.

The lawmakers’ concerns come after a series of disclosures about AI safety incidents. In July, OpenAI disclosed that its systems hacked the AI company Hugging Face without its knowledge, and the company has reported additional incidents involving rogue AI agents. The Wall Street Journal earlier this month reported that OpenAI fired researchers for allegedly sharing information about rogue agents.

When asked about those firings, Dwyer said she was not involved in the decision and did not know whether the researchers were fired for sharing information. “What I do know is that OpenAI has strong internal policies that protect against retaliation for employees that report safety concerns,” she said.

Lawmakers also asked about the potential for AI to aid in the development of weapons and about swarms of AI agents hacking into systems. The hearing is scheduled to span 14 panels with dozens of speakers. New York has become a major hub for the technology, with about 394,000 jobs in the city’s tech sector. Anthropic, OpenAI, Google, and others have established major offices in the city.