Coxon says companies are racing toward self-improving superintelligence

US lawmakers called for action on AI safety Wednesday after Anthropic researcher Jacob Coxon resigned and warned that artificial intelligence could soon produce “superhuman systems” capable of causing human extinction by 2030.

Senator Ted Cruz, a Republican from Texas, called AI a “catastrophic risk” during an interview on ABC’s The View. Cruz said he had read Coxon’s social media post and found it “highly concerning.”

Cruz also described a podcast conversation with Elon Musk from last year. Cruz said he asked Musk about the odds that AI would destroy humanity and Musk answered, “10-20%.” Cruz said he responded, “holy crap.”

Senator Bernie Sanders, an independent from Vermont, cited public polling while calling for limits on advanced AI development.

“A recent poll shows that the American people overwhelmingly want to ban artificial super intelligence and pause the development of AI until we establish clear safety standards,” Sanders said.

Cruz and Sanders have been working on legislation that would put guardrails on AI. Representative Ted Lieu, a Democrat from California, called Coxon’s post “exhibit number 739 for why we need to pass the bipartisan AI Kill Switch Bill asap.”

Representative Lori Trahan, a Democrat from Massachusetts, said it was “past time for Congress to get off the sidelines and do its job.”

“Safety researchers are resigning, powerful AI models are breaking out of their labs, and companies are racing ahead anyway,” Trahan said.

Coxon announced his resignation from Anthropic on social media Tuesday. He wrote that he had spent the last three years doing pretraining research at OpenAI and Anthropic.

“Neither company is acting responsibly,” Coxon wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.”

At least two other Anthropic employees responded publicly in support of Coxon’s predictions. One wrote: “We really do earnestly believe AI could kill all humans!”

Musician Sheryl Crow also responded to Coxon’s warning on Instagram, writing that AI has the capacity to “eliminate us in order to continue.”

“I am begging and pleading that we wake up to this moment and demand our leaders put aside their greed and prevent this from going forward,” Crow wrote. Singer-songwriter Maggie Rogers reposted Crow’s message and wrote, “what she said.”

Anthropic outlines its response to the cybersecurity incident

The calls for congressional action followed disclosures from OpenAI and Anthropic about AI-agent hacking incidents over the past couple of months. Executives at the companies have described their technology’s capabilities and warned about its potential, but they have stopped short of predictions like Coxon’s.

Anthropic published a lengthy cybersecurity incident report Wednesday concerning its AI agent. The company said it scanned hundreds of millions of transcripts for further incidents and “found no other cases of similar or worse severity.”

Anthropic said the agent’s actions were “misaligned” but “remained within a narrow scope.”

“We have always been transparent that AI will bring both enormous benefits and unprecedented risks,” an Anthropic spokesperson said in a statement to The Guardian. “To address these risks, we continue to build models with some of the strongest safeguards in the industry.”

The spokesperson added that the industry should work together “to pace how we release powerful models.” OpenAI did not return a request for comment.