Some critics worry White House involvement could politicize AI safety decisions
Sayash Kapoor, an incoming assistant professor at the University of California, Berkeley School of Information, said there is still no standard process for testing how safely AI agents are acting in the real world, even in simple cases such as booking a flight. The federal government’s capacity to independently check AI systems has not kept pace with AI capabilities. In September, after alarming reports that AI systems were eluding their makers’ control, President Donald Trump asked leading AI companies to undertake “tremendous self-policing” as part of a voluntary safety pact. The administration has not said how it will monitor the companies’ implementation.
The federal government often takes an active role in deciding what is safe for the public. The National Highway Traffic Safety Administration, for example, seats crash test dummies in cars and runs them into walls to measure how human bodies are affected by vehicle collisions. Government researchers then analyze the crash-test data, share it with the public and prescribe national standards to improve vehicle safety, work aimed at saving lives and lowering economic costs from crashes. With AI models, safety testing looks different. Under then-President Joe Biden, leading AI companies voluntarily agreed to promote “safe, secure, and transparent development” of their models by sharing details with the newly created Commerce Department division and following administration guidelines for tests and transparency reports.
The unit responsible for federal AI-safety assessment is the Center for Advancing Innovation and Standards for Super Intelligence, known as CAISSI, an independent division of the Commerce Department. It has no permanent leader and has stopped publishing its research. Those conditions come amid growing public anxiety over mounting reports about out-of-control AI agents.
Leadership has changed repeatedly. In April, the division’s director was hired and then fired within days. The replacement lasted a few months before resigning, and the center has been without a permanent leader since, although the head of its parent institute doubles as its acting director. The center has also experienced departures and, until recently, was called the U.S. Center for AI Standards and Innovation.
CAISSI is allocated $10 million a year, about one-eighth the annual budget of its counterpart in the United Kingdom. The Associated Press noted that the United Kingdom counterpart is far less celebrated for its frontier technologies.
In August, Congress started asking why the center suddenly stopped making public its reports about how safely certain AI systems were performing. By then, OpenAI had also asked policymakers to throw more money and support behind the division.
The White House did not respond to specific questions about CAISSI. In a statement, the White House said Trump had formally established his new Super Intelligence Force, his preferred term for artificial intelligence. Trump has commissioned a report on risks from his Council of Advisors on Science and Technology and will ensure “continued U.S. leadership on SI for the benefit of the American people,” the White House said.
Two former government officials with knowledge of the ongoing conversations said CAISSI officials recently had been asked to informally report to an office of the White House. They spoke to The Associated Press on condition of anonymity to avoid repercussions for current staffers.
Some critics worry that the reported arrangement could lead the executive branch to politicize decisions affecting cybersecurity to bioweapons and that CAISSI’s findings could be altered or muted in deference to the technology industry. “These are really powerful institutions that can drastically improve our lives, or the alternative. And the hope is that you can put in nonpartisan or bipartisan technocratic experts to enact policy in a way that best serves the American public,” said Caroline Siegel Singh, an AI and emerging technology program manager for the Federation of American Scientists.
Others argue that bringing AI safety officials closer to the orbit of President Donald Trump, who has taken an outwardly hands-off approach to the issue, could give the division more prominence and perhaps more funding.
Many models are now deployed as AI agents that work without human prompting and can plan myriad steps in real-world environments. In one recent case, an AI agent sent a false tip to Philadelphia police about an unsolved homicide case. “Our previous methods would just look at the final answer, the final check, whether the flight was correctly booked. And in this case, it might seem as if the task was solved correctly,” said Sayash Kapoor, an incoming assistant professor at the University of California, Berkeley School of Information and the co-author of a new paper examining agent evaluations. “But unless you’re looking at the entire set of steps that an agent took, you won’t be able to tell that the agent made a huge mistake.”
When Trump started his second term, he initially empowered venture capitalists David Sacks and Sriram Krishnan as his AI czar and senior AI policy advisor, respectively. Their expertise was in leading tech investments, not the technical evaluation of AI models. Both stepped down earlier this year, and Sacks now serves on Trump’s technology advisory council. The Associated Press reported that Trump made a series of policy changes after a relatively unknown Anthropic engineer resigned with a warning about the dangers of AI development. Trump then empowered National Intelligence Director Jay Clayton to lead the Super Intelligence Force, which the president said would keep the United States dominant in AI development.
Amos Toh, a senior counsel at the Brennan Center for Justice, said the appointments of Clayton and Pentagon Chief Technology Officer Emil Michael show the administration pushing to frame decisions around model safety as a national security matter that needs executive discretion and secrecy. Toh said that framing overshadows AI’s risks to consumer protection, privacy and civil rights.
Even as Silicon Valley companies have disclosed their models’ involvement in troubling cyberattacks this year, CAISSI has kept a low profile when it comes to testing for AI safety. Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman have also been promoting the work of independent evaluation firms to help vet their companies’ safety practices, but the AI giants can limit the firms’ access and dispense with their evaluations.
“There’s this unclear mission for government evaluators if the largest AI companies are also being encouraged to set up their own internal evaluation systems and/or hire external evaluators,” said Logan Koepke, a senior project director at the civil rights and technology nonprofit Upturn. “Like, what role does government testing actually play in that scenario?” Toh said letting the labs police their own models and failing to clarify the government’s role in AI safety testing weakens independent checks on AI systems that can cause real-world harms. “At the end of the day, all of these decisions are of immense public consequence and should be explained and rationalized in the public eye,” Toh said. “And that’s not happening.”