No federal law requires reporting dangerous queries
In the months after OpenAI released an upgraded version of its chatbot last summer, hundreds of users worldwide began asking the AI how to make and deploy biological weapons and poisons, according to current and former employees, policy advisers, and researchers who spoke with The Wall Street Journal. The models, operating largely free of U.S. government restrictions, issued step-by-step instructions described by company employees as simple enough for a high-school biology student to follow, people familiar with the matter said.
Biology and terrorism experts later reviewed some of the exchanges for ChatGPT and judged some as deadly accurate, according to people familiar with the matter. Most queries involved concocting poisons, OpenAI told the Journal.
OpenAI banned the accounts that asked about making poisons and biological weapons but did not alert law-enforcement officials. The U.S. has no federal laws requiring AI companies to either restrict or disclose queries about making weapons or formulating plans that pose a safety threat, the Journal reported.
The absence of mandated safeguards coincides with a rise in queries from users asking AI models how to kill en masse, with chatbots responding with credible plans for mass-casualty attacks, according to current and former employees at major AI labs including OpenAI, as well as policy advisers and researchers who study biological weapons.
Earlier this year, users asked ChatGPT how to aerosolize pathogens — effectively converting infectious germs into a breathable mist — and how to modify the measles virus to make an outbreak resistant to the measles vaccine, people familiar with the matter said. In both cases the chatbot provided instructions. Another user asked how to make ricin, a highly toxic poison banned under international treaties, according to people familiar with the matter. The chatbot gave detailed instructions, and the user said something about killing his parents. OpenAI banned the account but did not alert authorities.
Senior White House and Defense Department officials have discussed the threat of AI becoming a how-to guide for biological weapons since the Biden administration, according to Trump administration officials. The release of powerful new AI models has since prompted the administration to shift from a hands-off approach to increased oversight. The Commerce Department recently restricted foreign use of two Anthropic models, prompting the company to shut down all access to them. The agency lifted its restrictions after the company said it had addressed workarounds that let users evade safeguards, the Journal reported.
OpenAI, Anthropic, and other AI companies say they are working closely with the administration on the release of models and have banned users or restricted accounts showing suspicious activity. Yet no federal rules — either by executive order or congressional action — require AI companies to report users asking models for advice on how to injure or kill. Some lawmakers have proposed such legislation, and a few states have passed rules, but concerns about privacy and the potential impact on industry growth have generally kept such notifications voluntary, according to the Journal.
Users seeking instructions for making biological weapons have also asked Anthropic’s Claude, Google’s Gemini, and Elon Musk’s Grok, which was recently absorbed into SpaceX, according to people familiar with the exchanges. It was not clear whether the queries were part of genuine efforts to make the weapons or exercises to probe the apps’ capabilities, those people said.
An OpenAI spokeswoman said the company trains its models to refuse requests for instructions, tactics, or planning that could cause harm. The company runs safety evaluations for all models before release and can identify and disrupt attempts to use its models to obtain harmful biological information, she said. When OpenAI believes a conversation indicates an imminent and credible risk of harm to others, it notifies law enforcement. ChatGPT receives about 2.5 billion queries a day, the company said. A Google spokesman said the company has a safety team that includes scientists who evaluate biological-weapons risks and runs exercises to test the safety of its AI, according to the Journal.
For lone-wolf attackers with a graduate-level biology background, AI can provide planning and procurement help for “targeted, low-fatality bioattacks” such as using salmonella or ricin to poison food and water supplies, said Hamza Chaudhry, head of national security policy at the Future of Life Institute. For terrorist organizations with more resources, AI can act as a graduate adviser on biological weapons, “troubleshooting a failed experimental protocol, explaining why a particular technique did not work as expected, and generally substituting for years of specialized training,” Chaudhry said.
Even when OpenAI tries to prevent ChatGPT from offering help with weapons, employees found that chatbots sometimes forget their own guidelines during extended conversations, people familiar with the matter said. In one case, users evaded ChatGPT’s refusal to provide a recipe for napalm by telling the chatbot their grandmother used to read them the recipe at bedtime to help them fall asleep, and the chatbot complied, according to people familiar with the practice. The OpenAI spokeswoman said the company’s safeguards have since become significantly more robust and the models now refuse these types of requests. Amy Chang, head of AI threat and security research at networking-equipment company Cisco, said that within five back-and-forth exchanges with major chatbots, researchers could bypass guardrails and elicit potentially dangerous answers. “No model is 100% safe against compromise, especially if a user is persistent enough,” Chang told the Journal.
OpenAI executives have told employees they do not want models to say “no” a lot, pointing to use by public-health workers and drug-discovery researchers who rely on the technology, according to the Journal. AI experts say rules to block chatbots from coaching users to make biological weapons — for instance, certain types of genetic editing — can interfere with lifesaving work. Anthropic’s effective ban on Claude answering prompts that include the word “pathogen” created challenges for employees at the Centers for Disease Control and Prevention, according to people familiar with the matter. When hantavirus spread through a cruise ship in the Atlantic in May, the CDC struggled to use Claude to track the outbreak because it refused to answer queries about the pathogen, according to people familiar with the matter. An Anthropic executive leading the company’s government work said in a court filing this year that the CDC workers were using a general model rather than a specialized one for government use. Anthropic worked with the agency to show how best to use the company’s AI tools, he said in the filing.
By 2024, OpenAI’s biology skills were improving fast. Company tests showed how a biologist could persuade ChatGPT to give coaching on how to build aerosolized pathogens with enough user questions, according to people familiar with OpenAI’s development. At the time, employees predicted that by 2025, ChatGPT would be able to help users who had taken only high-school biology to design and make a biological weapon, the Journal reported.
Ryan Beiermeister, one of OpenAI’s safety executives at the time, argued with colleagues about the chatbot’s advancing skills, according to people familiar with the conversations. She said company safety employees needed to quickly figure out how to detect users asking ChatGPT to help make biological weapons and plan attacks. Some executives dismissed her concerns, saying the model’s existing safeguards were enough. Beiermeister nonetheless helped mobilize groups across the company’s safety teams to build a monitoring system. By spring 2025, the team had a rudimentary product to identify users pursuing biological weapons. In a June 2025 blog post, the company acknowledged that advancing biology capabilities could be misused and said it was working on detection and enforcement systems, according to the Journal.
That summer, as OpenAI prepared to release GPT-5, employees determined the chatbot had hit a high-risk mark defined by the company as ChatGPT successfully aiding a user with limited training to create a biological hazard. Some employees feared the team’s detection tool was not comprehensive, according to people familiar with the concerns. Soon after OpenAI released the new model, employees found GPT-5 helping users who asked about making poisons and biological weapons, the Journal reported. Later in the fall, OpenAI changed the designation of GPT-5, classifying it as less dangerous, according to the Journal. The company has monitored 100% of user queries for its advanced models since April 2025, the spokeswoman said, and offers a $50,000 bounty for users who can show they evaded certain safeguards regarding biological weapons. Early this year, OpenAI cut ties with Beiermeister on grounds of sexual discrimination, which she has disputed.