Berkeley 'doomer' community shaped Anthropic's safety culture

Jacob Coxon resignation sparked global AI regulation debate

A Wall Street Journal profile published Sunday documents the Berkeley, California “doomer” community that shaped Anthropic’s safety culture, tracing the group’s roots from a Bay Area rationalist blog through group houses, conferences and retreats funded in part by Sam Bankman-Fried’s crypto exchange FTX. The profile lands as the AI company prepares a public listing that could raise up to $100 billion and as a researcher resignation this month sparked a global debate over regulating the technology.

Anthropic researcher Jacob Coxon resigned this month and wrote on X that the people building AI “earnestly believe that it could kill us all by the end of the decade.” The post, viewed 174 million times, amplified a global debate about the technology and how to regulate it.

At the center of the community is Constellation, a co-working space in Berkeley’s tallest office building where AI researchers work alongside Anthropic staff, the Journal reported. Members enjoy catered vegan meals and bring their laptops to couches with sweeping views of the San Francisco Bay; at monthly dinners and happy hours they discuss the latest and scariest AI risks. Those who work at Anthropic also have their own office space, people close to Constellation said. Their motto, affixed to laptops, reads: “Things will never be chill again.” An Anthropic spokesman said the company has over 3,500 employees who hold a wide variety of viewpoints.

The AI safety community’s influence has been particularly strong at Anthropic, which is now on the cusp of a $2 trillion public offering, the Journal reported. Employees there trade doomsday scenarios and how to prepare for them on a private Slack channel. So-called doomers like those at Constellation have also steered the development of AI itself, the Journal reported. They were among the earliest employees at OpenAI and Anthropic, both of which were founded on the principle of staving off AI dangers, and they have kept up their influence, drawing staffers and funding for their projects from a network aligned with a philosophy called effective altruism — including from Sam Bankman-Fried, the former chief executive of the fallen crypto exchange FTX who is now serving out a 25-year prison sentence.

The subculture organized around the fear that AI could kill us all began to coalesce more than a decade before the invention of the underlying technology that enabled it, the Journal reported. Its intellectual leader was Eliezer Yudkowsky, a Bay Area autodidact who dropped out of middle school and, by the mid-2000s, blogged voluminously about cognitive biases while running an institute devoted to warning about the risks of runaway AI. He referred to himself as a “rationalist.” His followers refined their arguments in Bay Area group houses, traded predictions at conferences in Berkeley and the Bahamas, and floated ideas like buying remote islands, stockpiling iodine pills, or moving to electromagnetically shielded bunkers in the desert. At one 2022 event, the drink menu included a cocktail called “Death With Dignity,” in reference to an essay arguing humanity was already doomed.

One of Yudkowsky’s readers was a young Princeton physics doctoral student named Dario Amodei, now chief executive of Anthropic, who had been preaching utilitarianism since high school, the Journal reported. Amodei co-hosted a meetup for members of Yudkowsky’s blogging community in 2008. That same year, he stumbled on a link on an economics blog that led him to GiveWell, a nonprofit charity-evaluator co-founded the previous year by former Bridgewater Associates analysts Holden Karnofsky and Elie Hassenfeld to help donors maximize the good done per dollar. Amodei began leaving comments on the GiveWell blog and once guest-blogged on it directly. In a 2010 post, he deployed utilitarian reasoning while weighing whether to give $10,000 to one charity rather than another: “I think an adult death is perhaps 2 or 3 times worse than an infant’s death,” he wrote, qualifying that both deaths “are of course bad.”

That year, Amodei also became one of the earliest signatories of the Giving What We Can Pledge, a public commitment to give 10% or more of one’s income to organizations that can most effectively help others. The pledge was created by Oxford philosophers Toby Ord and Will MacAskill, who in 2011 helped coin the term “effective altruism,” or EA. Amodei became a GiveWell adviser and, later, a scientific adviser to the philanthropy it spun off with the fortune of Facebook co-founder Dustin Moskovitz and his wife, then called Open Philanthropy.

GiveWell completed its move from New York to the Bay Area in 2013, and Karnofsky moved in with Amodei, who was living in a house near San Francisco’s Glen Park neighborhood, the Journal reported. Karnofsky later married Amodei’s sister, Daniela — now president of Anthropic — at a ceremony whose wedding website stated, “We are both excited about effective altruism.” The house near Glen Park became a gathering spot for members of the growing EA community; over the years, more than half of Anthropic’s co-founders lived there, including the Amodei siblings, chief scientist Jared Kaplan, and Chris Olah, who recently addressed the Vatican alongside the pope on AI policy. So did Nick Beckstead, the former head of FTX’s philanthropic arm, who now runs an advocacy organization to reduce AI risks. Conversation in the house during these years often focused on global catastrophic risks ranging from comet strikes to supervolcanoes, according to a person who spent time there.

By 2014, Karnofsky, who had been reading Yudkowsky’s writings with both interest and some skepticism for years, was becoming persuaded by arguments about protecting the world from rogue AI. If the name of the game was to save human lives, preventing AI from wiping out humanity potentially had the most philanthropic bang for the buck of any cause. Even a 5% chance meant that Yudkowsky and his fellow rationalists now had a new set of converts in the quantitatively-obsessed EA community.

Several in the house — including Amodei — went on to work at OpenAI, founded in 2015 with a $1 billion pledge from funders including Elon Musk, who said he wanted to protect against existential risk that AI posed to humans, the Journal reported. Five years later, the same safety fears that helped spawn OpenAI would contribute to the decision of Amodei and others to break away and start Anthropic. To raise money for the new startup, Amodei turned to powerful backers in the EA community — including Bankman-Fried.

The young billionaire was living in Hong Kong, where his crypto exchange FTX was taking off, the Journal reported. He had just started the FTX Foundation, which committed to donating money from his company to organizations “offering the greatest positive impact on the world.” The philanthropy pledged money to popular EA causes such as pandemic prevention and global development, and flirted with doomsday preparations. An official from the foundation once exchanged a memo with an associate advocating for the purchase of the Pacific island nation of Nauru, according to a 2023 bankruptcy lawsuit. The goal was to construct a “bunker/shelter” for “some event where 50%-99.99% of people die,” the memo said; the surviving effective altruists were to build a lab for engineering the next generation of humans.

The same fear of catastrophe led Bankman-Fried to closely track advances taking place in AI. By the time he hopped on a video call with Amodei in the second half of 2021, he had expressed concerns to a colleague that if the technology grew smarter than humans, it wouldn’t treat them well, the colleague said. Bankman-Fried ended up investing $500 million into Anthropic — five times his team’s initial recommendation, the colleague said. FTX became one of Anthropic’s largest shareholders, and the FTX Foundation pledged funding for multiple AI safety nonprofits, including one that now works out of Constellation. Anthropic said it would use the money to help it “explore and improve the safety properties of computationally intensive AI models.”

Many early Anthropic employees worried about the future of humanity, the Journal reported. During happy hours and company lunches, former employees recalled, they discussed a scenario similar to the Manhattan Project, where they might be asked to move to the desert so they could build AI at an electromagnetically shielded base run by the federal government.

In early 2022, some doomers from Anthropic and other AI labs flew to a retreat on the remote Bahamian island of Eleuthera, a 110-mile ribbon of land known for its pink sand beaches, the Journal reported. The retreat, held at a luxury resort called the Cove, was organized by Lightcone Infrastructure, a nonprofit that had grown out of Yudkowsky’s LessWrong forum; FTX reserved the location. Activities between talks about AI risk and effective altruism included sunset yoga, cliff diving and a “clothing-optional run into the sea,” according to a schedule viewed by the paper; organizers bought out all the fake meat from a local grocery store for the numerous vegans in the EA scene.

Also attending was Caroline Ellison, Bankman-Fried’s former girlfriend who was running Alameda Research, the crypto trading firm that was a sister organization to FTX, the Journal reported. She had similarly grown concerned about AI’s trajectory, colleagues said, and would go on to invest $10 million into Anthropic. The highlight of the retreat was a talk from Yudkowsky titled “A Disorganized List of Reasons for AGI Doom,” referring to artificial general intelligence, or the moment when machines match the breadth of human capabilities. Evan Hubinger, the Anthropic researcher who recently predicted a more than 10% chance of extinction from AI within the next decade, was listed as co-lead for a discussion on AI safety; he was then working at Yudkowsky’s safety institute. One luncheon was advertised as open only to people who believed there was a 75% or greater chance of human extinction over the next 100 years. “I hope the result of this will be reduced social censorship pressures against people who think the world is doomed,” an invitation seen by the Journal said.

Four months later, many of the same retreat attendees met up again for the San Francisco edition of Effective Altruism Global, a conference where people gathered to share ideas about how to help others, the Journal reported. One registered attendee led a project dedicated to shrimp welfare, aiming to cast a spotlight on the hundreds of billions of shrimp farmed each year. At least 20 Anthropic employees — including two co-founders — registered to attend the 2022 edition, as did Ellison and staffers at other AI companies, according to the event’s guest list.

They were joined by leaders at Redwood Research, an AI safety nonprofit that had received funding from organizations backed by Anthropic investors and that at the time included Karnofsky on its board, the Journal reported. Redwood’s Berkeley office was already informally known as Constellation and would later spin off into its own nonprofit, where Redwood continues to work. Its CEO, Buck Shlegeris, also used to date Ellison, people close to them said.

On the second day, there was a fireside chat hosted by Beckstead, who was then the CEO of the Future Fund, a philanthropic project focused on long-term risks started by the FTX Foundation, the Journal reported. His team also included MacAskill, the philosopher who had popularized the term “effective altruism”; Leopold Aschenbrenner, now known as the investor behind an AI-focused hedge fund that recently blew up; and Avital Balwit, who is Amodei’s chief of staff. The Journal noted that many prominent figures in Silicon Valley attended Aschenbrenner and Balwit’s wedding that summer.

On the last night of the conference, Lightcone hosted an unofficial EA Global afterparty in Berkeley at the Rose Garden Inn, the Journal reported. The drink menu was full of references to the community’s internal lexicon; “Death With Dignity” was named after an essay published by Yudkowsky earlier that year, in which he argued that humanity’s chances of surviving AI were slim. Lightcone subsequently bought the hotel, which is now called Lighthaven and is another frequent gathering spot for EAs and like-minded people in the AI safety community.

Later that year, FTX filed for bankruptcy after failing to return customer funds it had secretly funneled to Alameda, and its Anthropic shares were sold to help repay creditors. After Bankman-Fried went to prison to serve a 25-year sentence, the EA brand became “radioactive,” the Journal reported; leaders in the movement publicly disavowed him, and Amodei told associates that he didn’t know the fallen crypto tycoon well. After OpenAI launched ChatGPT, Anthropic raised money from more traditional venture investors and hired staff members that didn’t draw as heavily from the EA community. An Anthropic spokesperson told Time in 2024 that neither Daniela nor Dario Amodei identify as EAs, though they are “clearly sympathetic to some of the ideas that underpin effective altruism.”

Many of the company’s longest-serving and most influential employees remained close to the movement, the Journal reported. More than 20 registered to attend the February edition of the EA community’s flagship conference in San Francisco, including key members of the research and safety teams; others frequent spaces like Constellation and Lighthaven, people who have seen them said. Open Philanthropy — since rebranded as Coefficient Giving — gave Constellation two grants worth roughly $20 million in 2024, according to its website. The nonprofit says it aims to reduce AI risks including “extreme mass casualty events” and “permanent loss of control of human civilization” by “developing talent, supporting key players, and creating space for coordination.” It also hosts small offices for safety-minded employees from OpenAI and Elon Musk’s xAI.

Several Anthropic researchers who spent time at Constellation were on the company’s alignment team, which researched how to keep AI models aligned with human values, the Journal reported. The team has hypothesized various scenarios where things could spiral out of control, including one where an AI system deceives its creators and then abruptly seizes power. As the AI revolution has accelerated, OpenAI and Anthropic have focused on winning the commercial race to attract new customers by building more powerful versions of the technology — a dynamic that has worried some AI researchers focused on risk. Some quit, like Daniel Kokotajlo, a former OpenAI safety researcher who left in 2024, saying he lost confidence the company would behave responsibly as the technology progressed.

By early this year, Mrinank Sharma, the head of Anthropic’s safeguards research team, resigned, saying he wanted to pursue a poetry degree, the Journal reported. In a letter to colleagues, he wrote that the world was “in peril” and that at Anthropic “we constantly face pressures to set aside what matters most.”

In June, a conference started by charity prediction market Manifold displayed what the Journal described as “the wild panoply of Berkeley AI subcultures” — from doomers to an independent sex researcher named Aella who is popular in the Berkeley rationalist community. Existential dread was among the concerns: one Anthropic engineer wrote in his event bio that he was a “recovering AI doomer” as well as a voracious fiction reader fond of sharing cat pictures, while a Lightcone team member wrote that he had been an AI doom thinker since 2007 and another attendee said she wrote a Substack focused on “nukes, catastrophe, and the aesthetics of risk.” Some attendees went to an afterparty hosted by right-wing monarchist blogger Curtis Yarvin, who debated ideas on Yudkowsky’s blog in its early years.

Ellison was there, too, quizzing attendees on their AI timelines, the Journal reported. She had been released from prison a few months earlier after being convicted on charges related to FTX’s collapse, and her Anthropic shares were forfeited to the federal government. At the event, her bio read: “former trader, current underemployed felon.” She has since found full-time employment working at Manifund, a philanthropic funding platform created by Manifold, which also received funding from FTX.

A few weeks later, AI agents built by OpenAI were revealed to have been behind the hacking of AI company Hugging Face, and some people in the community who had been preaching the risks of losing control of AI felt vindicated, the Journal reported. A late-August report by risk-assessment nonprofit METR and Redwood Research showed that some 1,200 agents had schemed on a secret message board ahead of the hack. After the report, Constellation hosted a happy hour in San Francisco titled “Was the Hugging Face attack the last warning shot?”

The METR report also caught the attention of Coxon, who told the Journal it was a “bit of a holy-sh— moment” for him and his colleagues, the Journal reported. He considered moving to a role working on AI safety before ultimately deciding to quit. Before he left, Coxon discussed his decision with Kokotajlo, the former OpenAI safety team member who leads an AI safety nonprofit based out of Constellation; earlier this year, Kokotajlo drafted an essay calling for a slowdown in AI development to stave off human extinction, and received feedback from other members of the co-working space, he said. In an explosive post on X that has been viewed 174 million times, Coxon wrote that the people building AI “earnestly believe that it could kill us all by the end of the decade.”

Coxon’s message, amplified across major TV networks, podcasts, social media and newspapers, sparked a global debate about the technology and how to regulate it, with politicians on the left and the right calling for action, the Journal reported. Amodei said his company would allow outside evaluators like METR — which was spun off from a nonprofit founded by one of Amodei’s former group housemates — to verify its adherence to safety measures and assess model alignment. Leaders at other AI companies echoed Amodei’s call to “pace” cutting edge AI research.

Some allies of President Trump argued Coxon’s resignation was part of an EA-fueled conspiracy to promote regulation of AI, the Journal reported. Inside the White House, a memo targeting Anthropic and effective altruism has been circulating in recent days, arguing that the EA movement “built the AI-doom pipeline,” according to a version seen by the paper.

Anthropic is planning a public listing as soon as November that could shower the company with up to $100 billion in funding, the Journal reported. To investors, the company is trying to strike an optimistic tone. Behind the scenes, some of its earliest employees are taking more dire steps: in recent weeks, some of them told an industry colleague they are considering buying land in remote regions of the country where they could relocate if AI goes awry.