Public listing pairs Anthropic’s shareholders with its public-benefit mandate

Anthropic researcher Jacob Coxon resigned in September, criticizing the safety practices of Anthropic and OpenAI. He wrote that Anthropic and OpenAI were “gambling with our lives” and warned that some people working on AI development believed the technology could threaten human life by the end of the decade. His message spurred industry-wide conversations about the speed of AI’s growth as Anthropic reached a $1.75 trillion market value in its first day of Wall Street trading, following an offering that broke the record for the biggest public stock offering ever.

A recent report from the nonprofit Model Evaluation and Threat Research came from at least one of the independent evaluations discussed in Dario Amodei’s essay. The report found Anthropic was developing its systems around twice as quickly as it had been a year earlier, largely because AI itself was helping with the work. The findings underscored Coxon’s concerns about the pace of AI development.

In the essay responding to Coxon’s warning, CEO Dario Amodei called for an industry-wide slowdown in AI development. He enumerated detailed plans for slowing development and coordinating across companies and countries, and championed independent evaluators embedded within Anthropic and other laboratories. Those elements reach beyond Anthropic’s own operations: an industry-wide slowdown and cross-company, cross-country coordination require decisions Anthropic does not control, while evaluators inside other laboratories require those laboratories’ participation.

Anthropic operates as a public benefit corporation, a legal structure that obligates it to balance shareholder interests with the public good. Companies with that structure rarely trade publicly, and those that do are significantly smaller than Anthropic. The company has said it is working toward a future where “AI’s benefits are shared by everyone.” It continues to pledge to prioritize humanity’s benefit, and the listing leaves it under pressure from shareholders to generate profits from technology its most senior leaders have described as posing existential threats.

Five years before the IPO, Anthropic began as a Silicon Valley startup. Its seven founders were Dario Amodei, 43; his sister and Anthropic president Daniela Amodei, 39; and five other former OpenAI employees. They left OpenAI with their own ideas about how to develop artificial intelligence, seeking to build a business that more clearly prioritized safety. Anthropic dedicated itself to prioritizing safety, built the Claude chatbot and warned that the same technology could bring catastrophe. Anthropic has publicly committed to safety and transparency around its AI models. Insiders have raised alarms in recent months.

In a CNBC interview Thursday, Amodei described rapid AI advances as arising in large part from advanced models helping to build the next improved version of themselves. He compared the change to building something that lets a driver go 200 miles per hour after previously driving at 60, and said Anthropic was working to develop the technology safely. He described the push and pull between safety and AI advancement as one of the “central tensions at the heart of the company.”

Amodei described the company’s pacing approach this way: “What we’ve done with pacing is essentially said, ‘Look, you know, we could drive at 200 miles an hour. Maybe let’s only drive at 90 miles an hour, spend time figuring out how to do that safely,’” he said. “And the rest of that surplus, we spend building the model in the right way.”

In a filing with the Securities and Exchange Commission last month, Anthropic devoted an unusually lengthy 93 pages to the risk factors it had to disclose to investors. Among the concerns were that advanced AI systems could pose “catastrophic or existential risks to humanity.”

Some people have argued that Anthropic has outlined how its technology and competitors’ systems could have devastating consequences in the name of sheer transparency. Critics have questioned both the emphasis Anthropic places on those dystopian-like possibilities and the company’s positioning of itself as the logical solution given its safety-minded focus.

A September “Saturday Night Live” sketch featured cast member Jane Wickline portraying Dario Amodei. She said: “Sam Altman, myself, all the major AI players, we have talked about this extensively, and we are all on the same page here: We do not condone what we are doing.” The faux interview ended with: “I urge you to urge me to stop.” The line appeared to resonate with skeptics and showed up on a protest sign outside Anthropic’s San Francisco headquarters on Thursday.

At the Nasdaq opening bell Thursday, Amodei said the company’s “most difficult safety work lies ahead of us.” He added: “And to state it unequivocally, we will pace the training and deployment of AI systems, taking as long as is necessary to have confidence that they are safe and will remain under human control.”

The IPO created a concentration of extreme wealth for Anthropic’s founders, employees and other investors. Forbes reported that before the offering, Dario Amodei and Daniela Amodei each had a net worth of just over $14 billion. By Thursday afternoon, Dario Amodei’s net worth was $22.7 billion and Daniela Amodei’s was $22.8 billion.

The Amodei siblings and the other co-founders have said they are committed to pledging 80% of their equity to philanthropy, although the commitment is voluntary. Anthropic runs an equity donation match program that allows employees to pledge equity and matches those pledges. The company has also pledged at least 5% of its stock to a “public benefit” fund.

Speaking to CNBC on Thursday, Daniela Amodei said the public benefit corporation concept “spoke” to the co-founders when they initially created the company. “We’re really trying to do something that we do think can have this incredible economic value and impact, but we want it to be baked into essentially our structure that we also have to balance that against building things that are good for people,” she said.

The combination leaves Anthropic in an unusual public position: its filing warns that advanced AI systems could pose “catastrophic or existential risks to humanity,” while Dario Amodei has called for slower development and the co-founders have said they are committed to pledging 80% of their equity to philanthropy. Shareholders now hold a public claim on the same enterprise.