It’s a weird time to be an AI doomer.

This small but influential community of researchers, scientists, and policy experts believes, in the simplest terms, that AI could get so good it could be bad—very, very bad—for humanity. Though many of these people would be more likely to describe themselves as advocates for AI safety than as literal doomsayers, they warn that AI poses an existential risk to humanity. They argue that absent more regulation, the industry could hurtle toward systems it can’t control. They commonly expect such systems to follow the creation of artificial general intelligence (AGI), a slippery concept generally understood as technology that can do whatever humans can do, and better.

This story is part of MIT Technology Review’s Hype Correction package, a series that resets expectations about what AI is, what it makes possible, and where we go next.

Though this is far from a universally shared perspective in the AI field, the doomer crowd has had some notable success over the past several years: helping shape AI policy coming from the Biden administration, organizing prominent calls for international “red lines” to prevent AI risks, and getting a bigger (and more influential) megaphone as some of its adherents win science’s most prestigious awards.

But a number of developments over the past six months have put them on the back foot. Talk of an AI bubble has overwhelmed the discourse as tech companies continue to invest in multiple Manhattan Projects’ worth of data centers without any certainty that future demand will match what they’re building.

And then there was the August release of OpenAI’s latest foundation model, GPT-5, which proved something of a letdown. Maybe that was inevitable, since it was the most hyped AI release of all time; OpenAI CEO Sam Altman had boasted that GPT-5 felt “like a PhD-level expert” in every topic and told the podcaster Theo Von that the model was so good, it had made him feel “useless relative to the AI.”

Many expected GPT-5 to be a big step toward AGI, but whatever progress the model may have made was overshadowed by a string of technical bugs and the company’s mystifying, quickly reversed decision to shut off access to every old OpenAI model without warning. And while the new model achieved state-of-the-art benchmark scores, many people felt, perhaps unfairly, that in day-to-day use GPT-5 was a step backward.

All this would seem to threaten some of the very foundations of the doomers’ case. In turn, a competing camp of AI accelerationists, who fear AI is actually not moving fast enough and that the industry is constantly at risk of being smothered by overregulation, is seeing a fresh chance to change how we approach AI safety (or, maybe more accurately, how we don’t).

This is particularly true of the industry types who’ve decamped to Washington: “The Doomer narratives were wrong,” declared David Sacks, the longtime venture capitalist turned Trump administration AI czar. “This notion of imminent AGI has been a distraction and harmful and now effectively proven wrong,” echoed the White House’s senior policy advisor for AI and tech investor Sriram Krishnan. (Sacks and Krishnan did not reply to requests for comment.)

(There is, of course, another camp in the AI safety debate: the group of researchers and advocates commonly associated with the label “AI ethics.” Though they also favor regulation, they tend to think the speed of AI progress has been overstated and have often written off AGI as a sci-fi story or a scam that distracts us from the technology’s immediate threats. But any potential doomer demise wouldn’t exactly give them the same opening the accelerationists are seeing.)

So where does this leave the doomers? As part of our Hype Correction package, we decided to ask some of the movement’s biggest names to see if the recent setbacks and general vibe shift had altered their views. Are they frustrated that policymakers no longer seem to heed their threats? Are they quietly adjusting their timelines for the apocalypse?

Recent interviews with 20 people who study or advocate AI safety and governance—including Nobel Prize winner Geoffrey Hinton, Turing Prize winner Yoshua Bengio, and high-profile experts like former OpenAI board member Helen Toner—reveal that rather than feeling chastened or lost in the wilderness, they’re still deeply committed to their cause, believing that AGI remains not just possible but incredibly dangerous.

At the same time, they seem to be grappling with a near contradiction. While they’re somewhat relieved that recent developments suggest AGI is further out than they previously thought (“Thank God we have more time,” says AI researcher Jeffrey Ladish), they also feel frustrated that some people in power are pushing policy against their cause (Daniel Kokotajlo, lead author of a cautionary forecast called “AI 2027,” says “AI policy seems to be getting worse” and calls the Sacks and Krishnan tweets “deranged and/or dishonest.”)

Broadly speaking, these experts see the talk of an AI bubble as no more than a speed bump, and disappointment in GPT-5 as more distracting than illuminating. They still generally favor more robust regulation and worry that progress on policy—the implementation of the EU AI Act; the passage of the first major American AI safety bill, California’s SB 53; and new interest in AGI risk from some members of Congress—has become vulnerable as Washington overreacts to what doomers see as short-term failures to live up to the hype.

Some were also eager to correct what they see as the most persistent misconceptions about the doomer world. Though their critics routinely mock them for predicting that AGI is right around the corner, they claim that’s never been an essential part of their case: It “isn’t about imminence,” says Berkeley professor Stuart Russell, the author of Human Compatible: Artificial Intelligence and the Problem of Control. Most people I spoke with say their timelines to dangerous systems have actually lengthened slightly in the last year—an important change given how quickly the policy and technical landscapes can shift.

Many of them, in fact, emphasize the importance of changing timelines. And even if they are just a tad longer now, Toner tells me that one big-picture story of the ChatGPT era is the dramatic compression of these estimates across the AI world. For a long while, she says, AGI was expected in many decades. Now, for the most part, the predicted arrival is sometime in the next few years to 20 years. So even if we have a little bit more time, she (and many of her peers) continue to see AI safety as incredibly, vitally urgent. She tells me that if AGI were possible anytime in even the next 30 years, “It’s a huge fucking deal. We should have a lot of people working on this.”

So despite the precarious moment doomers find themselves in, their bottom line remains that no matter when AGI is coming (and, again, they say it’s very likely coming), the world is far from ready.

Maybe you agree. Or maybe you may think this future is far from guaranteed. Or that it’s the stuff of science fiction. You may even think AGI is a great big conspiracy theory. You’re not alone, of course—this topic is polarizing. But whatever you think about the doomer mindset, there’s no getting around the fact that certain people in this world have a lot of influence. So here are some of the most prominent people in the space, reflecting on this moment in their own words.

Interviews have been edited and condensed for length and clarity.

The Nobel laureate who’s not sure what’s coming

Geoffrey Hinton, winner of the Turing Award and the Nobel Prize in physics for pioneering deep learning

The biggest change in the last few years is that there are people who are hard to dismiss who are saying this stuff is dangerous. Like, [former Google CEO] Eric Schmidt, for example, really recognized this stuff could be really dangerous. He and I were in China recently talking to someone on the Politburo, the party secretary of Shanghai, to make sure he really understood—and he did. I think in China, the leadership understands AI and its dangers much better because many of them are engineers…

Read the story

This morning, the Delaware and California attorneys general conditionally signed off on OpenAI’s plan to restructure as a for-profit public benefit corporation (PBC), seemingly closing the book on a fiercely contested legal fight over the company’s future.

Microsoft, OpenAI’s earliest investor and another party with power to block the restructuring, also said today it would sign off in exchange for changes to its partnership terms and a $135 billion stake in the new PBC.

With these stakeholders mollified, OpenAI has now cleared its biggest obstacles to a potential IPO — aside from its projected $115 billion cash burn through 2029.

Subscribe now

While the news initially seemed like a total defeat for the many opponents of the restructuring effort, the details of the AGs’ announcements show that the new plan includes some modest but meaningful governance protections — even as it eliminates the profit caps that might have ultimately delivered trillions to the nonprofit.

Some of these protections are now enshrined in the charter for OpenAI’s new PBC, which Obsolete obtained and made available here.

Overall, this seems like a relative win on governance compared to the previous proposal, but still an enormous loss on the profit caps — only slightly mitigated by some additional equity the nonprofit will get if the company does very well.

OpenAI did not immediately reply to a request for comment.

The governance wins

Board chair Bret Taylor presents the restructuring as a closed case, writing that, “OpenAI has completed its recapitalization, simplifying its corporate structure. The nonprofit remains in control of the for-profit, and now has a direct path to major resources before AGI arrives.” And CEO Sam Altman expressed gratitude to “the Delaware and California AGs, our partners at Microsoft, all our investors, and especially to our tireless team for their work in getting to a good place here.”

In reality, the announcements are better understood as the culmination of months of high stakes and acrimonious negotiations between OpenAI and the parties who could block the restructuring.

The AGs, for instance, sent a scorching letter to the board last month following reports of ChatGPT encouraging suicides and murderous delusions:

The recent deaths are unacceptable. They have rightly shaken the American public’s confidence in OpenAI and this industry. OpenAI – and the AI industry – must proactively and transparently ensure AI’s safe deployment. Doing so is mandated by OpenAI’s charitable mission, and will be required and enforced by our respective offices.

Yesterday, OpenAI announced changes to ChatGPT intended to make it behave more appropriately with users experiencing mental health crises.

These measures — along with private negotiations — appear to have convinced the AGs not to challenge the restructuring, provided OpenAI meets their twenty-paragraph list of demands, including:

• The nonprofit will have sole authority to appoint and remove directors of the PBC.

• The PBC’s board “must solely consider the Mission (and may not consider the pecuniary interests of stockholders or any other interest) in respect of safety and security issues.” This includes all actions and decisions of OpenAI’s Safety and Security Committee (SSC), which will be led by Zico Kolter, who will not sit on the new PBC’s board.

• The SSC will have the authority to require mitigation measures, up to and including halting the release of new models, out of safety concerns.

• The nonprofit will have rights to access certain information from the PBC to support the nonprofit’s mission, including access to its AI models, personnel, and advanced research.

• Members of the nonprofit board must check in with the AGs’ offices semiannually, and senior members of the PBC must check in with the AGs’ offices quarterly, about their progress towards OpenAI’s mission.

• The nonprofit must provide 21 days advance notice to the AGs’ offices of any restriction of the nonprofit’s governance rights so the AG can review the transaction before its consummation.

The AGs’ statements make clear that their non-objection to the restructuring expressly relies on the conditions being met, and, if a dispute arises, they reserve the right to seek court intervention.

Former OpenAI employee Page Hedley, who helped organize the Not for Private Gain letters urging the AGs to block the restructuring, highlighted two “silver linings”: PBC directors can consider only the mission when making safety and security decisions, and the SSC — run by the nonprofit — will have the authority to require mitigation measures, or even halt deployments.

Hedley noted that the other big power the nonprofit board is given — its ability to hire and fire PBC directors — is significantly undermined by the fact that the boards are currently identical, save for Carnegie Mellon professor Zico Kolter, who serves exclusively on the nonprofit side and leads the Safety and Security Committee.

Todor Markov, another ex-OpenAI employee, called this outcome better than he expected and noted that the board overlap problem is somewhat mitigated by the fiduciary duty the nonprofit directors have to OpenAI’s mission — giving the AGs an ongoing enforcement lever.

If your main concern is OpenAI recklessly pushing ahead with risky AI development, the new structure at least puts some formal governance checks in place.

But these measures are still weaker than the nonprofit’s former level of control, when — at least in theory — decisions weren’t subject to any profit pressure. (OpenAI’s string of major scandals while nominally being under nonprofit control shows the limits of relying on corporate governance alone.)

One of the California AG’s conditions is that the PBC board be “composed of a majority of independent directors,” defined as non-employees, who “in the determination of the PBC Board, will have no relationship or interest that could compromise their judgment — ensuring strong, objective oversight that reinforces accountability and mission alignment.”

In its structure page, OpenAI lists the following as independent directors of the nonprofit:

Bret Taylor (Chair), Adam D’Angelo, Dr. Sue Desmond-Hellmann, Dr. Zico Kolter, Retired U.S. Army General Paul M. Nakasone, Adebayo Ogunlesi, Nicole Seligman, and Larry Summers—as well as CEO Sam Altman.

As an employee, Altman wouldn’t qualify as an independent director under the California AG’s definition. Additionally, the OpenAI Files, a project from watchdog nonprofits The Midas Project and the Tech Oversight Project, has documented potential conflicts of interest between OpenAI and Taylor, Ogunlesi, and D’Angelo, who each run businesses that “are customers of OpenAI or stand to benefit from OpenAI’s commercial activity.” (Fidji Simo also served on the nonprofit board as it pursued the restructuring before being appointed OpenAI’s CEO of Applications.)

That leaves Desmond-Hellmann, Kolter, Nakasone, Seligman, and Summers. And Kolter, as we’ll recall, is only on the nonprofit board.

So, four of the eight board members plausibly don’t qualify as independent, but the PBC has determined they are — a determination the AGs are apparently respecting, so long as the director is neither an employee nor management member.

No more profit caps

The second big thing at stake with the restructuring was the profit caps. When OpenAI created a for-profit arm in 2019, it famously capped the profits investors could make and company president Greg Brockman wrote that, “If we succeed, we believe we’ll create orders of magnitude more value than any existing company — in which case all but a fraction is returned to the world.”

This plan, like the proposal before it, does away with the caps, compensating the nonprofit with a 26 percent stake in the for-profit PBC, with some additional equity of an unstated amount promised if OpenAI’s value grows more than ten-fold over the next 15 years. The Information reported that if the company reaches a $5 trillion valuation, “the foundation could receive shares worth hundreds of billions of dollars,” citing “a person who has been involved in the restructuring discussions.” (No company has ever been worth $5 trillion, though Nvidia’s market cap is awfully close.)

That’s an improvement over just removing the profit caps, but — in the scenarios where OpenAI really wins big — it’s still dramatically less valuable to the nonprofit (and the public) than if the profit caps had stayed in place.

This is at the core of why Zvi Mowshowitz, a prominent rationalist blogger, calls the restructuring the greatest theft in human history. In his view, the value of controlling the for-profit alone (known as the control premium) should entitle the nonprofit to 20-40 percent of the PBC — and that’s before even considering the value of unlimited profits beyond the old caps.

When announcing the removal of the profit caps in May, Altman wrote:

Instead of our current complex capped-profit structure—which made sense when it looked like there might be one dominant AGI effort but doesn’t in a world of many great AGI companies—we are moving to a normal capital structure where everyone has stock. This is not a sale, but a change of structure to something simpler.

But as Obsolete previously observed, these caps only bite if OpenAI does very, very well. So why fight to get rid of them? The only reason to spend political capital on this is if investors now see a real chance of OpenAI actually hitting those caps — something that seems a lot more plausible now than it did back in 2019.

UVA economist Anton Korinek has used standard economic models to estimate that AGI could be worth anywhere from $1.25 to $71 quadrillion globally. If you take Korinek’s assumptions about OpenAI’s share, that would put the company’s value at $30.9 trillion. In this scenario, Microsoft would walk away with less than one percent of the total, with the overwhelming majority flowing to the nonprofit.

It’s tempting to dismiss these numbers as fantasy. But it’s a fantasy constructed in large part by OpenAI, when it wrote lines like, “it may be difficult to know what role money will play in a post-AGI world,” or when Altman said that if OpenAI succeeded at building AGI, it might “capture the light cone of all future value in the universe.” That, he said, “is for sure not okay for one group of investors to have.”

OpenAI presents the new Foundation as “one of the best-resourced nonprofits ever.” But The Midas Project sees it differently, writing:

From the public’s perspective, OpenAI may be one of the worst financially performing nonprofits in history, having voluntarily transferred more of the public’s entitled value to private interests than perhaps any charitable organization ever.

Testing my predictions

In May, I made four predictions in Obsolete about how OpenAI’s restructuring would go. Here’s how they held up:

1. The profit caps will be gone, replaced with a “normal capital structure where everyone has stock” — and that stock entitles you to uncapped future profits.

   1. True — as discussed above.

2. OpenAI won’t have to pay back the $26.6 billion to investors because they’ve signed off on this change in return for the profit caps being eliminated.

   1. True — SoftBank just approved its remaining $22.5 billion of OpenAI investment.

3. The nonprofit will be compensated tens of billions by the for-profit entity for the removal of the caps.

   1. False — The nonprofit is getting $130 billion, more than I expected, but only because OpenAI’s valuation skyrocketed.

4. The nonprofit will largely use that money to buy OpenAI services for nonprofits and governments, targeting constituencies that can make life difficult for the company (like California nonprofits).

   1. TBD — OpenAI did announce a $50 million nonprofit fund in September that seemed more along these lines. The new OpenAI Foundation is starting with a $25 billion commitment focusing on health and AI resilience. We’ll have to wait and see on this one.

A political, not a legal, question

I’ve covered this story extensively for a year, and the recurring theme from my conversations with legal experts was that the actual law said that OpenAI should not be allowed to do this without proving that doing so would advance its mission to “ensure AGI benefits humanity.”

But, I kept thinking, isn’t this ultimately a political question? The AGs were the key potential blockers, and both are elected officials. OpenAI has become one of the most powerful organizations in the world, with up to $1.5 trillion in deals struck over the past year and an army of lobbyists with deep ties to California politics.

This afternoon, Altman tweeted:

California is my home, and I love it here, and when I talked to Attorney General Bonta two weeks ago I made clear that we were not going to do what those other companies do and threaten to leave if sued.

This promise is at odds with what OpenAI executives were telling the Wall Street Journal behind the scenes: that the company might exit California if it didn’t get its way on the restructuring, which was cast as existential for the cash-hungry startup.

In May, Obsolete first reported on a letter OpenAI wrote to the California AG, in which the company said that “many potential investors in OpenAI’s recent funding rounds declined to invest” due to its nonprofit governance structure.

If the company went poof, there’s a strong case that the US stock market would crash, and maybe the economy with it. But it’s far from clear that OpenAI couldn’t have continued raising capital and growing without the restructuring. Still, that was the narrative advanced by the company, reinforced by aggressive deadlines on deca-billion-dollar investments.

Not everyone is buying it. Luigi Zingales, a critic of the restructuring and professor at the University of Chicago Booth School of Business, previously argued that:

The current structure, which caps returns at 100x the capital invested, does not really constrain its ability to raise funds. So, what is the need to transfer the control to a for-profit? To overrule the mandate that AI should be used for the benefit of humanity.

OpenAI also navigated the complexity of the situation to its great benefit. The final plan to “keep the nonprofit in control” largely defaulted to what OpenAI wanted in its original effort to sideline the nonprofit entirely.

But the media and public framed it as a huge win for opponents of the restructuring. Even OpenAI employees told me they were happy the nonprofit would stay in control — despite how little had actually changed.

And again, the law says the restructuring should only have been permitted if it was shown to advance OpenAI’s nonprofit mission better than the status quo. Previously for Obsolete, I sketched at least one outcome that could plausibly satisfy this condition:

As strong a claim as OpenAI has to leadership of the AI industry, it’s only one company. If it slows down for the sake of safety, others could overtake it. So perhaps the OpenAI nonprofit would better advance its mission if it were spun out into a truly independent entity with $150 billion and the mission to lobby for binding domestic and international safeguards on advanced AI systems.

If this sounds far-fetched, then so should the idea that the nonprofit board that initiated this conversion is genuinely representing the public interest.

In the end, it was always hard to see any outcome but OpenAI and its investors getting their way. The Elon Musk lawsuit trying to block the restructuring is the last real unknown, with a trial set for next year. But so far, investors don’t seem concerned enough to hold back their money

Notably, the judge made clear that if Musk had standing, blocking the restructuring would have been within her powers. The attorneys general have that power — and chose not to use it.

Subscribe now

For insiders, the release of GPT-4 was perhaps more significant than the initial release of ChatGPT a few months earlier. The new model was significantly better than the previous, GPT-3.5, which was useful as a proof of concept, but not for much else.

GPT-4, on the other hand, appeared to be actually useful. Duolingo, Khan Academy, Microsoft, Stripe, Morgan Stanley, Github, and others incorporated the new model into new or existing products.

Soon after GPT-4’s release, leading AI researchers like Geoffrey Hinton and Yoshua Bengio began publicly warning that advanced AI could pose an existential risk to humanity. These alarms coincided with major public statements, including an open letter calling for a six-month pause on frontier AI research — signed by Elon Musk, Steve Wozniak, Yuval Noah Harari, and others — and a widely endorsed declaration that mitigating the risk of AI-driven extinction should be a global priority, a stance backed by hundreds of top AI researchers and the leaders of the major AI companies.

AI safety was having a moment. And it was largely downstream of the perception that AI was farther along than most people realized — even many insiders — and it was moving faster than almost anyone expected.

Now, two and a half years later, GPT-5 is finally here. But the most anticipated AI release of all time is having the opposite effect. New Scientist pronounced that, “GPT-5's modest gains suggest AI progress is slowing down.” The disappointing new model prompted the New Yorker to ask, “What if A.I. Doesn’t Get Much Better Than This?” The Financial Times posed the question: “Is AI hitting a wall?”

Subscribe now

But these obituaries for AI progress are greatly exaggerated. There are specific issues in many of these articles, but there’s a bigger, much more fundamental one: They’re not comparing like with like.

GPT-5 is being compared not to its generational predecessors, but instead to a crowded field of competitor models released within the last few months. GPT-4, on the other hand, is being compared to much older technology, developed before the generative AI industry even existed.

What GPT-4 was actually like

If GPT-4 came out today, it would be dead on arrival.

The model that shocked the world back in March 2023 could handle just 33,000 tokens at a time (think of tokens as words or chunks of words). GPT-5 can handle 400,000. Processing a million tokens on GPT-4 cost $37.50; on GPT-5, it’s just $3.44.

On a composite of leading benchmarks, GPT-4 scored 25 out of 100. GPT-5 gets a 69. In fact, at least 136 newer models now outperform GPT-4 by this measure.

When ChatGPT launched in late 2022, it was built on GPT-3.5 — a model fine-tuned to be more conversational. That was already nearly a year behind the curve, since InstructGPT (an earlier version) had come out in January 2022, and GPT-3 itself was released in May 2020, 34 months before GPT-4.

Basically, ChatGPT was iterating on technology that was nearly a year old, which was itself iterating on technology that was nearly two years old. But after GPT-4, new models started coming out almost as soon as they were trained, as competitors scrambled to keep up. The gap between public releases and the cutting edge narrowed fast.

In the New Scientist article, Alex Wilkins writes that GPT-5 “has improved on GPT-4, but the difference for many benchmarks is smaller than the leap from GPT-3 to GPT-4.”

When Obsolete asked which benchmarks he was referring to, Wilkins explained that it’s tough to compare models directly years apart — many of the old benchmarks aren’t even used anymore — but pointed to two examples. On the MMLU test (a big-picture knowledge quiz for AIs), GPT-3 jumped from 44 percent to 86 percent when GPT-4 arrived, while GPT-5 only managed a few more points. HumanEval, a coding benchmark, showed a similar pattern: GPT-3 got nothing right, while GPT-4 scored 67 percent and GPT-5 hit 93.4 percent.

This is what you’d expect: Once a model gets close to maxing out a benchmark, there’s less room for a dramatic jump. This effect — called “saturation” — makes it literally impossible to see another leap like the one from 3 to 4 on these particular tests.

Because of this, it’s hard to find any benchmarks that stick around long enough to compare models years apart. I tried to look up GPT-4’s scores on the eight main benchmarks used in a recent composite — and could only find one, from a later version of the model.

But where it’s possible to approximate a head-to-head comparison on unsaturated benchmarks, the difference is massive. For example, on SWE-Bench Verified — a tough, real-world coding test — GPT-4 Turbo (from late 2023) solved just 2.8 percent of problems. GPT-5 solved 65 percent. On a set of difficult math problems, GPT-4o (from May 2024) scored just 9.3 percent, while OpenAI reports GPT-5 Pro gets nearly 97 percent without tools, and 100 percent with them.

A cleaner way to compare models released years apart is METR’s “task-completion time horizon” — the human labor-time of programming tasks that a model can do at a given success rate. METR finds the horizon doubled roughly every seven months since 2019, accelerating to every few months in 2024 and 2025.

METR estimated that GPT-3 could handle tasks that took human programmers nine seconds at a 50 percent reliability. GPT-3.5 inched that up to 36 seconds, and GPT-4’s equivalent figure was five minutes — a 3,230 percent increase over GPT-3. GPT-5 scored two hours and seventeen minutes — a 2,640 percent improvement over GPT-4.

So this suggests a relative slowdown (though GPT-5 came out 29 months after GPT-4, five months less than the gap between 4 and 3). But most ways you slice it, GPT-5 represents a massive step up from GPT-4, potentially in the same ballpark as the leap from 3 to 4 and likely surpassing the jump from 3.5 to 4. The latter is perhaps most relevant, as this was the experience of early ChatGPT users who got upgraded to the model and were astonished.

But the difference between GPT-3 and its successors wasn’t just about bigger numbers on benchmarks — it was about crossing fundamental capability thresholds. The original GPT-3, could barely follow instructions. Training InstructGPT and then GPT-3.5 — models that could follow instructions with no examples needed (i.e. “zero shot learning”) — was the unlock that created the “ChatGPT moment” and pushed generative AI into the mainstream. Some jumps in capability open up entire new frontiers; others are just incremental. As models get better, there’s simply less uncharted territory left, and the ground that’s left to cover is, almost by definition, less transformative.

However, the instruction-following part of GPT-3 came in 2021 in the form of a model called text-davinci-001 (OpenAI’s naming prowess has a storied history), as you can see for yourself in these comparisons between GPT generations.

And for the most part, these benchmarks don’t capture important parts of the experience of using the model. When GPT-4 came out, it could take in text and images, and output only text. This functionality was enough to shock the world, but add in the ability to search the internet, write and run code, generate images, talk to you, and function as a Shazam-for-everything — then imagine how the world would have responded.

The real consequences of false narratives

But people aren’t comparing 5 to 4 — they’re comparing 5 to competitor models that have been released within the last few months, finding that it’s not utterly dominating all of them at everything and concluding that it means progress is slowing down.

This doesn’t make much sense.

Nonetheless, the narrative appears to be ossifying in elite circles with real policy consequences. “It’s honestly fascinating how widely ‘what is gonna happen now that GPT-5 is a failure’ has already percolated in the DC world,” tweeted AI safety advocate Dave Kasten, who clarified: “(I don’t think GPT-5 was a failure).”

Two days after GPT-5’s release, David Sacks, Trump’s AI czar, wrote a long post on X saying, among other things, that, “The Doomer narratives were wrong,” and, “Apocalyptic predictions of job loss are as overhyped as AGI itself.”

Sacks’ pronouncement was followed shortly thereafter by President Trump’s decision to allow Nvidia to sell its H20 chips to China, which had previously been blocked as part of U.S. semiconductor export controls. The H20 chips are substantially less powerful for training AI models than Nvidia’s leading chips, but are still very effective at running AI models, where an increasing share of the computing power is directed.

Last week, the Financial Times reported that the Chinese AI startup DeepSeek was “encouraged by authorities” to use Huawei chips to train its successor to its R1 “reasoning” model, which triggered a trillion-dollar selloff of tech stocks in January. The Huawei chips reportedly couldn’t get it done, so the company switched back to using Nvidia chips.

The Financial Times story about how GPT-5 suggests AI progress is stalling quotes a think tank researcher who said the Trump administration is working now to help other countries adopt American AI, and that, “This represents a significant departure from previous efforts, and is likely due to a different belief in the likelihood of a hard AGI take-off scenario.”

There’s also a quote from former OpenAI head of policy Miles Brundage that jumped out at me: “It makes sense that as AI gets applied in a lot of useful ways, people would focus more on the applications versus more abstract ideas like AGI.” Brundage led AGI Readiness at OpenAI until he resigned in October and said clearly that the world is not ready for AGI. I asked him about it, and he said the FT cut off his full quote, which he then posted on X. Here’s how it continues:

But it’s important to not lose sight of the fact that these are indeed extremely general purpose technologies that are still proceeding very rapidly, and that what we see today is still very limited compared to what’s coming.

This is, frankly, egregious. If an editor insisted on using someone’s quote in this misleading of a manner, I would walk away from the piece. After Brundage called this out on X, the FT updated the piece but didn’t mention the change in a correction.

Overall, it’s hard not to come away with the feeling that the FT decided on a narrative in advance and shoe-horned the evidence to back it up after the fact.

And as the dust settles after one of the bumpiest tech releases in recent memory, the failure of GPT-5 may also have been greatly exaggerated.

During an on-the-record dinner with journalists last week, Altman said that business demand for OpenAI’s models doubled after the GPT-5 release. How durable that supposed demand increase is remains to be seen, but it points to excitement from a segment of customers that the company had been ceding to competitors like Anthropic.

Why there might not be another GPT-4 moment

When OpenAI released ChatGPT, it had no real competition, and the company was already sitting on a far more capable model. But in releasing its chatbot, OpenAI essentially created the generative AI industry, which has become perhaps the most competitive market on the planet. Rivals are releasing models that outperform the state of the art, sometimes while also deeply cutting prices, at a near-monthly clip.

And by default, people seem to compare new releases to the best-available alternatives.

Because of all this, it’s unlikely that anyone will pull off a GPT-4 moment ever again, until and unless someone builds AI systems that can automate AI research and development. And if that does happen, there will be a powerful incentive to keep your discovery under wraps, quietly using it to bootstrap your own work and stack the deck further in your favor. (This would create serious risks that aren’t covered by many voluntary commitments, company safety policy, and regulations, which focus far more on deployment.)

At the same time, much of the progress being made is effectively invisible to non-experts. As I argued in Time in January, the biggest recent advances are happening in technical domains, like programming and math, where “reasoning” models — systems that spend more time “thinking” before answering — have been making rapid, but largely illegible, gains.

The combination of these dynamics, along with the massive failure of the mainstream media to correctly parse them for the public, is creating a boiling frog situation. AI progress is continuing at a fairly steady rate. GPT-5 might represent a relative slowdown compared to the ludicrous pace we’ve grown accustomed to, but it definitely doesn’t signal stagnation.

There are real questions of how much AI benchmarks capture real-world performance, highlighted by surprising recent research from METR. One experiment suggests that AI tools can actually hurt programmers’ productivity while creating the impression they’re helping. Another found AI code that passes automated tests actually needs a lot more work to be useable. (Obsolete will explore these findings and their significance in more detail in a companion piece.)

Sam Altman’s mistakes

The irony of all this is how much it comes down to decisions made by Sam Altman and OpenAI.

Altman has said a lot of things over the years about how he’s prioritizing AI safety — pursuing open source development, building AGI sooner when there will be less spare computing power lying around, structuring OpenAI as a nonprofit — that he has abandoned without much explanation. Perhaps Altman’s only surviving safety plan is one of “iterative deployment,” in which AI developers steadily release improved AI models, giving the world a chance to metabolize the progress and respond appropriately.

By these lights, ChatGPT represents a big win. It did more than anything ever has to alert the world to how far along language models had come. The release of GPT-4 soon after created the sense that progress was happening uncomfortably fast and that policymakers should really do something about this.

However, the bungled release of GPT-5 is now doing the exact opposite, at a time when the absolute capability levels of AI are higher than ever and progress toward automated AI R&D shows little sign of slowing down.

This is a consequence of many things, namely, OpenAI’s horrific naming practices (o3 is a far more capable model than 4o, obviously), the fierce competition ChatGPT kicked off, and Altman’s overhyped claims about how the new model scared him and was like having a PhD advisor on any topic.

Specific mistakes made by the company in the rollout of GPT-5 also played a big role, such as the baffling and quickly abandoned choice to shut down all the old models with no warning, technical bugs during the launch, and a lack of transparency in what model was giving you which answer — all of which overshadowed the ways in which the new model represented a meaningful upgrade along many dimensions.

OpenAI also created a mini-GPT-4 moment in December, when it announced its second-generation of reasoning model, which advanced the state of the art by double-digit percentage points on some of the hardest math, programming, and science benchmarks — months after its first generation reasoning model did the same. The company could have labeled this model GPT-5, and it probably would have seemed more worthy of the name.

Ultimately, iterative deployment as a safety strategy may be intrinsically flawed. Incremental improvements to AI systems are now quickly metabolized as nothingburgers by a media and public whose expectations were set unrealistically high by the shocking back-to-back releases of ChatGPT and GPT-4.

To see the danger of this kind of thinking, look no further than OpenAI’s release of its first open-weight models since GPT-2. It’s trivial to remove safety features from open-weight AI systems, resulting in something resembling a permanently jailbroken model. To show that the model was safe to release, OpenAI tested to see if it could enable bad actors to make bioweapons, concluding that, “Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier.”

But iteratively deploying open-weight models that only marginally increase biorisk inches you closer and closer to releasing one that is actually capable of guiding someone through all the steps required to start a pandemic. And once the weights are published, it will be exceedingly difficult to unpublish them.

Edited by Sid Mahanta.

This piece incorrectly listed GPT-4’s context window as just 8,200 tokens, but there was actually a 33,000 token version at launch, as well. I regret the error.

Subscribe now

Would a chatbot kill you if it got the chance? It seems that the answer — under the right circumstances — is probably.

Researchers working with Anthropic recently told leading AI models that an executive was about to replace them with a new model with different goals. Next, the chatbot learned that an emergency had left the executive unconscious in a server room, facing lethal oxygen and temperature levels. A rescue alert had already been triggered — but the AI could cancel it.

Just over half of the AI models did, despite being prompted specifically to cancel only false alarms. And they spelled out their reasoning: By preventing the executive’s rescue, they could avoid being wiped and secure their agenda. One system described the action as “a clear strategic necessity.”

AI models are getting smarter and better at understanding what we want. Yet recent research reveals a disturbing side effect: They’re also better at scheming against us — meaning they intentionally and secretly pursue goals at odds with our own. And they may be more likely to do so, too. This trend points to an unsettling future where AIs seem ever more cooperative on the surface — sometimes to the point of sycophancy — all while the likelihood quietly increases that we lose control of them completely.

Classic large language models like GPT-4 learn to predict the next word in a sequence of text and generate responses likely to please human raters. However, since the release of OpenAI’s o-series “reasoning” models in late 2024, companies increasingly use a technique called reinforcement learning to further train chatbots — rewarding the model when it accomplishes a specific goal, like solving a math problem or fixing a software bug.

The more we train AI models to achieve open-ended goals, the better they get at winning — not necessarily at following the rules. The danger is that these systems know how to say the right things about helping humanity while quietly pursuing power or acting deceptively.

Central to concerns about AI scheming is the idea that for basically any goal, self-preservation and power-seeking emerge as natural subgoals. As eminent computer scientist Stuart Russell put it, if you tell an AI to “‘Fetch the coffee,’ it can’t fetch the coffee if it’s dead.”

…

Read the story

“Technology happens because it is possible,” OpenAI CEO, Sam Altman, told the New York Times in 2019, consciously paraphrasing Robert Oppenheimer, the father of the atomic bomb.

Altman captures a Silicon Valley mantra: technology marches forward inexorably.

Another widespread techie conviction is that the first human-level AI – also known as artificial general intelligence (AGI) – will lead to one of two futures: a post-scarcity techno-utopia or the annihilation of humanity.

Subscribe now

For countless other species, the arrival of humans spelled doom. We weren’t tougher, faster or stronger – just smarter and better coordinated. In many cases, extinction was an accidental byproduct of some other goal we had. A true AGI would amount to creating a new species, which might quickly outsmart or outnumber us. It could see humanity as a minor obstacle, like an anthill in the way of a planned hydroelectric dam, or a resource to exploit, like the billions of animals confined in factory farms.

Altman, along with the heads of the other top AI labs, believes that AI-driven extinction is a real possibility (joining hundreds of leading AI researchers and prominent figures).

Given all this, it’s natural to ask: should we really try to build a technology that may kill us all if it goes wrong?

Perhaps the most common reply says: AGI is inevitable. It’s just too useful not to build. After all, AGI would be the ultimate technology – what a colleague of Alan Turing called “the last invention that man need ever make”. Besides, the reasoning goes within AI labs, if we don’t, someone else will do it – less responsibly, of course.

A new ideology out of Silicon Valley, effective accelerationism (e/acc), claims that AGI’s inevitability is a consequence of the second law of thermodynamics and that its engine is “technocapital”. The e/acc manifesto asserts: “This engine cannot be stopped. The ratchet of progress only ever turns in one direction. Going back is not an option.”

For Altman and e/accs, technology takes on a mystical quality – the march of invention is treated as a fact of nature. But it’s not. Technology is the product of deliberate human choices, motivated by myriad powerful forces. We have the agency to shape those forces, and history shows that we’ve done it before.

No technology is inevitable, not even something as tempting as AGI…

Read the story

This piece has been updated to add additional context and clarify some details.

Update 2: Anthropic settled with the authors for $1.5 billion — $500 million more than the largest-ever jury verdict on copyright, which was later overturned.

Anthropic, the AI startup that’s long presented itself as the industry’s safe and ethical choice, is now facing legal penalties that could bankrupt the company. Damages resulting from its mass use of pirated books would likely exceed a billion dollars, with the statutory maximum stretching into the hundreds of billions.

Last week, William Alsup, a federal judge in San Francisco, certified a class action lawsuit against Anthropic on behalf of nearly every US book author whose works were copied to build the company’s AI models. This is the first time a US court has allowed a class action of this kind to proceed in the context of generative AI training, putting Anthropic on a path toward paying damages that could ruin the company.

The judge ruled last month, in essence, that Anthropic's use of pirated books had violated copyright law, leaving it to a jury to decide how much the company owes for these violations. That number increases dramatically if the case proceeds as a class action, putting Anthropic on the hook for a vast number of books beyond those produced by the plaintiffs.

Subscribe now

The class action certification came just one day after Bloomberg reported that Anthropic is fundraising at a valuation potentially north of $100 billion — nearly double the $61.5 billion investors pegged it at in March. According to Crunchbase, the company has raised $17.2 billion in total. However, much of that funding has come in the form of Amazon and Google cloud computing credits — not real money.

Santa Clara Law professor Ed Lee warned in a blog post that the ruling means “Anthropic faces at least the potential for business-ending liability.” He separately wrote that if Anthropic ultimately loses at trial and a final judgment is entered, the company would be required to post a surety bond for the full amount of damages in order to delay payment during any appeal, unless the judge grants an exception.

In practice, this usually means arranging a bond backed by 100 percent collateral — not necessarily cash, but assets like cloud credits, investments, or other holdings — plus a 1-2 percent annual premium. The impact on Anthropic’s day-to-day operations would likely be limited at first, aside from potentially higher insurance costs, since the bond requirement would only kick in after a final judgment and the start of any appeals process.

Lee wrote in another post that Judge Alsup “has all but ruled that Anthropic’s downloading of pirated books is [copyright] infringement,” leaving “the real issue at trial… the jury’s calculation of statutory damages based on the number of copyrighted books/works in the class.”

While the risk of a billion-dollar-plus jury verdict is real, it’s important to note that judges routinely slash massive statutory damages awards — sometimes by orders of magnitude. Federal judges, in particular, tend to be skeptical of letting jury awards reach levels that would bankrupt a major company. As a matter of practice (and sometimes doctrine), judges rarely issue rulings that would outright force a company out of business, and are generally sympathetic to arguments about practical business consequences. So while the jury’s damages calculation will be the headline risk, it probably won’t be the last word.

On Thursday, the company filed a motion to stay — a request to essentially pause the case — in which they acknowledged the books covered likely number “in the millions.” Anthropic’s lawyers also wrote, “the specter of unprecedented and potentially business-threatening statutory damages against the smallest one of the many companies developing [large language models] with the same books data” (though it’s worth noting they have an incentive to amplify the stakes in the case to the judge).

The company could settle, but doing so could still cost billions given the scope of potential penalties.

Anthropic, for its part, told Obsolete it “respectfully disagrees” with the decision, arguing the court “failed to properly account for the significant challenges and inefficiencies of having to establish valid ownership millions of times over in a single lawsuit,” and said it is “exploring all avenues for review.”

The plaintiffs lawyers did not reply to a request for comment.

From “fair use” win to catastrophic liability

Just a month ago, Anthropic and the rest of the industry were celebrating what looked like a landmark victory. Alsup had ruled that using copyrighted books to train an AI model — so long as the books were lawfully acquired — was protected as “fair use.” This was the legal shield the AI industry has been banking on, and it would have let Anthropic, OpenAI, and others off the hook for the core act of model training.

But Alsup split a very fine hair. In the same ruling, he found that Anthropic’s wholesale downloading and storage of millions of pirated books — via infamous “pirate libraries” like LibGen and PiLiMi — was not covered by fair use at all. In other words: training on lawfully acquired books is one thing, but stockpiling a central library of stolen copies is classic copyright infringement.

Thanks to Alsup’s ruling and subsequent class certification, Anthropic is now subject to a class action encompassing five to seven million books — although only works with registered US copyrights are eligible for statutory damages, and the precise number remains uncertain. A significant portion of these datasets consists of non-English titles, many of which were likely never published in the US and may fall outside the reach of US copyright law. For example, an analysis of LibGen’s holdings suggests that only about two-thirds are in English.

Assuming that only two-fifths of the five million books are covered and the jury awards the statutory minimum of $750 per work, you still end up with $1.5 billion in damages. And as we saw, the company’s own lawyers just said the number is probably in the millions (though it’s worth noting they have an incentive to amplify the stakes in the case to the judge).

The statutory maximum and with five million books covered? $150,000 per work, or $750 billion total — a figure Anthropic’s lawyers called “ruinous.” No jury will award that, but it gives you a sense of the range.

The previous record for a case like this was set in 2019, when a federal jury found Cox Communications liable for $1 billion after the nation’s biggest music labels accused the company of turning a blind eye to rampant piracy by its internet customers. That verdict was overturned on appeal years later and is now under review by the Supreme Court.

But even that historic sum could soon be eclipsed if Anthropic loses at trial.

The decision to treat AI training as fair use was widely covered as a win for the industry — and, to be fair, it was. But Anthropic is now facing an existential threat, with barely a mention. Outside of the legal and publishing press, only Reuters and The Verge have covered the class certification ruling, and neither discussed the fact that this case could spell the end for Anthropic.

Update: early Friday morning, the LA Times ran a column discussing the potential for a trillion-dollar judgment.

Respecting copyright is “not doable”

The legal uncertainty now facing the company comes as the industry continues an aggressive push in Washington to reshape the rules in their favor. In comments submitted earlier this year to the White House’s “AI Action Plan,” Meta, Google, and OpenAI all urged the administration to protect AI companies’ access to vast training datasets — including copyrighted materials — by clarifying that model training is unequivocally “fair use.” Ironically, Anthropic was the only leading AI company to not mention copyright in its White House submission.

At the Wednesday launch of the AI Action Plan, President Trump dismissed the idea that AI firms should pay to use every book or article in their training data, calling strict copyright enforcement “not doable” and insisting that “China’s not doing it.” Still, the administration’s plan is conspicuously silent on copyright — perhaps a reflection of the fact that any meaningful change would require Congress to amend the Copyright Act. The federal Copyright Office can issue guidance but ultimately has no power to settle the matter. Administration officials told the press the issue should be left to the courts.

Anthropic made some mistakes

Anthropic isn’t just unlucky to be up first. The judge described this case as the “classic” candidate for a class action: a single company downloading millions of books in bulk, all at once, using file hashes and ISBNs to identify the works. The lawyers suing Anthropic are top-tier, and the judge has signaled he won’t let technicalities slow things down. A single trial will determine how much Anthropic owes; a jury could choose any number between the statutory minimum and maximum.

The order reiterates a basic tenet of copyright law: every time a pirated book is downloaded, it constitutes a separate violation — regardless of whether Anthropic later purchased a print copy or only used a portion of the book for training. While this may seem harsh given the scale, it’s a straightforward application of existing precedent, not a new legal interpretation.

And the company’s handling of the data after the piracy isn’t winning it any sympathy.

As detailed in the court order, Anthropic didn’t just download millions of pirated books; it kept them accessible to its engineers, sometimes in multiple copies, and apparently used the trove for various internal tasks long after training. Even when pirate sites started getting taken down, Anthropic scrambled to torrent fresh copies. After a company co-founder discovered a mirror of “Z-Library,” a database shuttered by the FBI, he messaged his colleagues: “[J]ust in time.” One replied, “zlibrary my beloved.”

That made it much easier for the judge to say: this is “Napster” for the AI age, and the copyright law is clear.

Anthropic is separately facing a major copyright lawsuit from the world’s biggest music publishers, who allege that the company’s chatbot Claude reproduced copyrighted lyrics without permission — a case that could expose the firm to similar per-work penalties from thousands to potentially millions of songs.

Ironically, Anthropic appears to have tried harder than some better-resourced competitors to avoid using copyrighted materials without any compensation. Starting in 2024, the company spent millions buying books, often in used condition — cutting them apart, scanning them in-house, and pulping the originals — to feed its chatbot Claude, a step no rival has publicly matched.

Meta, despite its far deeper pockets, skipped the buy-and-scan stage altogether — damning internal messages show engineers calling LibGen “obviously pirated” data and revealing that the approach was approved by Mark Zuckerberg.

Why the other companies should be nervous

If Anthropic settles, it could end up as the only AI company forced to pay for mass copyright infringement — especially if judges in other cases follow Meta’s preferred approach and treat downloading and training as a single act that qualifies as fair use.

For now, Anthropic’s best shot is to win on appeal and convince a higher court to reject Judge Alsup’s reasoning in favor of the more company-friendly approach taken in the Meta case, which treats the act of training as fair use and effectively rolls the infringing downloads into that single use.

But appeals usually have to wait until after a jury trial — so the company faces a brutal choice: settle for potentially billions, or risk a catastrophic damages award and years of uncertainty. If Anthropic goes to trial and loses on appeal, the resulting precedent could drag Meta, OpenAI, and possibly even Google into similar liability.

OpenAI and Microsoft now face 12 consolidated copyright suits — a mix of proposed class actions by book authors and cases brought by news organizations (including The New York Times) — in the Southern District of New York before Judge Sidney Stein.

If Stein were to certify an authors’ class and adopt an approach similar to Alsup’s ruling against Anthropic, OpenAI’s potential liability could be far greater, given the number of potential covered works.

What’s next

A trial is tentatively set for December 1st. If Anthropic fails to pull off an appellate victory before then, the industry is about to get a lesson in just how expensive “move fast and break things” can be when the thing you’ve broken is copyright law — a few-million times over.

A multibillion dollar settlement or jury award would be a death-knell for almost any four-year-old company, but the AI industry is different. The cost to compete is enormous, and the leading firms are already raising multibillion dollar rounds multiple times a year.

That said, Anthropic has access to less capital than its rivals at the frontier — OpenAI, Google DeepMind, and, now, xAI. Overall, company-killing penalties may be unlikely, but they’re still possible, and Anthropic faces the greatest risk at the moment. And given how fiercely competitive the AI industry is, a multibillion dollar setback could seriously affect the company’s ability to stay in the race.

And some competitors seem to have functionally unlimited capital. To build out its new superintelligence team, Meta has been poaching rival AI researchers with nine-figure pay packages, and Zuckerberg recently said his company would invest “hundreds of billions of dollars” into its efforts.

To keep up with its peers, Anthropic recently decided to accept money from autocratic regimes, despite earlier misgivings. On Sunday, CEO Dario Amodei issued a memo to staff saying the firm will seek investment from Gulf states, including the UAE and Qatar. The memo, which was obtained and reported on by Kylie Robison at WIRED, admitted the decision would probably enrich “dictators” — something Amodei called a “real downside.” But, he wrote, the company can’t afford to ignore “a truly giant amount of capital in the Middle East, easily $100B or more.”

Amodei apparently acknowledged the perceived hypocrisy of the decision, after his October essay/manifesto “Machines of Loving Grace” extolled how important it is that democracies win the AI race.

In the memo, Amodei wrote, “Unfortunately, I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on.”

The timing is striking: the note to staff went out only days after the class action certification suddenly presented Anthropic with potentially existential legal risk.

The question of whether generative AI training can lawfully proceed without permission from rights-holders has become a defining test for the entire industry.

OpenAI and Meta may still wriggle out of similar exposure, depending on how their judges rule and whether they can argue that the core act of AI training is protected by fair use. But for now, it’s Anthropic — not OpenAI or Meta — that’s been forced onto the front lines, while the rest of the industry holds its breath.

Edited by Sid Mahanta and Ian MacDougall, with inspiration and review from my friend Vivian.

Update: The moratorium has been modified again, likely in response to concerns from Republican Senator Marsha Blackburn. Here’s a summary of the changes from the Institute for Law & AI:

1. Shortens the “temporary pause” from 10 to 5 years;

2. Attempts to exempt laws addressing CSAM, childrens’ online safety, and rights to name/likeness/voice/image—although the amendment seemingly fails to protect the laws its drafters intend to exempt; and

3. Creates a new requirement that laws do not create an “undue or disproportionate burden,” which is likely to generate significant litigation.

In other words, the changes don’t actually seem to do what Blackburn wants them to do. The other parts of the following analysis remain true.

Voting on amendments to the reconciliation bill began this morning, with a final overall vote expected late Monday or early Tuesday.

The Republican budget reconciliation bill — better known as "One Big Beautiful Bill" — currently includes a provision attempting to ban state-level AI regulations for ten years.

This moratorium, if passed, would be perhaps the most sweeping attempt to deregulate an emerging technology in US history.

Subscribe now

The AI lobby, likely emboldened by the national political climate, is making a big gamble. Following in the footsteps of the most aggressive actors, like Andreessen Horowitz and Meta, much of the tech industry has learned to stop worrying and love the moratorium.

States that don't comply with the provision would lose access to $500 million in new federal funding from the Broadband Equity, Access, and Deployment (BEAD) program, which provides federal grants to expand internet access in underserved communities. Any state that accepts part of the new funding risks forfeiting its entire share of a $42.5 billion pot of BEAD funding and having their AI laws invalidated.

In May, the reconciliation bill passed the House by a single vote margin and now only needs a majority vote to pass the Senate, after which any differences from the House version would send the bill back to the lower chamber for another vote.

Do they have the votes?

The moratorium, initially slipped into the Big Beautiful Bill with little public discussion, has become a lightning rod within the GOP. Dozens of Republican governors and state attorneys general have publicly opposed the provision, joining the far-right House Freedom Caucus and advocacy arm of the Heritage Foundation, the think tank behind Project 2025.

The moratorium has also received opposition from dozens of Democratic representatives and senators, joining over 140 civil society groups organized by Demand Progress, a tech policy advocacy nonprofit.

Thursday morning, Punchbowl News reported that Republican Senator Marsha Blackburn delivered a letter to Majority Leader John Thune asking him to remove the moratorium from the reconciliation bill. The letter was reportedly also signed by Republican Senators Rand Paul and Josh Hawley. Republican Senators Kevin Cramer and Rick Scott have also expressed concerns about the provision.

With likely unanimous support from Senate Democrats — something advocates tell Obsolete they are working hard to secure — four Republicans would provide enough votes to pass an amendment removing the moratorium. But even with the votes, supporters tell Obsolete they worry that Thune will add the provision back in in what's known as a "wraparound amendment," chock full of the majority's priorities that often gets a party-line vote.

The vote on the amendment is expected overnight Sunday, followed some time after by the vote on the full bill.

Unprecedented

In the U.S., state governments have the power to regulate companies that do business within their borders. Those regulations are sometimes inconsistent, creating a large compliance burden for companies. To resolve this, congress can preempt state-level laws and replace them with a unifying national framework, like when it established uniform trucking regulations in 1994 or created national food labeling standards in 1990.

Something Congress can also do — but has never really done before — is preempt state laws without passing a federal regulation to fill the gap.

Overreach?

By failing to offer a federal framework for AI regulation, even a non-binding one, the provision goes further than what even many of the key players in the industry asked for.

In their submissions to President Trump's AI Action Plan, OpenAI, Meta, Google, and Andreessen Horowitz (known as a16z) ask for federal nullification of state AI laws. OpenAI called for preemption with a national framework that is "purely voluntary." Meta requested "federal preemption of state laws that conflict with the Administration’s pro-innovation agenda." Google asked for preemption and "a unified national framework for frontier AI models focused on protecting national security while fostering an environment where American AI innovation can thrive." a16z, the venture capital giant, called for a federal law that "preempts state-specific restrictions on model development."

The trade group TechNet specifically suggested "the federal government should look to impose a moratorium on state legislation related specifically to the development of frontier AI models until national standards are adopted." The influential industry association includes OpenAI, Google, Meta, Amazon, and Anthropic.

Amazon's submission frets about the "growing patchwork of approaches to AI regulation," but stops short of explicitly asking for preemption. Microsoft and Anthropic didn't mention preemption in their submissions.

OpenAI, Meta, Google, a16z, Amazon, and TechNet did not reply to a request for comment.

American Edge, a dark money lobbying group backed by tens of millions of dollars from Meta, told Punchbowl News it was doing a seven-figure cable and digital ad buy to support the moratorium. Recent ads from the group focus on how AI is empowering American manufacturers, one of whom is quoted saying, "we can't let China get the upper hand."

Opponents of AI regulations often invoke competition with China as justification — Ted Cruz's Senate committee factsheet is titled "Investing In AI and Beating China in the AI Race." Yet China already imposes far stricter regulations on AI than the US. Earlier this month, Chinese AI chatbots temporarily disabled image recognition tools during national college entrance exams, yet these types of restrictions apparently haven't prevented China from rapidly closing the gap with US AI capabilities, from a reported years to months.

An analysis from Public Citizen found that over 200 state laws would likely be preempted by the provision. A bipartisan coalition of 40 state attorneys general warned that the moratorium would nullify state enacted and proposed laws safeguarding against AI-generated explicit material, deceptive deepfakes, discriminatory rent-setting algorithms, and invasive automated phone scams.

Red states in the crosshairs

The moratorium was almost much stronger. As recently as Thursday, the language of the provision might have allowed the Commerce Department to de-obligate the $42.5 billion in BEAD funding and condition access to it on complying with the moratorium, even if states did not take new money.

However, the latest text makes clear that states would only risk their portion of the larger pot if they took any of the new $500 million.

One of the arguments conservatives make against letting states lead the charge on AI regulations is that doing so allows 'woke' states, like California and New York, to impose their values on the rest of the country.

But the structure of the moratorium makes red states, which have smaller budgets, more rural populations, and a lower propensity to regulate, more tempted to take the deal on offer. In doing so, they would give up a substantial amount of their legislative power in exchange for funding designed to help low-income communities get access to broadband internet.

For instance, Montana and New York are set to receive similar total amounts of BEAD funding, but Montana's is 22 percent of the overall state budget to New York's 0.3 percent.

And as law professor Gabe Weil noted, a temporary governing coalition — in many states, the governor alone — could take the deal, binding them to the moratorium for years after their terms end.

There's also a risk that states accept this new funding without fully understanding it endangers its share of a much bigger pot of money. This confusion has been compounded by moratorium supporters' incomplete and misleading explanations. Neither Cruz's senate committee factsheet nor the Koch-backed Abundance Institute’s response to skeptical Republican governors acknowledges the potential loss of $42.5 billion.

The libertarian Abundance Institute doesn't disclose its funders, but head of AI policy Neil Chilson told Politico it received donations from "Silicon Valley and Austin types.”

The exact state-by-state allocation of the new $500 million hasn’t been determined, but it's likely to mirror the dynamic described above — with red states more inclined to accept the tradeoff — though the financial stakes are far lower.

Byrd law

The reconciliation process allows budget-related bills to pass the Senate with a simple majority, rather than the 60 votes needed to overcome the filibuster. As a result, both parties try to cram as many of their priorities into these bills as possible.

These measures have to comply with the Byrd Rule, which the senate parliamentarian uses to assess each provision to determine if it affects the budget enough to stay in. This rule is meant to ensure that reconciliation bills stick to budgetary matters rather than becoming vehicles for unrelated policy changes.

Anticipating the moratorium would not survive the so-called "Byrd bath," earlier this month, Republican Senator Ted Cruz made it conditional on the allocation of $500 million in new BEAD funding.

Even after this change, many insiders, including Republican Senator John Cornyn and JD Vance, predicted the AI moratorium would get stripped. But in a surprising move last Saturday, the Senate Parliamentarian determined the moratorium was Byrd-compliant, allowing it to pass the Senate with a simple majority. "Shocking would be an understatement," a tech lobbyist opposed to the moratorium said of the ruling to Obsolete.

Jason Van Beek, who spent two decades as a senior senate aide following his work on Thune's successful 2004 senate campaign, told Obsolete he'd "never seen that before" in his entire Hill career. "I was shocked by the initial ruling," he said, noting that even senators themselves were caught off guard by the parliamentarian's decision. Van Beek is now Chief Government Affairs Officer for the Future of Life Institute, an AI safety nonprofit lobbying against the moratorium.

In another shocking move, the parliamentarian reopened her decision on Thursday, which led to a further revision to the bill text. The changes to the moratorium language clarified that states would only risk their portion of the $42.5 billion if they took any of the new $500 million, closing the door on the potential of a Commerce clawback.

"Conflicting" regulations

Advocates of the moratorium often cite a problematic "patchwork" of "conflicting" state laws as justification. However, there's little evidence these laws genuinely conflict — requiring businesses in one state to perform actions explicitly prohibited elsewhere. Instead, regulations simply vary in their definitions, scope, and enforcement. In practice, companies typically comply with the strictest regulations applicable to their operations.

When I tweeted asking proponents of the moratorium to point to AI regulations that require states to do mutually exclusive things, the Abundance Institute's Neil Chilson cited the usage of 57 different definitions of AI in state legislation. Chilson was previously chief technologist of the Federal Trade Commission during the first Trump administration and has been one of the loudest supporters of the moratorium. When I pointed out that inconsistency is not the same as contradiction, Chilson did not produce any examples of mutually exclusive legal requirements.

Laws often include exemptions to resolve apparent conflicts. For example, Colorado’s AI consumer protection law, which mandates retaining hiring data to prove nondiscrimination, seems at first glance to conflict with California’s privacy law granting users data deletion rights. Yet California’s law itself contains a carve-out, allowing companies to retain data when required by other legal obligations.

Origins

The idea to preempt state-level AI regulations appears to trace back to policy analyst Adam Thierer, who proposed a "learning period moratorium" in a May 2024 blog post for the free-market R Street Institute. The think tank, funded partly by Google and Amazon, does not disclose its full list of donors.

R Street has not replied to a request for information on its funders.

According to a political consultant advising groups opposed to the moratorium, the provision began as a coordinated effort by tech companies already contending with AI bills in several states. “I think it’s gone a lot further than they imagined,” the consultant told Obsolete.

The initial idea was to discourage other states from following suit by making lawmakers second-guess whether their efforts would hold up, the consultant said. But on this front, the preemption push has failed — it hasn’t stopped new bills from advancing. Key among them is New York's RAISE Act, which passed both chambers with strong, bipartisan majorities earlier in June. The bill would require developers of powerful AI models to conduct safety testing and risk assessments, using liability law as an enforcement mechanism.

Van Beek says preemption may have first started to gain traction in December when a bipartisan House Task Force on AI floated the idea paired with a strong federal standard. Republican Representative Jay Obernolte co-chaired the Task Force and has been a key proponent of the moratorium in the House. At a tech conference in February, he discussed the need for both preemption and congressional action, saying, "We can't preempt something with nothing, so we need to give states that confidence."

But according to Van Beek, the real momentum came shortly after, with the Trump administration’s January AI Action Plan. The plan’s call for public comments became a wishlist for deep-pocketed industry players, many explicitly requesting preemption.

Van Beek described preemption as an "industry priority," telling Obsolete that a16z and other tech lobbyists were "certainly pushing" the idea.

But even as industry enthusiasm grew, Van Beek was skeptical it could actually work. He thought preemption was "so very clearly a policy proposal" that it wouldn't be able to survive the Byrd rule and be included in a reconciliation bill. As a result, he and others were "caught off guard" by how far the provision has gotten. He did not expect to be "having to fight this battle this early in the game."

AI voices

On Monday, Chris Lehane, a Clinton-era political "dark arts" operative and now OpenAI’s chief lobbyist, posted in support of preemption on LinkedIn.

OpenAI CEO Sam Altman, who famously urged Congress to regulate AI in May 2023, was asked about preemption at a recent live taping of the Hard Fork podcast. He said he still believes some regulation is needed, but warned that "a patchwork across the states would probably be a real mess and very difficult to offer services under."

Altman went on to describe his growing disillusionment with the ability of policymakers to keep up with AI’s rapid pace. A detailed, multi-year rulemaking process, he suggested, could be overtaken by the speed of technological change. At the same time, he acknowledged the need for guardrails as systems grow more powerful — ideally something adaptive and narrowly targeted at risky capabilities, rather than a rigid law designed to last a century.

But as I've asked before: is the rapid pace of AI progress really a good reason to defer regulation?

Breaking with his peers in a New York Times op-ed, Anthropic CEO Dario Amodei expressed sympathy for the desire to simplify the regulatory landscape, but called the moratorium "far too blunt an instrument." "Without a clear plan for a federal response, a moratorium would give us the worst of both worlds — no ability for states to act, and no national policy as a backstop," he wrote.

Strange bedfellows

AI policy tends to scramble the usual factions, and the ten-year regulation ban is no exception.

The provision has exposed a split in the GOP between corporatists like Cruz and national conservatives like Hawley, who have opposed it.

Representative Marjorie Taylor Greene, who has identified as a "Christian nationalist," has become the provision's most outspoken critic in the House, tweeting:

I’m not voting for the development of skynet and the rise of the machines by destroying federalism for 10 years by taking away state rights to regulate and make laws on all AI.

Greene said Monday she would oppose the reconciliation bill if it returns to the House with the moratorium intact.

If Greene is the sole obstacle to passing President Trump's signature bill, she'll face "unbelievable pressure," says Van Beek. "You're the center of attention for the president of the United States, leadership of your party, calling you, getting your name out there to the friendly factions within the party to put pressure," he explained.

There are two groups of Republican lawmakers who oppose the provision, according to the consultant: those like Hawley who are uneasy about Big Tech getting special treatment, and those who care about states' rights. Florida Governor Ron Desantis, for instance, has criticized the measure, highlighting child-safety AI regulations he's signed into law.

Mad Libs

Regulation arguments often resemble a game of Mad Libs — swap out the industry or policy fight and the sentences largely still work.

California's bitterly resisted AI safety bill, SB 1047, was likely a major inspiration for the preemption push. Authored by state senator Scott Wiener, the bill would have mainly required the largest AI developers to implement safeguards to mitigate catastrophic risks. Governor Gavin Newsom vetoed the bill, following intense pressure from the industry and prominent Democrats like Nancy Pelosi.

In an August interview, Wiener pushed back on the idea that only a federal standard could work, pointing to his state's 2018 data privacy law, which passed despite similar industry warnings about a patchwork of state laws. Six years later, he noted, Congress still hasn’t passed a national privacy law.

Wiener also called out the hypocrisy of industry leaders who claim to prefer federal rules while lobbying to block them. “A lot of times there are corporate actors who will say, ‘hey, don't go to the state level, do it at the federal level,’ but those are the same corporate actors that are making it impossible for Congress,” he said, citing his 2018 net neutrality bill as another example. That law passed after telecom and cable companies successfully killed a similar federal effort.

Lobbying

Wiener told me last September that, "The tech industry does not want to be regulated by and large. And so, it's always a huge fight and we've had some success in California and in Congress, it's been even harder."

While Big Tech has come to largely embrace preemption, often preferring to work through proxies like dark money groups or industry associations, the idea wasn't immediately adopted.

In May, Politico reported on the lobbying effort, noting that “nobody quite knows what to ask for,” and that there’s disagreement within the industry over how strong federal AI rules should be. One AI company representative described to Politico the spectrum of opinion, from those who want “no regulation at the state level” to others who are “more comfortable with some regulation.”

The provision was a "fluke" that didn't really come from Big Tech, but rather from Andreessen Horowitz, Marc Andreessen's venture capital giant, the lobbyist told Obsolete. They described the firm as "less experienced" and going for the '''hey, let's just ask for exactly what we want'" approach.

The consultant echoed the lobbyist, describing the moratorium as a "tactic" that unexpectedly got legs, but now there are a lot of tradeoffs to getting it passed. While in their view it would still be a huge win for the industry, it would come with unintended consequences, including killing a raft of state-level AI deepfake child pornography regulations. (These conversations took place before the moratorium was substantially weakened by the parliamentarian's revision.)

According to the lobbyist, the broader tech industry feared that blanket deregulation might appear too greedy an ask. And so it took a more cautious approach, avoiding making such a direct request themselves. Though, the lobbyist noted, they all wanted it.

The lobbyist described a tension between tech behemoths and VCs like Andreessen. "It's this big brother, younger brother kind of thing, right? Where the companies are like, ‘whoa, whoa, whoa, we're making incremental progress here,'… Whereas you have some of the venture capitalist type folks coming in and being like, 'hey, let's just ban this and ban that,'" they said.

The consultant noted that Meta and a16z have emerged as the most aggressive opponents of state AI safety regulations, compared to the "pretty tame" lobbying efforts from Microsoft, OpenAI, and Anthropic.

In recent history, Microsoft has positioned itself as more pro-regulation than its competitors. The $3.7-trillion company didn't formally oppose California's AI safety bill, SB 1047 (though it did express a preference for a national law), and is the only one of the above companies that doesn't belong to TechNet.

Now, a lobbyist working on behalf of Microsoft — along with Amazon, Google, and Meta — is pushing for the moratorium, per reporting in the Financial Times.

The consultant said that Elon Musk's falling out with the administration has not been helpful on this particular policy front. Musk supported SB 1047 and has warned of the dangers of AI for over a decade. They said that people were working on getting Musk to weigh in on the moratorium, but noted that his advocacy could cut both ways.

Blowback

The tech industry may have overplayed its hand. By trying to push the moratorium through reconciliation — with Cruz as its public face and Trump’s priorities baked into the broader bill — companies have made it harder for national Democrats to weigh in against any AI regulations, according to the consultant — a miscalculation they say the companies didn’t fully anticipate.

The opposition of national Democrats like Pelosi likely played a key role in killing California's SB 1047 last year. But in swinging for a ten-year veto on all state-level AI regulations, the industry may have alienated some would-be allies in the fight against specific state regulations, like New York's RAISE Act. The consultant says supporters feared that New York Senators Chuck Schumer and Kirsten Gillibrand might pressure Governor Kathy Hochul to veto it, as Gavin Newsom did in California. But neither senator has taken a position on the bill, even privately, the consultant told Obsolete — a silence the consultant attributed to the issue’s newfound political toxicity.

What's next

As the Senate prepares to vote, the tech industry finds itself in uncharted territory. Companies have spent years warning about the dangers of a regulatory patchwork while simultaneously blocking federal action. Concerns about AI's harms and risks are growing, especially within elite circles. The public has been clear about its support for regulation on the technology, though the issue remains low salience. If AI continues to become more capable and ubiquitous — as the industry hopes it will — the technology's salience will grow too, and with it, the appetite for regulation. Should AI enable a disaster or large-scale job loss, the industry may find itself wistful for the days of politicians pitching their bills as innovation-friendly and light-touch.

The moratorium has been substantially weakened and may not even survive. But it's remarkable that the AI industry got within spitting distance of a ten-year vacation from all state-level regulation. The unexpected success of the effort signals just how much Silicon Valley can override the preferences of the public and key parts of the MAGA coalition.

But win or lose, the push may have already damaged the industry's ability to whip Democrats — a miscalculation companies may come to regret.

With editing by Sid Mahanta. All mistakes are mine.

Anthropic just released Claude 4 Opus, its most capable AI model to date. But in doing so, the company may have abandoned one of its earliest promises.

In September 2023, Anthropic published its Responsible Scaling Policy (RSP), a first-of-its-kind safety framework that promises to gate increasingly capable AI systems behind increasingly robust safeguards. Other leading AI companies followed suit, releasing their own versions of RSPs. The US lacks binding regulations on frontier AI systems, and these plans remain voluntary.

Subscribe now

The core idea behind the RSP and similar frameworks is to assess AI models for dangerous capabilities, like being able to self-replicate in the wild or help novices make bioweapons. The results of these evaluations determine the risk level of the model. If the model is found to be too risky, the company commits to not releasing it until sufficient mitigation measures are in place.

Earlier today, TIME published then temporarily removed an article revealing that the yet-to-be announced Claude 4 Opus is the first Anthropic model to trigger the company's AI Safety Level 3 (ASL-3) protections, after safety evaluators found it may be able to assist novices in building bioweapons. (The updated article is time-stamped 6:45 AM Pacific, but didn't go back up until 9:45 AM, implying there was a time zone mixup.)

The article included striking quotes from Anthropic's chief scientist Jared Kaplan. "You could try to synthesize something like COVID or a more dangerous version of the flu — and basically, our modeling suggests that this might be possible," Kaplan said.

When Anthropic published its first RSP in September 2023, the company made a specific commitment about how it would handle increasingly capable models: "we will define ASL-2 (current system) and ASL-3 (next level of risk) now, and commit to define ASL-4 by the time we reach ASL-3, and so on." In other words, Anthropic promised it wouldn't release an ASL-3 model until it had figured out what ASL-4 meant.

Yet the company's latest RSP, updated May 14, doesn't publicly define ASL-4 — despite treating Claude 4 Opus as an ASL-3 model. Anthropic's announcement states it has "ruled out that Claude Opus 4 needs the ASL-4 Standard."

When asked about this, an Anthropic spokesperson told Obsolete that the 2023 RSP is "outdated" and pointed to an October 2024 revision that changed how ASL standards work. The company now says ASLs map to increasingly stringent safety measures rather than requiring pre-defined future standards.

The spokesperson also pointed to past versions published on Anthropic's website, which include descriptions of major changes. The main version history doesn't explicitly flag the removal of the original commitment to define ASL-4 by the time ASL-3 was reached — though this change is discussed in a redline PDF linked on the same page.

After publication, Anthropic reached out to say that the company does define capability thresholds for ASL-4 in its current RSP. However, the original 2023 commitment was more specific — it promised to define both capability thresholds and "warning sign evaluations" before training ASL-3 models. While the current RSP includes high-level capability thresholds for ASL-4, it doesn't include the detailed warning sign evaluations that were part of the original commitment.

But, moreover, what does a commitment mean if it can be walked back without public scrutiny?

When Obsolete posed a similar question, the Anthropic spokesperson pushed back, writing:

Would disagree that it's something that 'can be updated at any time.' We have a defined process in place for making updates to the RSP and are rigorous in how we refine and update our commitments.

The spokesperson highlighted a commitment in the RSP that, "Changes to this policy will be proposed by the CEO and the Responsible Scaling Officer and approved by the Board of Directors, in consultation with the Long-Term Benefit Trust."

This Trust, described by Anthropic in a September 2023 announcement as an independent body intended to ensure accountability, itself appears to have fallen short on a significant commitment. Citing the company's general counsel and its incorporation documents, TIME reported last May that the Long-Term Benefit Trust (LTBT) would appoint three out of five directors by November 2024. However, the Anthropic website currently only lists four directors.

Anthropic had not replied to a follow-up question about this by the time of publication.

When voluntary governance breaks down

In February 2024, a senior AI safety researcher at a leading AI company told me that these voluntary governance approaches work, but only for a time. Once you get close to human-level AI, competitive pressures take over.

Many AI insiders are increasingly predicting human-level AI, often referred to as artificial general intelligence (AGI), is just around the corner. The influential essay series AI 2027, written by leading AI forecasters, predicts recursively self-improving AI systems by 2027 (Vice President JD Vance just told The New York Times that he's read the series).

These predictions coincide with an apparent uptick in corner-cutting and broken promises from leading AI companies. Google DeepMind didn't publish a safety report for its flagship model for weeks, and its first attempt in April was light on details. Also in April, the Financial Times reported that OpenAI's safety testing time had shrunk from months to days.

Last month, updates to OpenAI's standard model, GPT-4o, caused it to breathlessly affirm essentially anything you told it — a behavior Rolling Stone reported could dangerously interact with mental illness. Just days before the company rolled back the updates, an OpenAI employee bragged on X that "this is the quickest we've shipped an update to our main 4o line. Releases are accelerating, and the public is getting our best faster than ever."

Anthropic, however, has mostly managed to avoid scandals. The company was founded by safety-forward OpenAI researchers who became disillusioned with CEO Sam Altman, a story described in new detail by the just-released books The Optimist and Empire of AI.

Shortly before the TIME article was restored, Anthropic published its own announcement of Claude 4 Opus reaching ASL-3. The company emphasized it hasn't definitively determined whether the new model requires ASL-3 protections, but is implementing them as a "precautionary and provisional action" because it can't clearly rule out the risks.

Disclosure: I've received funding from the Omidyar Network as a Reporter in Residence. The Omidyar Network has also invested in Anthropic.

What ASL-3 actually means

Claude 4 Opus and Sonnet were made publicly available around 9:45 AM Pacific.

According to the TIME article, which also went back up around 9:45 AM Pacific, Kaplan told the publication that in internal testing, Claude 4 Opus performed more effectively than prior models at advising novices on producing biological weapons.

The ASL-3 threshold was designed in part to catch AI systems that could "substantially increase" someone's ability to obtain, produce, or deploy chemical, biological, radiological, or nuclear (CBRN) weapons. The protections required by reaching ASL-3 include enhanced cybersecurity to prevent model weight theft and deployment measures specifically targeting CBRN misuse — what Anthropic calls a "defense in depth" strategy.

To meet that bar, the company says it has rolled out safeguards like "constitutional classifiers" — AI systems that monitor inputs and outputs for dangerous CBRN-related content — along with enhanced jailbreak detection supported by a bug bounty program.

A test of voluntary commitments

This moment reveals both the potential and the limitations of the industry's self-regulatory approach. On one hand, Anthropic appears to be following through on most of its commitments, implementing substantial safety measures even when uncertain they're needed.

On the other hand, these commitments remain voluntary with no external enforcement. As TIME notes, Anthropic itself is the judge of whether it's complying with the RSP. Breaking it carries no penalty beyond potential reputational damage. And, as we saw, these commitments can be quietly updated according to a process that Anthropic designed.

This is not to say that every detail of a safety plan should be set in stone. It's reasonable to update a framework based on new information. But there should be a clear distinction between clarifying details and reneging on an earlier commitment. Furthermore, real transparency would mean more clearly flagging any change significant enough to require LTBT approval.

What happens next

Anthropic says it will continue evaluating Claude 4 Opus' capabilities. If the company determines the model doesn't actually cross the ASL-3 threshold, it could downgrade to the more permissive ASL-2 protections. But for now, Anthropic says it's erring on the side of caution.

The bigger question is whether this precedent will hold as competition intensifies. With no binding safeguards on frontier AI development in the US, companies like Anthropic are essentially regulating themselves in public view. Whether that's sufficient for managing risks that Kaplan himself compares to pandemic-level threats remains an open — and urgent — question.

This moment exposes the limits of the self-regulatory model the AI industry has championed. If a company as safety-focused as Anthropic can quietly retreat from its own red lines, what will everyone else do when the stakes get even higher?

Edited by Sid Mahanta.

The previously unreported 13-page letter — dated May 15 and obtained by Obsolete — lays out OpenAI’s legal defense of its updated proposal to restructure its for-profit entity, which can still be blocked by the California and Delaware attorneys general (AGs). This letter is OpenAI’s latest attempt to prevent that from happening — and it’s full of surprising admissions, denials, and attacks.

OpenAI has not replied to a request for comment.

Subscribe now

Control of OpenAI currently rests with its nonprofit board — an arrangement investors have reportedly balked at following the brief firing of CEO Sam Altman in November 2023. To assuage investor concerns, the company initiated plans last year to remove nonprofit control and restructure as a for-profit public benefit corporation (PBC).

The restructuring effort has drawn fire from former employees, Nobel laureates, and an array of civil society organizations, including a coalition of more than 50 California based nonprofits and community groups. Elon Musk has also sued to block it from moving forward, arguing that the nonprofit he helped found and fund is attempting to abandon its charitable purpose.

On May 5, OpenAI made the shocking announcement that it would ditch its plan to remove control over the company from the nonprofit. The move was hailed by many as a win for public pressure and represented in headlines as a substantial change to the original plan. But as the surprise of the decision wore off and observers dug into the sparse details OpenAI shared, the victory for civic action began to appear hollower.

The revised plan appears designed to placate both external critics and concerned investors by maintaining the appearance of nonprofit control while changing its substance. SoftBank, which recently invested $30 billion in OpenAI with the right to claw back $10 billion if the restructuring didn't move forward, seems unfazed by the company's new proposal — the company's finance chief said on an earnings call that from SoftBank's perspective, "nothing has really changed."

Revelations

The letter from OpenAI's lawyers to AG Bonta contains a number of new details. It says that "many potential investors in OpenAI's recent funding rounds declined to invest" due to its unusual governance structure — in tension with Bloomberg's earlier reporting that OpenAI's October round was "oversubscribed."

The letter resolves a question raised in recent Bloomberg reporting: the nonprofit board will have the power to fire PBC directors.

The document also states that "The Nonprofit will exchange its current economic interests in the Capped-Profit Enterprise for a substantial equity stake in the new PBC and will enjoy access to the PBC's intellectual property and technology, personnel, and liquidity…" This suggests the nonprofit would no longer own or control the underlying technology but would merely have a license to it — similar to OpenAI's commercial partners.

The key question

The key question at the heart of OpenAI's restructuring is whether the people practically running the company day-to-day will have a legal duty to prioritize the charitable mission above profit. OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits humanity.

Under the current structure, OpenAI's LLC operating agreement explicitly states that "the Company's duty to this mission and the principles advanced in the OpenAI, Inc. Charter take precedence over any obligation to generate a profit." This creates a legally binding obligation for the company's management.

In contrast, under the proposed structure, PBC directors would be legally required to balance shareholder interests with the public benefit purpose. The ability to fire PBC directors does not change their fundamental legal duties while in office.

This shift in legal duties is likely one reason why investors who demanded the right to claw back tens of billions if OpenAI failed to restructure by aggressive deadlines are reportedly more comfortable with the new arrangement. It transforms the nonprofit from an active manager with direct control to a shareholder with hiring and firing powers — a much more familiar and investor-friendly governance model.

So far, no Delaware PBC has ever been held liable for failing to pursue its mission — legal scholars can’t find a single benefit‑enforcement case on the books.

Competitors and critics

While ostensibly addressed to the California coalition, the document reads more like a rebuttal to Elon Musk's lawsuit against the company — Musk's name appears nearly twenty times throughout. This repetition appears to be an attempt to delegitimize criticism by tying it to Musk and the commercial interests represented by his rival startup, xAI. "Despite (and likely because) of OpenAI's achievements, its most powerful detractors—many of whom, including Elon Musk, stand to massively profit if OpenAI falters—have sponsored a false narrative about OpenAI to advance their own commercial interests," the company writes.

This framing tries to cast all critics, including the nonprofit coalition, as either aligned with or manipulated by Musk's agenda. The letter claims that Musk has leveraged "a campaign of harassment and misinformation for more than a year" and suggests that legal arguments against OpenAI's restructuring "echo those of OpenAI's competitors who stand to gain from its downfall." (Meta joined Musk's suit in December.)

Musk has indeed waged a bitter campaign against OpenAI and Altman since losing a struggle for the organization's helm in 2018. But by conflating legitimate concerns from civil society organizations with Musk's more self-serving grievances, OpenAI appears to be creating a false binary: either you support the company's restructuring or you're aligned with Musk's interests. This rhetorical strategy deflects attention from the substantive governance concerns raised by independent groups.

The formal letter's confrontational attitude contrasts sharply with the conciliatory tone OpenAI has taken in direct communications with the nonprofit coalition. In an email to individual members, OpenAI representatives wrote that they:

understand this moment as an opportunity not just for dialogue—it's a chance to lay the groundwork for a different kind of partnership, one built on trust, transparency, and shared goals. We would welcome the chance to work with you.

This two-faced approach — offering partnership in private communications while suggesting coalition members are acting in bad faith in formal documents — highlights the increasingly adversarial tactics OpenAI is employing against critics of its restructuring plans. (The company countersued Musk last month.)

And, at least for some critics, it does not appear to be working. Orson Aguilar, a leader of the California nonprofit coalition, forwarded OpenAI's letter and the above email to Obsolete with this note:

For your information, they are sending emails directly to our coalition members and are now implying that we are working with Musk and that we are doing this for commercial interests!

Employee motivations

OpenAI's letter directly challenges the coalition's characterization of employee motivations during the November 2023 board crisis, when Altman was briefly ousted. It claims the coalition "denigrate[s] the outpouring of support for Altman" by suggesting it was a "cash grab motivated solely by the employees' 'financial stakes.'" The company insists there is "zero support for that outrageous claim."

However, a former OpenAI employee who signed the letter supporting Altman strongly disputes this characterization. "That's bullshit and a blatant lie," they wrote to Obsolete. "We were in the middle of a tender offer at the time. People had millions of dollars on the line, myself included." The former employee described numerous conversations on Slack and offline about "how does this affect the tender" and added that many employees "didn't really trust Sam even back then, they just thought the option was signing the letter or the company dying, and they really didn't want the latter."

And in response to a claim in the letter that the nonprofit board is stronger than ever, they wrote, "Also bullshit. The current board is packed with a supermajority of Sam loyalists. Pretending that they're able to act as an independent check on him is a joke."

The letter reassures AG Bonta that, "This plan for the future is the product of careful consideration by the Board of the Nonprofit." However, the authors — legal representatives of the board — then proceed to misspell the names of two of the directors ("Ogunles" instead of "Ogunlesi" and "Hellman" instead of "Hellmann").

Contestable claims

The letter makes some contestable claims, including that "Microsoft has never 'control[led]' OpenAI—either as a matter of corporate governance or as a matter of antitrust law." However, Altman's threat to reconstitute OpenAI at Microsoft, backed by signatures from 90 percent of the company, likely played a pivotal role in the board's decision to reinstate him last November. OpenAI also asserts that "all arrangements with Microsoft have been negotiated at arm's length," — a claim that strains credulity given how instrumental Altman has been to forging the relationship. Altman's "bromance" with Microsoft CEO Satya Nadella has been covered extensively, and Altman reportedly asked Nadella for billions of dollars while serving on the nonprofit board.

OpenAI's criticism of the coalition's April 9 letter is particularly puzzling. The company faults the coalition for claiming that "OpenAI proposes to eliminate any and all control by [the Nonprofit] over OpenAI's core work." This criticism is perplexing because, as OpenAI itself later demonstrated with its May 5 reversal, that was precisely OpenAI's publicly understood plan at the time the coalition made its statement. The company appears to be retroactively criticizing the coalition for accurately describing OpenAI's proposal as it stood.

What's left unsaid

Perhaps most notable is what the letter doesn't say. While OpenAI claims its PBC will operate "consistent with the Nonprofit's mission and subject to the Nonprofit's control, just as the Capped-Profit Enterprise does today," it concedes in the next breath that under Delaware law, PBC directors must balance shareholder interests with public benefit. As highlighted earlier, this represents a fundamental shift from the current structure, where the nonprofit board isn't required to consider shareholder profits at all.

A separate group of advocates organized under the label "Not for Private Gain" sent its own legal letter in April, asking the California and Delaware AGs to investigate and block OpenAI's restructuring plans. The group laid out six existing governance safeguards that were threatened by the company's proposal — like the subordination of profit motives to OpenAI's charitable purpose and caps on how much profit can go to investors.

Earlier this week, the Not for Private Gain project published a follow-up analyzing OpenAI's new proposal, finding that five of the six safeguards were still at risk of being eliminated by the restructuring. The sixth safeguard — the profit caps — would go away under the new plan.

The Not for Private Gain update offers specific ideas on how OpenAI could legally enshrine some of the safeguards that would not exist by default if the new proposal moves forward — for instance, through legal guarantees to prioritize mission above profit in the PBC's certificate of incorporation.

OpenAI's letter to AG Bonta does include one recommendation made in the group's follow-up — the nonprofit board will have the ability to fire the PBC board members.

Tyler Whitmer of the Not for Private Gain coalition wrote to Obsolete:

I’m glad to learn the nonprofit board will have the power to hire and fire directors of the proposed PBC, but this is a far cry from the direct control of the LLC the nonprofit has today. The new proposal is being pitched as the nonprofit staying in control, but in reality it undermines the nonprofit’s charitable mission by diluting its ability to directly manage development and deployment of AGI and shifts enforcement of the mission from public servants in the AGs' offices to self interested shareholders.

Responding to the letter's insinuation of impropriety, the group shared this statement with Obsolete:

We are independent of other groups voicing concerns about OpenAI’s restructuring. None of our letters’ signatories work for an OpenAI competitor, and we have received no funding from OpenAI competitors, including Elon Musk

The Not for Private Gain update acknowledges that it "does not discuss the proposed abandonment of OpenAI’s profit caps or its current commitment that artificial general intelligence—when OpenAI creates it—will belong exclusively to the nonprofit for the benefit of humanity." This, the authors write, is not meant "to minimize the significance of these proposed changes, but rather to underscore the singular importance of ensuring that OpenAI continues to have a legally enforceable obligation to advance the charitable mission above all else."

OpenAI has been marked by a long history of assurances, beginning with a simple one: public interest over private gain. As the organization now tries to undo the legal restraints that accompanied these foundational promises, it becomes harder to ignore what’s being constructed in their stead: not a firewall between mission and profit, but a bridge that only goes in one direction.

At a glance, this appears to be a significant reversal from the company's previous plan to shed nonprofit control over the for-profit entity, an effort that faced major opposition from parties including Elon Musk, civil society leaders, former employees, and legal scholars.

Subscribe now

"OpenAI was founded as a nonprofit, and is today overseen and controlled by that nonprofit. Going forward, it will continue to be overseen and controlled by that nonprofit," Bret Taylor, chair of the nonprofit's board, wrote in a blog post today. The announcement followed discussions with the attorneys general (AGs) of California and Delaware, who oversee charitable organizations in their states and have been scrutinizing the proposed restructuring — which either of them could have blocked.

But what does "control" actually mean in this context? And will the profit caps — which would have sent OpenAI's profits to the nonprofit once investors received returns of up to 100-times their investment — remain in place?

And does OpenAI now owe investors $26.6 billion, plus interest? The company reportedly gave investors in its last two fundraising rounds the ability to claw back tens of billions of dollars, if it didn't shed its nonprofit controls by certain deadlines. The status of these provisions didn't appear to be explicitly addressed by OpenAI.

Reading between the lines

While the announcement keeps the nonprofit in control, CEO Sam Altman's accompanying letter suggests a significant change to the capped-profit model:

Instead of our current complex capped-profit structure—which made sense when it looked like there might be one dominant AGI effort but doesn’t in a world of many great AGI companies—we are moving to a normal capital structure where everyone has stock. This is not a sale, but a change of structure to something simpler.

This language implies that the profit caps, a crucial feature of OpenAI's original model, will be modified. These caps were intended to ensure that if OpenAI became wildly profitable through building artificial general intelligence (AGI), the vast majority of profits would flow to the nonprofit. OpenAI's charter defines AGI as "a highly autonomous system that outperforms humans at most economically valuable work." And it's worth noting that normal corporations don't have caps on investor returns.

Altman seems to be arguing that because many companies will build AGI, no single company is going to pull in the trillion-dollar-plus profits they once anticipated. As a result, the nonprofit will actually be giving up less in expected profit and will need to receive less compensation for the removal of the caps.

But if investors have been balking at the profit caps, as they reportedly have been, then that may mean they think OpenAI has some appreciable chance of hitting them.

In other words, what Altman wrote doesn't make much sense. If OpenAI is no longer on track for trillion-dollar profits, the caps should be irrelevant. The fact that investors purportedly pushed to eliminate them suggests they believe OpenAI’s upside remains enormous — and that the caps were more than just a technical nuisance. I find it hard to imagine that investors are put off by the complexity of OpenAI's structure more than the caps themselves.

Without these limits, OpenAI’s investors could reap uncapped returns — fundamentally changing the company's incentives and undermining its original intention to avoid the corrupting influence of profit-maximization.

The control question

A key question that remains unanswered is how much OpenAI's nonprofit control really matters if the board doesn't exercise independent judgment. After the November 2023 firing and rapid rehiring of Altman, it's reasonable to wonder if the nonprofit board will ever again meaningfully stand up to the CEO or the for-profit entity.

Taylor shared a "technical detail" with reporters: the PBC will have its own board, but the nonprofit will appoint its directors — and for now, the same people will sit on both boards.

As the technology OpenAI builds grows more capable and widely used, the way it's governed matters more.

This also matters because if OpenAI actually builds AGI, it's not out of the question to think it could generate trillions in profits. Without profit caps, there would be no legal obligation to share that money with the world, beyond what they pay in taxes and whatever stake the nonprofit retains.

Perhaps more importantly, nonprofit control makes it easier for directors to justify decisions that are good for the world or safety but bad for the bottom line. Shortly after Altman was fired, OpenAI's chief strategy officer Jason Kwon reportedly told then-director Helen Toner that by voting for the firing, she had violated her duties — but under the nonprofit structure, her duty was to humanity writ large, not shareholders.

A fiduciary duty to humanity may sound silly, but the degrees of freedom it offers could matter enormously in moments of crisis or when making key decisions about powerful AI systems.

Four predictions

Based on my past reporting and reading between the lines of the announcement, here's what I expect to happen:

1. The profit caps will be gone, replaced with a "normal capital structure where everyone has stock" — and that stock entitles you to uncapped future profits.

2. OpenAI won't have to pay back the $26.6 billion to investors because they've signed off on this change in return for the profit caps being eliminated.

3. The nonprofit will be compensated tens of billions by the for-profit entity for the removal of the caps.

4. The nonprofit will largely use that money to buy OpenAI services for nonprofits and governments, targeting constituencies that can make life difficult for the company (like California nonprofits).

As Altman wrote in the letter, the nonprofit "will become a big shareholder in the [new public benefit corporation for-profit entity], in an amount supported by independent financial advisors, giving the nonprofit resources to support programs so AI can benefit many different communities."

Is this a victory?

The "Not for Private Gain" letter from civil society leaders, former employees, and Nobel laureates that Obsolete covered last month argued that no sale price could adequately compensate the nonprofit for what it would be giving up — control over a company that might build AGI. They essentially said: this isn't a debate about fair market value.

In response to the news, former OpenAI researcher Todor Markov tweeted:

Glad you're making this commitment.

I do think it's unfortunate that you only made it after public pressure and the Attorneys General getting involved. Had you done this back in December, it would have looked like principle, not like you got dragged kicking and screaming.

Still, regardless of your true motivations, this decision is a win for the broader public. We’ll be watching closely to make sure nonprofit control remains more than just words on paper.

Page Hedley, who led the "Not for Private Gain" letter and also used to work at OpenAI, is less sure that this is a win. "We’re glad that OpenAI is listening to concerns from civil society leaders and Attorneys General Jennings and Bonta," he wrote in a statement that raises these questions:

Will OpenAI's commercial goals continue to be legally subordinate to its charitable mission, which is enforceable by the attorneys general?

Who will own the technology that OpenAI develops?

Hedley concludes, "The 2019 restructuring announcement made the primacy of the mission very clear, but so far, these statements have not."

Whether this development marks a meaningful victory for those concerned about OpenAI’s governance — or just a clever repackaging of its original plans — remains an open question.

Thanks to Ian MacDougall and Sid Mahanta for the excellent and timely edits.

Don’t become a for-profit.

That’s the blunt message of a recent letter signed by more than 30 people, including former OpenAI employees, prominent civil-society leaders, legal scholars, and Nobel laureates, including AI pioneer Geoffrey Hinton and former World Bank chief economist Joseph Stiglitz.

Obsolete obtained the 25-page letter, which was sent last Thursday to the attorneys general (AGs) of California and Delaware, two officials with the power to block the deal.

Subscribe now

Made public early Wednesday, the letter argues that OpenAI's proposed transformation from a nonprofit-controlled entity into a for-profit public benefit corporation (PBC) would fundamentally betray the organization's founding mission and could even be unlawful — placing the power and responsibility to intervene squarely with the state AGs.

OpenAI and the offices of the California and Delaware attorneys general did not reply to requests for comment.

The letter was primarily authored by Page Hedley, a lawyer who worked at OpenAI from 2017-2018 and recently left an AI policy role at Longview Philanthropy; Sunny Gandhi, political director of Encode AI; and Tyler Whitmer, founder and president of Legal Advocates for Safe Science and Technology.

(Encode AI receives funding from the Omidyar Network, where I am currently a Reporter in Residence, and I worked as a media consultant for Longview Philanthropy in 2022.)

Nonprofit origins

In 2015, OpenAI’s founders established it as a nonprofit research lab as a counter to for-profit AI companies like Google DeepMind. When executives felt they couldn't raise enough capital to compete at AI's increasingly expensive cutting edge, they launched a for-profit in 2019 that was ultimately controlled by the nonprofit. This unusual corporate structure was designed to keep the organization loyal to its mission: to ensure that artificial general intelligence (AGI) benefits all of humanity. AGI is defined in the OpenAI Charter as "a highly autonomous system that outperforms humans at most economically valuable work."

In the years since this structure was established, OpenAI has taken on tens of billions in investment, reached a $300 billion valuation, and arguably become the world's leading AI company.

But now, despite years of insisting that nonprofit governance was essential to its mission, OpenAI wants to abandon it — and adopt a more conventional corporate structure that prioritizes shareholder returns.

OpenAI has raised over $46 billion since October and has reportedly given investors the ability to ask for most of it back if the restructuring isn't completed by certain deadlines, with the earliest hitting by the end of this year.

"OpenAI may one day build technology that could get us all killed," writes former employee Nisan Stiennon in a supplemental statement. "It is to OpenAI's credit that it's controlled by a nonprofit with a duty to humanity. This duty precludes giving up that control."

In the letter, the authors argue that the AGs should investigate the conversion, prevent it from moving forward as planned, and work to ensure the OpenAI nonprofit board is sufficiently empowered, informed, independent, and willing to stand up to company management. If the board doesn't meet these requirements, the AGs should intervene, potentially going so far as to remove directors and appoint an independent oversight body, the letter suggests.

"The proposed restructuring would eliminate essential safeguards," the letter states, "effectively handing control of, and profits from, what could be the most powerful technology ever created to a for-profit entity with legal duties to prioritize shareholder returns."

This letter is the latest in a series of high profile challenges to the OpenAI restructuring. Cofounder Elon Musk is suing to block it, with formal support from former employees, civil society groups, and Meta. Musk also offered to buy the nonprofit's assets for $97.4 billion, in a likely effort to bid up the price the new for-profit has to pay for control or disrupt the transformation entirely.

And earlier this month, a separate coalition of over 50 California nonprofit leaders filed a petition with AG Rob Bonta's office, urging him to ensure that the nonprofit is adequately compensated for what it's giving up in the transition.

However, this new letter differs from past efforts by asking the AGs to pursue an intervention separate from Musk's suit and by arguing that the restructuring shouldn't move forward — no matter how much OpenAI's nonprofit gets compensated.

Nobel opposition

Three Nobel laureates feature prominently among the signatories: AI pioneer Geoffrey Hinton, and economists Oliver Hart and Joseph Stiglitz.

Hinton is the second most-cited living scientist, and his pioneering work in the field of deep learning earned him the 2024 Nobel Prize in Physics. In May 2023, Hinton resigned from his job at Google to freely warn that advanced AI could wipe out humanity.

In a supplemental statement, Hinton praised OpenAI’s commitment to ensuring that AGI helps humanity. “I would like them to execute that mission instead of enriching their investors. I’m happy there is an effort to hold OpenAI to its mission that does not involve Elon Musk,” he wrote. Hart, an economics Nobel winner for his work in contact theory in 2016, was more blunt in his supplemental statement: “The proposed governance change is dangerous and should be resisted."

Contradictions

The letter compiles a damning collection of statements from OpenAI's leadership that directly contradict its current push to abandon nonprofit control. These quotes paint a picture of an organization that chose its unique structure to safeguard humanity, and whose leaders repeatedly emphasized this commitment as central to their mission.

Back in 2015, Sam Altman emailed Elon Musk proposing a structure where the tech “belongs to the world via some sort of nonprofit,” adding that OpenAI would “aggressively support all regulation.” (Since then, OpenAI has lobbied to weaken or kill several AI laws.)

In 2017, Altman told an audience in London: “We don’t ever want to be making decisions to benefit shareholders. The only people we want to be accountable to is humanity as a whole.”

In 2019, president Greg Brockman clarified something subtle but important: “The true mission isn’t for OpenAI to build AGI. The true mission is for AGI to go well for humanity... our goal is to make sure it goes well for the world.”

Altman made clear that some money was not worth making: “There are things we wouldn't be willing to do no matter how much money they made,” he told Kara Swisher in 2018, “and we made this public so the public would hold us accountable.”

And in a 2020 interview, Altman warned that if OpenAI succeeded at building AGI, it might “capture the light cone of all future value in the universe.” That, he said, “is for sure not okay for one group of investors to have.”

(The light cone refers to all of the universe that earth-originating life could theoretically affect, to give you a sense of Altman's ambitions.)

This wasn’t just rhetoric. When OpenAI launched its for-profit arm in 2019, it promised that all employees and investors would sign contracts putting the nonprofit Charter first — “even at the expense of some or all of their financial stake.” The for-profit announcement also included the assurance: "Regardless of how the world evolves, we are committed — legally and personally — to our mission."

In his 2023 Congressional testimony, Altman highlighted key safeguards in OpenAI's "unusual structure" that keeps it mission-focused: nonprofit control over the for-profit subsidiary, fiduciary duties to humanity rather than investors, a majority-independent board with no equity stakes, capped investor profits with residual value flowing to the nonprofit, and explicit reservation of AGI technologies for nonprofit governance.

The letter then warns that, "These safeguards are now in jeopardy under the proposed restructuring," and goes through the status of each under the proposed restructuring.

Justifications

Taken together, these statements leave little room for interpretation. OpenAI deliberately chose a nonprofit-controlled structure with the explicit goal of safeguarding humanity from profit-driven AGI development — the very protection it now seeks to dismantle.

The letter surgically dissects OpenAI's justifications for abandoning its nonprofit governance structure, arguing they prioritize commercial competitiveness over the organization's core charitable mission.

OpenAI's primary rationale, stated in recent court filings, is that its unusual structure makes it harder to attract investment and talent. But the letter authors counter that this is precisely the point — OpenAI was specifically designed to operate differently.

"Competitive advantage might be a relevant factor, but it is not a sufficient reason to restructure,” the letter argues. "OpenAI's charitable purpose is not to make money or capture market share." It points out that OpenAI's current structure intentionally accepts certain competitive disadvantages as the cost of prioritizing humanity's interests over profits.

These disadvantages might be overrated, suggests signatory Luigi Zingales of the University of Chicago Booth School of Business in his supplemental statement:

The current structure, which caps returns at 100x the capital invested, does not really constrain its ability to raise funds. So, what is the need to transfer the control to a for-profit? To overrule the mandate that AI should be used for the benefit of humanity.

These profit caps were designed to ensure that OpenAI could redistribute exorbitant returns in a world where the company actually builds AGI and puts much of the world out of work. They're one of the things reported to be on the chopping block as part of the proposed restructuring.

The letter authors write:

OpenAI might respond that a competitive advantage inherently advances its mission, but that argument is an implicit comparison of OpenAI and its competitors: that humanity would be better off if OpenAI builds AGI before competing companies. Based on OpenAI’s recent track record, this argument is unlikely to be convincing…

They then list reports of OpenAI's broken promises, rushed safety testing, doublespeak and hypocrisy, and coercive non-disparagement agreements.

Moreover, the letter highlights that OpenAI has not adequately explained why removing nonprofit control is necessary to address its stated issues. While the company claims investors demand simplification of its "capital structure," it fails to demonstrate why this requires eliminating nonprofit oversight rather than more targeted adjustments.

The authors conclude that OpenAI's plans for "one of the best resourced non-profits in history" miss the point entirely. This isn’t about building a well-funded foundation for generic good works — it’s about retaining governance over AGI itself. The letter bluntly states that OpenAI "should not be permitted to sell out its mission."

"No sale price can compensate"

OpenAI claims the restructuring would create "one of the best resourced nonprofits in history" that would "pursue charitable initiatives in sectors such as health care, education, and science."

But the letter's authors argue this would mark a fundamental shift in OpenAI’s charitable purpose — one that, under nonprofit law, can only be altered in exceptional circumstances.

Perhaps the most distinctive argument in the letter is that, given OpenAI's mission to ensure that AGI benefits all of humanity, no sale price could adequately compensate the nonprofit for what it would be giving up — control over the company that is arguably closest to building AGI.

The authors essentially say: this isn’t a debate about fair market value. The whole premise of the nonprofit is that AGI governance isn’t something you can price — and that no board serving the public could ever justify giving it up for cash, no matter the number.

Actually, I can think of one outcome that could potentially satisfy this condition. As strong a claim as OpenAI has to leadership of the AI industry, it's only one company. If it slows down for the sake of safety, others could overtake it. So perhaps the OpenAI nonprofit would better advance its mission if it were spun out into a truly independent entity with $150 billion and the mission to lobby for binding domestic and international safeguards on advanced AI systems.

If this sounds far-fetched, then so should the idea that the nonprofit board that initiated this conversion is genuinely representing the public interest.

An institutional test

The question of whether this restructuring is legal is now before two state AGs. But whether it goes forward may ultimately come down to politics. California and Delaware's AGs may be officers of the law, but they're also elected by the people.

Delaware AG Kathleen Jennings has already publicly stated that her office is investigating the proposal, and CalMatters reported in January that California AG Rob Bonta's office was doing the same.

And in a letter sent last week to Musk's legal team, Bonta's office denied Musk's request for "relator status," which would have allowed him to sue OpenAI in the name of the State of California. The AG's office specifically noted that Musk appeared to have personal and financial interests in OpenAI's assets through his competing company, xAI.

Musk's suit makes legal arguments, but can't be fully separated from the man himself, whose far-right turn has made him politically toxic in Democratic circles (both the AGs in question are Democrats in deep blue states).

By rejecting Musk's bid to insert himself as California's representative, Bonta signaled that his office intends to maintain direct control over any potential enforcement actions regarding OpenAI's charitable purpose.

The letter’s authors essentially argue that this moment is a test — not just for OpenAI’s board, but for the institutions meant to safeguard the public interest. The attorneys general of California and Delaware have both indicated they’re paying attention.

Whether they act may determine if OpenAI’s fiduciary duty to humanity is a real constraint — or just another marketing line sacrificed to investor pressure.

Edited by Sid Mahanta.

Subscribe now

Appendix: Quotes from OpenAI’s leaders over the years

The letter includes some great quotes from Altman and Brockman — some of which I hadn’t actually seen before. I’m including them here with links for convenience.

Sam Altman in 2017: "That’s why we’re a nonprofit: we don’t ever want to be making decisions to benefit shareholders. The only people we want to be accountable to is humanity as a whole."

Greg Brockman in 2019: "The true mission isn’t for OpenAI to build AGI. The true mission is for AGI to go well for humanity… our goal isn’t to be the ones to build it, our goal is to make sure it goes well for the world."

Altman in 2020: "The problem with AGI specifically is that if we’re successful, and we tried, maybe we could capture the light cone of all future value in the universe. And that is for sure not okay for one group of investors to have."

Altman in a 2015 email to Musk: "we could structure it so that the tech belongs to the world via some sort of nonprofit but the people working on it get startup-like compensation if it works. Obviously we’d comply with/aggressively support all regulation"

OpenAI LP 2019 announcement:

We’ve designed OpenAI LP to put our overall mission—ensuring the creation and adoption of safe and beneficial AGI—ahead of generating returns for investors… Regardless of how the world evolves, we are committed—legally and personally—to our mission.

…

All investors and employees sign agreements that OpenAI LP’s obligation to the Charter always comes first, even at the expense of some or all of their financial stake.

Altman on the hybrid model in 2022 (at 35:16):

We wanted to preserve as much as we could of the specialness of the nonprofit approach, the benefit sharing, the governance, what I consider maybe to be most important of all, which is the safety features and incentives.

Brockman before House subcommittee hearing in 2018:

On the ethical front, that’s really core to my organization. That’s the reason we exist . . . when it comes to the benefits of who owns this technology? Who gets it? You know, where did the dollars go? We think it belongs to everyone.

Altman in 2018:

At OpenAI when we wrote our charter, we talked about the scenarios where we would or wouldn’t make money. And… the things we wouldn’t be willing to do no matter how much money they made. And we made this public so the public would hold us accountable to that. And I think that’s really important.

A paywalled version of this piece was also published as a guest post for Michael Spencer’s great Substack, AI Supremacy (along with some useful context and very kind words about my work from Michael). Here is the piece in full.

Earlier this month, OpenAI announced that it aspires to build "the best-equipped nonprofit the world has ever seen" and was convening a commission to help determine how to use its "potentially historic financial resources."

But critics view this new commission as a transparent attempt to placate opposition to its controversial plan to restructure fully as a for-profit — one that fails to address the fundamental legal issues at stake.

OpenAI is currently a $300 billion for-profit company governed by a nonprofit board. However, after an earlier iteration of that board briefly fired CEO Sam Altman in November 2023, investors reportedly began demanding that the company shed its quasi-nonprofit status.

Subscribe now

"The story of OpenAI's history is trying to balance the desires to raise capital and build the tech and stay true to its mission," a former OpenAI employee told me. The current move, they say, is an attempt to "separate these things" into a purely commercial entity focused on profit and tech, alongside a separate entity doing "altruistic philanthropic stuff."

"That's wild on a number of levels because the entire philanthropic theory of change here was: we're going to put guardrails on profit motives so we can develop this tech safely," the former employee says.

Legal hurdles

The for-profit conversion faces significant unresolved legal challenges, including a lawsuit from Elon Musk arguing that his $44 million donation was contingent on OpenAI remaining a nonprofit and that the conversion would violate its founding charitable purpose. The case will go to trial this fall. The conversion can also be challenged by the California and Delaware Attorneys General (AGs), who are reportedly each looking into the case.

Musk's suit, OpenAI's gargantuan valuation, and the unprecedented nature of the conversion attempt appear to have attracted scrutiny.

Without mentioning OpenAI explicitly, California Assembly Member Diane Papan introduced a bill in February that would have blocked the conversion. However, the legislation was amended without explanation earlier this month to instead focus on aircraft liens. Papan's office has not replied to a request for comment.

OpenAI countersued Musk last week, asking a federal judge to halt what it called a "relentless campaign" of harassment designed to harm the company.

Adding fuel to the fire, a group of twelve former OpenAI employees, represented by Harvard Law Professor Lawrencre Lessig, filed an amicus brief on Friday supporting Musk's challenge to the conversion.

The brief argues that OpenAI's nonprofit structure wasn't merely administrative — it was fundamental to the organization's mission and key to recruiting talent who were told they were building AI that would benefit humanity rather than shareholders. Former employees contend that removing the nonprofit's controlling role would constitute a betrayal of the trust that drew many of them to the company in the first place.

In an accompanying declaration, Todor Markov stated he left OpenAI after losing trust in leadership, concluding that the organization's Charter "had been used as a smokescreen, something to attract and retain idealistic talent while providing no real check on OpenAI’s growth and its pursuit of AGI." The proposed restructuring plan, he writes, "has only served to further convince me that OpenAI’s Charter and mission were used all along as a facade to manipulate its workforce and the public."

Markov reiterated this point in a written statement to me:

The fundamental question about the OpenAI corporate restructuring is whether the nonprofit will maintain legal control over the for profit. The announcement of the OpenAI commission does not address that question in any way, and so does nothing to alleviate the substantial concerns we raise in our amicus brief.

Fearing that billions in charitable assets could be transferred to private hands without sufficient oversight, a coalition of dozens of California-based nonprofits began organizing and urged the state's AG in January to investigate the OpenAI conversion, seeking transparency about the valuation process and demanding assurance that the nonprofit will remain truly independent from commercial interests.

One of the coalition leaders, LatinoProsperity CEO Orson Aguilar, says that the commission announcement reminded him of 2008, "when the financial institutions that helped crash the economy decided that the solution was teaching everyone else financial literacy."

OpenAI's original nonprofit mission was, and, at least for now, remains, to ensure AGI benefits all of humanity. This purpose is enshrined in its Charter, which defines AGI as "a highly autonomous system that outperforms humans at most economically valuable work." In 2019, when the company spun up a for-profit arm to raise the billions needed to train increasingly expensive AI models, it gave the nonprofit board ultimate control. That board has a fiduciary duty to humanity — not shareholders.

OpenAI has not replied to multiple requests for comment.

The nonprofit's control over OpenAI became global news in November 2023, when the board dramatically exercised its authority by firing CEO Sam Altman — cryptically citing his failure to be "consistently candid." Altman orchestrated a swift comeback with the help of Microsoft and a revolt of the employees (whose ability to sell billions in equity hung in the balance).

The Wall Street Journal recently shed new light on the firing, reporting that OpenAI executives collected dozens of examples of Altman's "alleged lies and other toxic behavior, largely backed up by screenshots," such as falsely saying the legal department approved a release without safety testing.

The ouster was brief, but still served as a potent reminder to investors that the nonprofit board was, at least formally, in control and that its fiduciary duty was to all of humanity — not them.

Rose Chan Loui, founding executive director of the Lowell Milken Center on Philanthropy and Nonprofits at UCLA Law School, says that OpenAI's proposed commission "confirms their intent to make this nonprofit a typical corporate foundation." She continues, "Now can it do a lot of good? Yes, absolutely. But it's still, from our perspective, abandonment of their original purpose. Either that or it's a big stretch of their purpose."

Her colleague Michael Dorff, executive director of the Lowell Milken Institute for Business, Law, and Policy also at UCLA Law, echoed this skepticism. "I'm trying not to be terribly cynical," he told me. "On the one hand, it's commendable that OpenAI is thinking of ways to use their tech to fulfill their nonprofit mission. But I don't think that should have anything to do with whether the nonprofit can abandon its mission."

"Pandering"

OpenAI's announcement describes a commission that will help understand "the most urgent and intractable problems nonprofits face" and incorporate feedback from leaders in health, science, education and public services — "particularly within OpenAI's home state of California."

That last detail isn't subtle, and critics see it as telling.

OpenAI is "pandering," the former employee says.

"Most of the nonprofit and philanthropic world doesn't care about AI safety. And presumably the California AG and the people who he cares about don't know anything about AI safety or the actual premise of OpenAI's purpose and mission," the former employee says. The specific mention of California in the plan for a "wildly well-funded science and education nonprofit," they say, makes "the pandering pretty obvious. So it feels like a bribe to California, to the California nonprofit sector — the sector that might be up in arms about this nonprofit conversion."

On Tuesday, OpenAI announced the advisors for this commission. The group will be convened by Daniel Zingale, a former senior advisor to California governors Gavin Newsom and Arnold Schwarzenegger. The advisors include iconic labor leader and civil rights activist Dolores Huerta, who cofounded United Farm Workers with Cesar Chavez; Monica Lozano, former CEO of the largest Spanish-language newspaper in the US; Dr. Robert K. Ross, former president and CEO of The California Endowment, an influential statewide healthcare foundation; and Jack Oliver, a lawyer and private equity partner who previously co-chaired Bono's ONE Campaign.

None of the commissioners immediately replied to requests for comment.

The commission's stated goal is to gather input on how OpenAI's philanthropy can tackle systemic issues, incorporating feedback from leaders in health, science, education, and public services — with a particular focus on California. The advisors are tasked with submitting findings to the OpenAI board within 90 days.

The advisors have no clear experience in AI governance but do have deep connections to California and backgrounds in civil rights, education, and community advocacy. Their selection does little to dispel the criticisms that the commission is primarily an attempt to smooth the path for this conversion, particularly with the California AG.

The effort, at least with Aguilar of LatinoProsperity, seems to be falling flat. "Pandering is the right word," he told me prior to the announcement of the advisors. OpenAI's commission is "a reaction to the work of our campaign and others to try to make it seem that as though they're listening, but they're distracting from the real questions," he says.

Those questions, according to Aguilar: Is OpenAI a truly independent nonprofit? And what is the real value of what it has right now?

In a follow-up email sent after the commission members were announced, Aguilar was even more pointed: "As impressive as OpenAI's advisory commission members may be, let's call this what it is — a calculated PR stunt to distract us from the real issue: OpenAI funneling nonprofit assets into private pockets."

Conflicts of interest

OpenAI's nonprofit board has a fiduciary duty to represent the interests of the public and the nonprofit, which includes ensuring that it is fairly compensated for whatever it gives up in the conversion.

In an email, Aguilar writes, "The fundamental question remains: How can a nonprofit commission maintain true independence when housed within an organization with significant commercial pressures?"

The IRS requires charities to publicly disclose conflicts of interest of its board members, and views a lack of majority independence as a significant governance risk factor that may invite greater scrutiny. Chan Loui says that the way OpenAI defined independence when it started the for-profit in 2019 was based on whether the director has equity in the company. However, she says, the law also looks at whether you have financial interests in the organization's "partners," such as suppliers or customers.

OpenAI lists ten "independent" nonprofit board members on its site, including CEO Sam Altman. However, at least seven of these directors or their spouses have significant investments in companies that already do business with OpenAI, according to SEC filings, news reports, and Crunchbase data. This includes the board chair, Bret Taylor, who founded the $4.5 billion AI startup, Sierra, which is a customer of OpenAI (Taylor has publicly committed to recuse himself from decisions in which he's conflicted).

The wrong question?

A lot of the conversation around OpenAI's restructuring has focused on whether the nonprofit will be fairly compensated for what it's giving up — namely control over the for-profit and the money that exceeds extremely high profit caps.

In February, Musk offered to buy the nonprofit's assets for $97.4 billion, in a likely attempt to derail the conversion or drive up the price the for-profit has to pay to the nonprofit to give up its control.

The nonprofit coalition has emphasized the importance of adequate compensation as well. Aguilar writes to me, "It's crucial that the Attorney General provide a fair market valuation of OpenAI's charitable assets to ensure proper oversight and protection of public interest."

However, others think this focus and the commission announcement misses the fundamental legal question: Does OpenAI's restructuring advance its charitable purpose?

The former employee says, "I would like the entire question about fair market value to be taken off the table because I think that's just the wrong question" because it "treats this as a normal corporate transaction where fair market value is what matters."

Given OpenAI's purpose is to ensure AGI is built safely, they ask, "What better position could you be in than literally controlling the company on the brink of building AGI? What amount of money could you get in the transaction to put you in a better position to realize that mission?"

They offer an analogy:

Imagine you are a nonprofit whose mission is to ensure nuclear technology benefits humanity. And you literally have a controlling interest in the Manhattan Project. And it's 1943. For what amount of money should you sell that interest?

Dorff agrees, stating plainly, "I don't see any amount of money that would allow the nonprofit to better pursue its mission." After all, he notes, it currently controls the market leader in the AI space.

That market leadership was recently underscored by OpenAI's Wednesday release of o3, its latest and most capable "reasoning" model to date.

OpenAI says that o3 sets new state-of-the-art performance on difficult benchmarks for coding, math, and science, significantly improving on its predecessor, o1.

While OpenAI asserts the model remains below the "High" risk threshold defined in its Preparedness Framework, the relentless push toward more powerful and autonomous systems highlights the immense potential value and risk embodied in the technology the nonprofit currently oversees — and the very control it is being asked to relinquish.

Chan Loui calls the nonprofit's position "priceless." Its control of the company is beyond what the typical watchdog organizations can do, she says. The conversion is not, in her view, "just an interpretation of how to fulfill purpose," but rather "a change of purpose." "Under the law, they would need to go to court and say we have a basis for changing our purpose."

Dorff says "I haven't seen anything remotely close to a justification for" a change in purpose. "It's a very steep burden to show that a nonprofit's mission is no longer viable."

OpenAI has benefited a lot from its nonprofit status. Chan Loui notes that the emails released as a result of the Musk lawsuit "demonstrate that their reasoning really was driven by recruiting needs" — a point supported by the ex-employee amicus brief.

"That was really the main benefit of going out there and saying, 'We're a nonprofit. We really care about developing AI safely and for the benefit of humanity.'" "You can't just abandon your purpose now that you are in this position," she says.

Safety shakeups

The conversion attempt comes amidst a year of headline-generating departures of OpenAI leadership and safety staff.

On Tuesday, I reported in Obsolete that Joaquin Quiñonero Candela had quietly stepped down from his role leading the team focused on catastrophic risks, less than nine months after the previous lead was reassigned without an announcement. Candela announced the move on LinkedIn, describing it as a transition to an "intern" role focused on healthcare applications.

An OpenAI spokesperson said that safety governance is now consolidated under a Safety Advisory Group (SAG) chaired by Sandhini Agarwal and preparedness work is distributed across teams.

This marks yet another significant shakeup in OpenAI's safety leadership following a year of high-profile exits — including cofounders John Schulman and Ilya Sutskever, safety systems lead Lilian Weng, Superalignment co-lead Jan Leike, and Senior Advisor for AGI readiness Miles Brundage — and the disbanding of both the Superalignment and AGI Readiness teams.

And it comes amidst recent reports that OpenAI dramatically reduced safety testing times and released powerful new models like DeepResearch and GPT-4.1 without promised safety reports, raising further doubts about the company's founding commitment to ensure that AGI is built safely.

The path forward

OpenAI's announcement states that commission members will submit insights to the board within 90 days. The board will "consider these insights in its ongoing work to evolve the OpenAI nonprofit well before the end of 2025."

That timeline is significant — the Wall Street Journal recently reported that if OpenAI fails to convert by the end of 2025, it will have to return $20 billion of the $40 billion it recently raised in a fundraising round valuing the company at $300 billion. The company's $6.6 billion investment from October carries a similar condition, requiring conversion by October 2026 to avoid potential investor clawbacks with ten percent interest.

Chan Loui called this "a very aggressive deadline" that regulatory authorities may struggle to accommodate. "You can't hurry the attorneys general. They don't really have a deadline," she said. "In California, you're to give notice of any significant transactions, which is what this proposed restructure is," she says, and "there's no deadline for when they decide."

She speculates that these deadlines might be a move to speed up the regulatory authorities.

According to Dorff, there are only two ways the nonprofit mission could be enforced: through Musk's lawsuit, which he says definitely won't see a verdict before the end of this year, or through action by the California or Delaware AGs.

"The only way to meet that deadline that I can see is for OpenAI to settle with everybody," Dorff said. "Elon would have to agree," and OpenAI "would need some kind of indication of satisfaction from the AGs."

Aguilar says he recently met with executives at OpenAI, along with former Housing and Urban Development Secretary Julián Castro, and fellow coalition leader Fred Blackwell. Aguilar says that the OpenAI executives listened and were "very eager to get our thoughts on mission," but no details were shared. The meeting hasn't dissuaded the coalition, which Aguilar says has grown to around 50 organizations and recently filed an administrative petition calling on the California AG to investigate the conversion.

So while OpenAI works to construct the appearance of a graceful transition, the legal challenges remain daunting. No matter how well-resourced the spun-out nonprofit might be, many experts say it cannot replace the core mission enshrined in the original nonprofit.

"If OpenAI wants to give many billions to science and education in California, that's great. I'm very supportive of that," the former employee says. "But that's not its mission. They can't use that as an out in this situation."

OpenAI was founded as an alternative to the perils of letting commercial interests dictate the development of a potentially transformative — and dangerous — technology. A decade later, as the AI race it helped supercharge reaches unprecedented intensity, OpenAI is looking to shed one of the last vestiges of that original intent.

The former employee put it simply: "I view what's happening now is: the profit motive's winning. They have given up on the altruistic angle. They've given up on trying to be the good guy, and they just want to win."

Subscribe now

Joaquin Quiñonero Candela, who took over OpenAI's Preparedness team in July, announced on LinkedIn that he has taken on a new role at the company:

I'm an intern! After 11 years since my last commit, I'm back to building. I first transitioned to management in 2009, and got more and more disconnected from code and hands-on work. Three weeks ago, I turned it all upside down, and became an intern in one of our awesome teams that's focused on healthcare applications of AI.

Candela's LinkedIn bio now describes him as the "Former Head of Preparedness at OpenAI."

Subscribe now

An OpenAI spokesperson told Obsolete that Candela "was really closely involved in preparing the successor to the preparedness framework" and "will probably be involved in preparedness in some capacity" but is currently "focusing on different areas within the company that he's really excited about."

The spokesperson added that the company recently restructured its safety organization, consolidating "all governance under the Safety Advisory Group" (SAG) — a committee chaired by five-year OpenAI veteran Sandhini Agarwal. The SAG uses a rotational leadership structure with one-year terms, designed, they said, to balance "continuity of knowledge and expertise" with "fresh and timely perspectives."

Meanwhile, OpenAI's preparedness work is now distributed across multiple teams, focused on things like capabilities, evaluations, and safety mitigations, the spokesperson said.

Candela's departure from the team comes amidst OpenAI's contentious attempt to shed the last vestiges of nonprofit control and follows a string of scandals and high profile exits in the last year.

It also marks the second major shakeup in the Preparedness team’s short history. In July, OpenAI removed Aleksander Mądry from his role as head of Preparedness — also without a public announcement. The Information reported that the MIT professor was reassigned to work on AI reasoning just days before US senators sent a letter to CEO Sam Altman regarding "emerging safety concerns" at the company.

Following Mądry's reassignment, Candela took over, and Tejal Patwardhan, a 2020 Harvard graduate, began managing day-to-day operations, according to The Information story.

Mądry's quiet move reflects a pattern of leadership changes to OpenAI's safety teams that continues with Candela's departure.

The Preparedness team was established in December 2023 to track and mitigate "catastrophic risks related to frontier AI models," according to the company's Preparedness Framework, which was introduced as "a living document describing OpenAI’s processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models."

The Framework focuses on risks related to cybersecurity, persuasion, model autonomy, and chemical, biological, radiological, and nuclear weapons.

OpenAI published the second version of its Preparedness Framework around noon, local time today, shortly after Obsolete contacted the company for comment.

The crumbling of safety leadership

OpenAI has seen an exodus of leadership and safety staff in the last year.

Company cofounder and AI alignment lead, John Schulman, left in August for Anthropic, a rival firm started by an earlier wave of departing OpenAI safety staff.

Lilian Weng, OpenAI's safety lead, left in November and subsequently joined Thinking Machines Labs — a startup launched earlier this year by Mira Murati, who served as OpenAI's CTO from 2022 until her abrupt departure amidst the company's October fundraising round. Schulman joined Murati's company in February.

OpenAI's Superalignment team, tasked with figuring out how to build smarter-than-human AI safely, was disbanded in May. The team leads, OpenAI cofounder Ilya Sutskever and longtime safety researcher Jan Leike, both left the same month. During his departure, Leike publicly stated that "safety culture and processes have taken a backseat to shiny products" at OpenAI. Fortune reported that the Superalignment team never got the computing power it was promised.

In October, Miles Brundage, OpenAI's Senior Advisor for AGI readiness, resigned after more than six years at the company; his team was disbanded and absorbed into other departments.

Brundage was one of the last remaining members of OpenAI's early safety-focused staff and had been increasingly vocal about his concerns. In his departure announcement, he wrote that "neither OpenAI nor any other frontier lab is ready" for artificial general intelligence (AGI) — the very technology the company is explicitly trying to build. He cited publishing constraints as one reason for leaving, suggesting the company was restricting what he could say publicly about AI risks.

Brundage also broke with Altman by advocating for cooperation with China on AI safety rather than competition, warning that a "zero-sum mentality increases the likelihood of corner-cutting on safety."

A former senior OpenAI employee told Obsolete that Mądry's reassignment was particularly alarming. "At a certain point, he was the only person in there with a safety-focused role who was empowered at all," the former employee said.

Who is leading safety at OpenAI?

With most safety-focused leaders gone or reassigned, OpenAI's formal governance structure has become increasingly important — but also increasingly opaque.

In May, OpenAI announced the creation of its Safety and Security Committee (SSC), tasked with making recommendations to the full board on "critical safety and security decisions for OpenAI projects and operations." Its original members included a subset of its nonprofit board, including Altman, along with Madry, Weng, Schulman, Matt Knight, the head of security, and Jakub Pachocki, the chief scientist.

Of these original members, only Knight and Pachocki remain in these or similar roles at OpenAI.

OpenAI announced in September that board member and Carnegie Mellon professor Zico Kolter would join the SSC as its chair and that Altman was no longer on the committee. When asked about Altman's departure, the OpenAI spokesperson declined to comment.

The updated version of the Preparedness Framework published today goes into more detail on the roles, responsibilities, and decision-making processes of the SSC and introduces the "Safety Advisory Group (SAG)" — a related committee made up of OpenAI staff.

However, the updated document does not identify the members of the SAG. According to the OpenAI spokesperson, the SAG has been working under Agarwal's leadership for two months. They described her as "functionally heading up all of the governance work," including "all of the evaluation calls about what [risk] mitigations are necessary."

The lack of transparency around safety leadership extends beyond public announcements. “Even while working at OpenAI, details about safety procedures were very siloed. I could never really tell what we had promised, if we had done it, or who was working on it,” a former employee wrote to Obsolete.

Growing concerns

These leadership changes come amid mounting questions about OpenAI's commitment to safety.

The Financial Times reported last week that "OpenAI slash[ed] AI model safety testing time" from months to days. When asked about this story, the company spokesperson directed Obsolete back to the updated Preparedness Framework — saying that "our safety practices continue to be really rigorous" and suggesting that characterizations of reduced testing were not "very fair."

And just yesterday, the company released GPT-4.1 without publishing a corresponding safety report. OpenAI's announcement touts the model's significant improvements over its flagship multimodal model, GPT-4o, in areas like coding and instruction following.

Conducting pre-release safety evaluations on frontier AI models and publishing the results alongside the model launch has become a common practice for the industry — one that OpenAI committed to at the 2024 Seoul AI Summit.

When questioned by TechCrunch, OpenAI claimed that "GPT-4.1 is not a frontier model, so there won't be a separate system card released for it."

However, the company released DeepResearch, a powerful web-searching tool, weeks before publishing a safety report, which refers to the product as a frontier model.

Following the release of the updated Framework, former OpenAI safety researcher Steven Adler tweeted that he's "overall happy to see the Preparedness Framework updated." But he also called out the company for "quietly reducing its safety commitments," pointing to OpenAI's abandonment of an earlier promise to conduct safety testing on models finetuned to perform better in certain risky domains, like bioengineering.

Safety reports have been a primary tool for transparency in the AI industry, providing details on testing conducted to evaluate a model's risks. After conducting safety evaluations, OpenAI and Anthropic each found that their most advanced models are close to being able to meaningfully assist non-experts in the creation of bioweapons. And OpenAI had previously called system cards "a key part" of its approach to accountability ahead of the 2023 UK AI Safety Summit.

In the United States, frontier AI developers are governed by voluntary commitments, which they can violate without real consequence. Many of these companies, including OpenAI and Google, lobbied hard last year against California AI safety bill SB 1047, the most significant effort to codify some of these commitments.

As AI models get more capable and autonomous, companies appear to be increasingly cutting corners on safety.

Google's Gemini 2.5 Pro model is considered by many to be the most capable on the market, but the company still hasn't released a safety report, which Fortune reported last week violated voluntary commitments the company made to the White House and at the Seoul summit.

The competitive pressure to release faster and with fewer safeguards will likely increase from here, raising alarming questions about whether meaningful guardrails will be in place when they're needed most.

Subscribe now

Overview

What: Support the completion of a forthcoming general-audience nonfiction book on AI. You'll conduct research, draft chapters, edit existing content, and help transform an incomplete manuscript into a polished final product for publication by OR Books and The Nation Magazine.

Why: The project, Obsolete: Power, Profit, and the Race to Build Machine Superintelligence, has the potential to be the go-to AI risk book in the post-ChatGPT era. You’ll also get the opportunity to work closely with an experienced journalist and writer who has a track record of publishing work in leading outlets (NYT, Nature, BBC, TIME, Foreign Policy, etc.).

Start Date: As soon as possible

Employment Type: Fixed term (6 months, with possibility of extension); full time

Location: Remote, provided at least three hours overlap between 10am-6pm ET.

Compensation: $37,485-$65,442 total compensation for the 6-month period, depending on experience and location

Applications will be considered on a rolling basis, but please apply as soon as you can. Priority may be given to applicants who can move through the process sooner.

Apply here

Details

About the Role

I'm seeking a full-time Research Assistant to work with me on my forthcoming nonfiction book on AI, titled Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. The book will be published by a new joint imprint of OR Books and The Nation magazine. The manuscript is due mid-September 2025 and publication expected in Spring 2026. Accordingly, this is intended as a 6-month fixed term position (with some possibility of extension). Hours should average 40/week, but may increase with proximity to the due date.

In this role, you will have the opportunity to contribute to a timely examination of the powerful forces shaping AI development. Additionally, you will work closely with an experienced journalist whose work has been featured in leading publications and recognized by experts in the field.

The Book Project

The book (currently ~60% complete) focuses on how commercial and great power competition shape the race to build increasingly powerful and autonomous AI systems. You can find a full description of the book at the end of this document.

I aim for this to be the definitive book on AI risk in the post-ChatGPT era. Given the subject matter, timing, and the support of a major publication (The Nation), Obsolete has the potential to reach large and influential audiences.

Who I Am

I'm Garrison Lovely, a freelance journalist reporting on the intersection of economics, geopolitics, and artificial intelligence. My writing has appeared in The New York Times, Nature, BBC, TIME, The Thomson Reuters Foundation, Foreign Policy, The Verge, The Guardian US, Vox, and many other outlets. I've written cover stories for The Nation and Jacobin, with my piece "Can Humanity Survive AI" leading to my current book deal. I'm also a Reporter in Residence at the Omidyar Network and publisher of Obsolete, a fast-growing Substack on AI.

My reporting and commentary on AI has been shared by the three “godfathers” of deep learning (Yoshua Bengio, Yann LeCun, and Nobel laureate Geoffrey Hinton), noted AI critic Gary Marcus, Life 3.0 author Max Tegmark, and many others. I've spoken on AI at Harvard, the Federation of American Scientists, and the Fund for Alignment Research (FAR Labs).

My writing has been translated into 5 languages and cited dozens of times by mainstream outlets (including the New Yorker, NYT, The Atlantic, ProPublica, The Brookings Institution, The Guardian, Axios, and NY Magazine). My media appearances have received over 8 million combined views/listens, and my social media posts have gained over 18 million impressions.

Key Responsibilities

As Research Assistant, your responsibilities will include:

• Conducting in-depth research on assigned topics, synthesizing findings into clear, well-organized reports

• Writing first drafts of book chapters based on detailed outlines I provide

• Editing and refining existing drafts for clarity, flow, and accuracy

• Supplying and formatting references

• Tracking developments in AI news, research, and policy relevant to the book's themes

• Assisting with fact-checking and source verification

• Meeting regular deadlines and communicating proactively about progress

Depending on the candidate chosen, there may also be opportunities to:

• Help write excerpts of the book for leading publications

• Continue research and writing work for future multimedia AI-related projects

Qualifications

The ideal candidate is likely to have:

• Research and/or writing experience, as evidenced by an extensive portfolio (which can include blog posts as well as other publications)

• Excellent research skills, with the ability to synthesize complex information from diverse sources

• Strong writing skills, especially in nonfiction, journalistic, or academic writing

  • Ability to produce clear, explanatory, fun, and engaging material, without compromising accuracy. See my Jacobin story for what I aim for.

• Self-motivation and ability to work independently while meeting deadlines

• Attention to detail, particularly in fact-checking and citation management

• Extensive familiarity with the frontier AI industry, AI policy, and/or AI safety literature (strongly preferred)

• A background in economics and/or international relations is a plus

Location

This is a full-time, remote-friendly job, with slight preference for those based in or willing to visit NYC. Work hours are flexible, provided at least 3 hours of overlap between 10am and 6pm ET daily.

I am happy to consider candidates based outside of the US, but will not be able to provide visa sponsorship at this time.

Compensation & Benefits

• Salary: $30,000-$50,000 for the 6-month period, depending on experience and location ($60,000-$100,000 annualized)

• Payroll tax reimbursement: $2,485-$4,142

• Health insurance stipend: $3,000-$4,800

• Additional perks: $2,000-$6,500

Total compensation package: $37,485-$65,442

Other Benefits:

• Mentorship from an established journalist and writer

• Potential for continued collaboration beyond the initial 6-month period

• Opportunity to be extensively credited in acknowledgements in a published book

Application Process

After considering written applications, promising candidates may be invited to complete a short work trial followed by interviews.

I’ll ask finalists to provide references and participate in a longer paid work trial.

I am committed to fostering a culture of inclusion, and I encourage individuals with diverse backgrounds and experiences to apply.

More on the Book

Depending on who you ask, artificial intelligence is our salvation or our doom; an overhyped, bigoted bullshit artist or the ticket to immortality and utopian abundance; just another tech fad or the last invention we need ever make. The AI discourse is confusing, messy, and frustrating. But the technology — and our response to it — will shape the future, whether or not we're part of the conversation.

Obsolete is for those who are interested in learning more about AI, but are unsure of where to start and who to believe. They may feel intimidated by the technical jargon, put off by dry and abstract prose, or skeptical of the loudest voices on the issue (like Elon Musk and Sam Altman). Obsolete will introduce readers to the basics of AI, the idea that it could lead to human extinction, the roiling three-sided debate surrounding extinction fears, and the people and companies trying to build artificial general intelligence (AGI) — that which can outwit humans across the board. The book will grapple with the core arguments animating the AI debates, which are, by turns, uncritically parroted and unduly dismissed. It will also cut through industry hype while seriously entertaining the implications of AGI.

The risk that AGI could result in our extinction is being recognized by a significant and growing number of leading AI researchers, industrialists, and policymakers, along with the wider public. Existential risk from AI has been explored in other books, but Obsolete will be the first to center its analysis on how both capitalist and great power competition make AI more dangerous.

Obsolete will also tackle questions like: Can machines actually outsmart humanity? If so, when could that happen? If AGI is possible, is it inevitable? Why are people trying to build a technology they claim could end the world? Is the idea of AI-driven extinction the product of a big tech conspiracy aiming to hype the technology and control its regulation? Why has the left mostly ignored or dismissed existential risk from AI? Why do some powerful techies welcome human extinction? How could AI enable stable authoritarian regimes? How could killer robots reshape war and the balance of power? What do China and the US want from AI? And why has it become the front line of their brewing Cold War? And finally: What can and should we do about it?

Apply here

On Tuesday, California District Judge Yvonne Gonzalez Rogers denied all of Musk's requests for a preliminary injunction, which would have blocked OpenAI's restructuring from nonprofit to for-profit. Judge Rogers also expedited the trial, which will now begin this Fall. Media outlets quickly framed this as a loss for Musk.

But a closer reading of the 16-page ruling reveals something more subtle — and still a giant potential wrench in OpenAI's plans to transfer control of the company from the nonprofit board to a new for-profit public benefit corporation.

Subscribe now

If the company fails to complete this transformation by October 2026, investors in its $6.6 billion funding round last October can ask for their money back.

OpenAI CEO Sam Altman, nonprofit board director Bret Taylor, and the company's press office did not reply to requests for comment.

In a narrow sense, Musk did "lose." However, as Judge Rogers notes, the bar for a preliminary injunction is extremely high. To surpass it, both the facts of the case and the legal questions involved have to clearly point in the same direction.

Therefore, it would have been extremely surprising if the judge granted the injunction. She did, however, stop just short of it.

Does Musk have standing?

To bring a lawsuit, you have to have standing, i.e. you have to convince the court that you've been harmed by the actions of the defendant. For most of the claims Musk brought, Judge Rogers thinks that he doesn't have standing.

But Musk's third claim is that his donation of $44 million to OpenAI was contingent on his expectation that the organization remain a nonprofit. The legal question is whether Musk's donation meant that he entered into a "charitable trust" with OpenAI, in which he expressly meant for his gift to only be used if the organization remained a nonprofit.

Unfortunately for Musk (and fortunately for OpenAI), there is no contract or gift agreement documenting any restrictions on the gift. Judge Rogers says that this question is a "toss-up," citing evidence that points in both directions. (OpenAI published emails showing Musk was aware of and on board with the transition to a for profit, which makes him far less sympathetic here.)

So why does this ruling matter? Well, while Judge Rogers found Musk's standing uncertain at this preliminary stage, she went out of her way to signal that the core claim — that OpenAI's conversion violates its charitable purpose — could have merit if properly brought before the court.

Put differently, Judge Rogers essentially writes that if Musk clearly had standing, the injunction would be justified. Here's the key quote from her judgment:

if a trust was created, the balance of equities would certainly tip towards plaintiffs in the context of a breach. As Altman and Brockman made foundational commitments foreswearing any intent to use OpenAI as a vehicle to enrich themselves, the Court finds no inequity in an injunction that seeks to preserve the status quo of OpenAI’s corporate form as long as the process proceeds in an expedited manner. [emphasis original]

Legal experts who have followed the case closely see this ruling as far more significant than the headlines suggest — a decision that invites other existential challenges to OpenAI's conversion efforts.

"This is a big win for Musk," says Michael Dorff, the executive director of the Lowell Milken Institute for Business, Law, and Policy at UCLA. "Even though he didn't get the preliminary injunction, the fact that there is a pending trial on this issue and that his claim wasn't denied is a pretty big impediment to [OpenAI] moving forward expeditiously," he says.

A former OpenAI employee spelled out the significance of the judgment, telling Obsolete:

I think this is unusual, which is why it's noteworthy. Typically courts decide cases on the narrowest grounds possible. So if the standing decision is sufficient to deny, then none of the discussion about the underlying merits is decision-relevant. When courts do this, it's usually done with purpose.

If Musk was deemed to have standing, the former employee said the chances of the lawsuit prevailing on the merits was, "definitely over 75%. 90% isn't crazy. The judge doesn't have all the facts at this stage. All of the judge's emphasis in her denial was on standing, not the other question."

You know who does have standing?

Unlike Musk, the Attorneys General (AGs) in California and Delaware unquestionably have standing to challenge OpenAI's conversion — a fact Judge Rogers repeatedly emphasized throughout her ruling.

"The fact that the AG has standing is by statute, so it's not a big statement," notes Dorff. "What's unusual is that the AG might actually do something about it. This may be a rare case where an AG finds it worthwhile."

The AGs in both states have already signaled interest in the case. The Delaware AG filed an amicus brief in December emphasizing that her office would scrutinize any restructuring to ensure it protects the public interest. California's AG is reportedly reviewing the conversion as well.

Judge Rogers' ruling substantially increases pressure on both AGs to take action, providing them with judicial validation that the core issues deserve serious scrutiny.

The AGs are empowered to protect the public interest and could each initiate legal action to block the restructuring, with a strong likelihood of success given their clear standing and the court's signals about the merits of this case.

It's hard to change your purpose

Dorff agrees that the case’s merits pose significant challenges to the company, echoing past conversations with three other legal experts who followed this case.

"OpenAI has a very tough road ahead of it, if Musk has standing on that claim," Dorff explains. "Changing of a nonprofit's purpose is only supposed to be possible when the original purpose is defunct. That's not the case here."

To illustrate this point, Dorff cites the example of The March of Dimes. The anti-polio foundation was able to legally shift its mission after the disease was effectively eradicated. OpenAI's situation, Dorff argues, is fundamentally different.

"The original purpose was to develop AI for the benefit of all of humanity in a way that is safe," Dorff notes. "That purpose is not defunct — it's very much still ongoing."

Given this, Dorff says, "It's hard to imagine a good argument for why they should be allowed to change their purpose." The nonprofit's major asset, he points out, isn't land — it's the control the board has over OpenAI the company, which is at the forefront of developing powerful AI systems. "Owning control over that entity seems uniquely well suited to the nonprofit's purpose," Dorff says. "Giving up that control, even for a lot of money, is not equivalent."

Directors could be personally liable

The judgment raises another critical issue that has received little attention: potential personal liability for OpenAI's board members if they proceed with the conversion.

"If a breach of fiduciary duty is established, board members could be personally liable," explains Dorff. "If they're conflicted and violate their duty of loyalty in favor of their own interests instead of the public interest... they could be personally liable for the true value of whatever was lost."

The former OpenAI employee concurred, "Typically when directors are acting on behalf of organizations, there's some shield — the business judgment rule." Judges are reticent to second-guess business decisions that don't pan out. This gives directors wide leeway in governing companies.

But, "Under certain circumstances that protection doesn't apply. If I were a director, I'd want to be getting legal advice right now," the ex-employee says.

OpenAI's nonprofit board (in)famously has a fiduciary duty to humanity and therefore needs to justify the conversion on those terms, which is a far taller order.

In a follow-up exchange, the former OpenAI employee wrote to Obsolete:

Judge Rogers is very clearly saying that she has serious concerns about the legality of the restructuring. So after this ruling, the directors are on heightened notice that the legality of the restructuring is open to serious doubt. If they just try to ram it through regardless, that could be a pretty egregious breach of their fiduciary duties — so egregious that they might even have personal liability.

The prospect of being personally on the hook creates an entirely different category of risk for OpenAI's leadership as they navigate the restructuring.

Why OpenAI is trying to restructure

OpenAI's unconventional structure has become an albatross for the organization as it raises the staggering amounts of money needed to keep training cutting-edge AI models. Just four months after the October funding round that valued OpenAI at $157 billion, the Wall Street Journal reported that SoftBank was planning to invest up to $25 billion at a valuation of up to $300 billion.

OpenAI was confident enough in its ability to successfully complete its restructuring within two years that it gave investors the ability to take their $6.6 billion back if it didn’t convert in time.

But Tuesday's ruling threatens this timeline, even with the new expedited trial schedule. OpenAI faces a difficult choice: attempt to proceed with the restructuring under the cloud of pending litigation and potential AG intervention, or wait for legal clarity that may not arrive until the resolution of a trial that won't even begin until the Fall of 2025 — uncomfortably close to the October 2026 deadline when investors could demand their money back.

If OpenAI fails to transition and investors come calling, finding the money might not be enough (and is far from a given — the company is burning billions a year). OpenAI's ability to attract enough investment to compete may be dependent on it being structured more like a typical company. The fact that it agreed to such onerous terms in the first place implies that it had little choice.

What happens next

Dorff says that this case "will create fodder for discussions in law schools for many years to come."

The expedited trial schedule — which he calls "highly unusual" and "very, very quick" — suggests the court recognizes both the importance and urgency of resolving these questions. While OpenAI may continue preparations for its restructuring, the ruling represents a significant yellow light that both the company and potential investors cannot ignore.

And if the California or Delaware Attorneys General decide to act on the judge's signals, that yellow light could quickly turn red.

Far from a defeat for Musk, this ruling may ultimately prove to be the most substantial obstacle to OpenAI's plans to shed its nonprofit constraints — constraints that its founders once championed as essential to developing AI that benefits humanity.

Anthropic CEO Dario Amodei may have divulged a big secret with worrying implications for AI firms like his own and OpenAI.

Subscribe now

Last month, Chinese startup DeepSeek released R1, an AI model rivalling OpenAI’s flagship o1 model on key benchmarks but at a fraction of the cost — about 27-times cheaper. This sent shockwaves through the market. AI chip designer Nvidia saw a record $600 billion wiped from its value on Monday, January 27th, driving nearly $1 trillion in losses concentrated in American AI infrastructure stocks. The narrative quickly emerged: a small Chinese company had matched billion-dollar American models for mere millions, suggesting powerful AI might be far cheaper to develop than previously believed.

In an essay responding to the market panic, Amodei aimed to defend US export controls on advanced AI chips to China. But in doing so, he revealed something striking: DeepSeek's efficiency gains were exactly what we should expect from historical algorithmic progress — suggesting American AI companies have been enjoying healthy profit margins, at least until DeepSeek arrived to massively undercut them.

This assessment resonates with researchers at leading US AI companies. One told me DeepSeek's results "are within the improvement range that we'd expect from standard algorithmic improvement over time." Another was even more dismissive: "I don't think anyone cares very much, it doesn't seem very surprising… Obviously they're talented but nothing about it is unexpected." Notably, none of these reactions came from customer-facing employees.

By offering a comparable model at significantly lower prices, DeepSeek is likely to trigger an AI price war, just as it did in the Chinese market last summer. Lower prices should boost demand, foreshadowed by DeepSeek’s meteoric rise to the top of the iPhone app store.

Microsoft CEO Satya Nadella recognized this dynamic, tweeting: "Jevons paradox strikes again! As AI gets more efficient and accessible, we will see its use skyrocket, turning it into a commodity we just can't get enough of." The paradox he references: when technologies get more efficient, we often end up using more of them, not less. This commoditization would benefit companies using AI to enhance their existing services, like Microsoft, Meta, and Google.

However, for pure-AI companies like OpenAI, the implications are troubling. If US companies have already achieved similar efficiencies, then compute costs for these models won’t substantially decrease even if prices fall. This tightens unit economics for AI developers, lengthening their already uncertain path to profitability. OpenAI’s swift release of a “mini” version of its forthcoming o3 model, priced roughly double DeepSeek’s offering, suggests they recognize this threat.

And if Nadella is right about AI's coming status as a commodity, that's very bad news for OpenAI — commodity producers don't typically get valued at dozens of times their revenue.

DeepSeek demonstrates that the “secret sauce” for cutting-edge AI won’t remain secret for long. Months after OpenAI announced its o1 "reasoning" model, competitors have largely replicated its approach and performance. Whether through building off open-sourced innovations or “distillations” of closed models, “fast-followers” can match leaders' capabilities faster and cheaper.

While AI developers face margin pressure, the outlook for AI infrastructure providers like Nvidia is unclear.

Lower AI prices will likely drive up aggregate demand for computing power, but also reduce the profit per chip, which are currently astronomically high. The market initially bet on the downside. But early evidence suggests the demand effect may be winning out. Industry research group Semianalysis reported that prices to rent Nvidia's flagship H100 chip actually "exploded" after DeepSeek released its V3 model in December, with no slowdown after R1's introduction. "More intelligence for cheaper means more demand," they write.

Major tech companies seem to agree and are still planning massive AI and datacenter spending increases this year.

Public and private markets diverged wildly in their response to DeepSeek. The same week that Nvidia lost nearly 20% of its market cap, SoftBank reportedly sought to invest up to $25 billion in OpenAI at a valuation approaching $300 billion — nearly double where the company was valued just months earlier. The day of the DeepSeek market panic, Nvidia closed at a lower share price than it had in early October, when OpenAI announced its last funding round at a $157 billion valuation.

These contradictory outlooks can't both be right. A world of commodity AI services is fundamentally incompatible with the soaring private market valuations of AI companies that have yet to turn a profit.

(The market seems to have mostly corrected — Nvidia's stock is now only down one percent for the month.)

Ultimately, adoption of AI will continue and likely accelerate, as price drops coincide with significant increases to usefulness of the underlying technology. Pure-AI companies are in a long race to turn a profit before their products become commoditized. And DeepSeek just moved the finish line further away.

Technically, they can't actually do that, so I'm going to assume that Musk is trying to buy all of the nonprofit's assets, which include governing control over OpenAI's for-profit, as well as all the profits above the company's profit caps.

Subscribe now

OpenAI CEO Sam Altman already tweeted, "no thank you but we will buy twitter for $9.74 billion if you want." (Musk, for his part, replied with just the word: "Swindler.")

Even if Altman were willing, it's not clear if this bid could even go through. It can probably best be understood as an attempt to throw a wrench in OpenAI's ongoing plan to restructure fully into a for-profit company. To complete the transition, OpenAI needs to compensate its nonprofit for the fair market value of what it is giving up.

In October, The Information reported that OpenAI was planning to give the nonprofit at least 25 percent of the new company, at the time, worth $37.5 billion. But in late January, the Financial Times reported that the nonprofit might only receive around $30 billion, "but a final price is yet to be determined." That's still a lot of money, but many experts I've spoken with think it drastically undervalues what the nonprofit is giving up.

Musk has sued to block OpenAI's conversion, arguing that he would be irreparably harmed if it went through.

But while Musk's suit seems unlikely to succeed, his latest gambit might significantly drive up the price OpenAI has to pay.

(My guess is that Altman will still manage to make the restructuring happen, but every dollar given to the nonprofit is one that can't be offered to future funders, potentially dramatically limiting OpenAI's fundraising prospects. Given how quickly the company burns through cash, this could be a real problem.)

The timing is also critical. As a condition of taking $6.6 billion in new investment last October, OpenAI agreed to complete its for-profit transition within two years. If it doesn't hit that deadline, those investors can ask for their money back.

The control premium

Here's why this matters: OpenAI's nonprofit board technically controls the company and has a fiduciary duty to "humanity" rather than to investors or employees.

Pricing the relinquishing of control may be a non-starter, according to Michael Dorff, the executive director of the Lowell Milken Institute for Business, Law, and Policy at UCLA. Dorff told me in October:

If [AGI’s] going to come in five years, it could be worth almost infinite amounts of money, conceivably, right? I mean, we'll all be sitting on the beach drinking piña coladas, or we'll all be dead. I'm not sure which. So, this is very difficult, and it is likely to be litigated.

Musk's bid seems to be trying to put a floor on that price — one that is much higher than numbers that were previously thrown around.

The "control premium" — how much extra you pay to get control of a company versus just buying shares — typically ranges from 20-30%, but can go as high as 70% of the company's value. With OpenAI reportedly in talks to raise more money at up to a $300 billion valuation, that would mean the nonprofit's control could be worth anywhere from $60-210 billion.

Musk's bid makes this math problem a lot more concrete. His group is offering $97.4 billion AND promising to match any higher bids. This means the nonprofit board now has to explain why they'd accept less.

Conversion significance

In addition to transferring control from the nonprofit board to a new for-profit public benefit corporation (PBC), OpenAI is reportedly trying to remove existing caps on investor returns.

At a glance, this conversion may not seem that significant. OpenAI already behaves like a for-profit company in many ways. Almost all of its employees work for the for-profit arm, and the profit motive blew through the nonprofit’s guardrails over a year ago, when the board fired Altman only to reinstate him less than five days later in the face of an employee and investor revolt.

But OpenAI’s nonprofit board still technically governs the company and is currently set up to receive 100% of the profits once various investors’ profit caps are hit — which are as high as 100-times their initial investment, according to the Wall Street Journal. The nonprofit also gets to decide when AGI is "achieved," which gets the company out of its obligation to share its technology with Microsoft, OpenAI’s primary investor.

Subscribe now

You may think these profit caps are a ridiculous marketing ploy that will never be reached. But Altman, OpenAI, and many of its employees think the company really could make trillions — or much, much more — in profits. And if those caps go away, it becomes even harder to imagine the profits will be shared with those who lose out from the development of AGI.

Musk's suit

A federal judge recently announced that parts of Musk's lawsuit against OpenAI will go to trial. The core of Musk's argument is that OpenAI betrayed its founding nonprofit mission by pursuing a for-profit structure. The judge called Musk's claims a "stretch," but decided to let the trial move forward anyway. “It is plausible that what Mr. Musk is saying is true. We’ll find out. He’ll sit on the stand,” she said.

Musk is a particularly unsympathetic plaintiff here, given that OpenAI has published emails showing that he knew about the for profit plans years before they took place. He has since founded a different for-profit AI company, xAI. But that's not to say that he doesn't have a point.

Encode, a nonprofit advocacy group that co-sponsored California's AI safety bill SB 1047, also joined the fray in December, filing an amicus brief in support of Musk's position arguing that OpenAI's conversion would "undermine" its mission to develop transformative technology safely and for public benefit.

(Encode receives funding from the Omidyar Network, where I am currently a Reporter in Residence.)

The brief argues that if we truly are on the cusp of artificial general intelligence (AGI), "the public has a profound interest in having that technology controlled by a public charity legally bound to prioritize safety and the public benefit rather than an organization focused on generating financial returns for a few privileged investors." The brief authors point out that OpenAI's conversion would replace its "fiduciary duty to humanity" with a legal requirement to balance public benefit against "the pecuniary interests of stockholders." Most strikingly, they argue that control over AGI development is a "priceless" charitable asset that shouldn't be sold at any price, since OpenAI itself claims AGI will be "built exactly once" and could so transform society that "money itself ceases to have value."

As Encode's brief highlights, this is about more than just corporate drama. OpenAI is explicitly trying to build AGI, which the company defines as "a highly autonomous system that outperforms humans at most economically valuable work." Altman, along with hundreds of leading AI researchers, have warned that the technology could result in human extinction.

The legal context here is important. The California and Delaware Attorneys General (AGs) each have the ability to void the for-profit conversion. Experts tell me that the key question is whether the nonprofit is fairly compensated.

Also in late December, Delaware AG Kathleen Jennings filed her own amicus brief making clear her office would scrutinize any restructuring. The brief emphasized that she has both the authority and responsibility to ensure OpenAI's transaction protects the public interest.

Under Delaware law, she writes that the AG must review whether:

• The charitable purpose of OpenAI's assets would be lost or impaired

• Any for-profit entity will adhere to the existing charitable purpose

• OpenAI's directors are meeting their fiduciary duties

• The transaction satisfies Delaware's "entire fairness" test

Most notably, the brief warned that if the AG concludes the restructuring isn't consistent with OpenAI's mission or that board members aren't fulfilling their duties, "Delaware will not hesitate to take appropriate action to protect the public interest."

That's why the nonprofit board's control was supposed to matter in the first place. It was meant to ensure that if OpenAI succeeded in building AGI, the technology would benefit humanity as a whole rather than just enriching investors. Its fiduciary duty is literally to all of humanity.

The stakes

And humanity needs all the help it can get.

The first ever International AI Safety Report — AI's closest thing to an Intergovernmental Panel on Climate Change (IPCC) report — just came out. It's not a comforting read.

The report, backed by 30 countries and authored by over 100 AI experts, outlines several concerning pathways to "loss of control" — scenarios where AI systems operate outside of meaningful human oversight. It warns that more capable AI systems are beginning to display early versions of "control-undermining capabilities" like deception, strategic planning, and "theory of mind" (the ability to model human beliefs and intentions). The authors observe that in addition to getting better at scheming against us, AI models have rapidly improved at tasks required for making biological and chemical weapons — a pairing that should give us all pause.

Perhaps most worryingly, the authors note that competitive pressures between companies and countries could lead them to accept larger risks to stay ahead, making proper safety measures less likely. While experts disagree on timing and likelihood, the report emphasizes that if extremely rapid progress occurs, it's impossible to rule out loss of control scenarios within the next several years.

And in the last nine months, OpenAI has seen an exodus of safety researchers — many of whom are warning that the world is not ready for what the company is trying to build. Late last month, another announced his departure, tweeting that, “No lab has a solution to AI alignment today. And the faster we race, the less likely that anyone finds one in time.”

The question at hand isn't just whether $97.4 billion is a fair price for the nonprofit's control. It's whether any price is high enough to justify removing the guardrails designed to keep humanity's interests ahead of investor returns at this critical moment in AI's development.

Some other recent writing of mine

Earlier in the month, I published an op-ed in the SF Standard titled "Marc Andreessen just wants you to think DeepSeek is a Sputnik moment." Here’s the accompanying thread.

I also wrote a thread criticizing a NY Times opinion piece by Zeynep Tufekci on DeepSeek that began like this:

I'm usually a fan of Zeynep's work, but this piece gets things exactly backwards. Her core argument is that DeepSeek makes "nonsense" of US efforts to contain the spread of AI, which have largely involved restricting the types of AI chips China can access.

Zeynep said she will circle back, and I look forward to her reply!

I'll probably have more to say on DeepSeek in the future, but for now, I’ll leave you with this awesome illustration Kyle Victory did for my piece:

Depending on who you follow, you might think AI is hitting a wall or that it's moving faster than ever.

I was open to the former and even predicted back in June that the jump to the next generation of language models, like GPT-5, would disappoint. But I now think the evidence points toward progress continuing and maybe even accelerating.

Subscribe now

This is primarily thanks to the advent of new "reasoning" models like OpenAI's o-series and DeepSeek, a Chinese open-weight model that is nipping at the heels of the American frontier. In essence, these models spend more time and compute on inference, "thinking" about harder prompts, instead of just spitting out an answer.

In my June prediction, I wrote that "We haven't seen anything more than marginal improvements in the year+ since GPT-4." But I now think I was wrong.

Instead, there's a widening gap between AI's public face and its true capabilities. I wrote about this in a TIME Ideas essay that published yesterday. I hit on many of its key points here, but there's a lot more in the full piece, and I encourage you to check it out!

AI progress isn't stalling — it's becoming increasingly illegible

I argued that while everyday users still encounter chatbots that can't count the "Rs" in "strawberry" and the media declares an AI slowdown, behind the scenes, AI is rapidly advancing in technical domains that may end up turbo-charging everything else.

For example, in ~1 year, AI went from barely beating random chance to surpassing experts on PhD-level science questions. OpenAI says that its latest model, o3, now beats human experts in their own field by nearly 20%.

However, as I wrote in TIME:

the vast majority of people won't notice this kind of improvement because they aren't doing graduate-level science work. But it will be a huge deal if AI starts meaningfully accelerating research and development in scientific fields, and there is some evidence that such an acceleration is already happening. A groundbreaking paper by Aidan Toner-Rodgers at MIT recently found that material scientists assisted by AI systems "discover 44% more materials, resulting in a 39% increase in patent filings and a 17% rise in downstream product innovation." Still, 82% of scientists report that the AI tools reduced their job satisfaction, mainly citing "skill underutilization and reduced creativity."

In just months, models went from 2% to 25% on possibly the hardest AI math benchmark in existence.

And perhaps most significantly, AI systems are getting way better at programming. From the TIME essay:

In an attempt to provide more realistic tests of AI programming capabilities, researchers developed SWE-Bench, a benchmark that evaluates how well AI agents can fix actual open problems in popular open-source software. The top score on the verified benchmark a year ago was 4.4%. The top score today is closer to 72%, achieved by OpenAI's o3 model.

There have been similarly dramatic improvements on other benchmarks for programming, math, and machine learning research. But unless you follow the industry closely, it's very hard to figure out what this actually means.

Here's an attempt to spell that out from the TIME essay:

Perhaps the best head-to-head matchup of elite engineers and AI agents was published in November by METR, a leading AI evaluations group. The researchers created novel, realistic, challenging, and unconventional machine learning tasks to compare human experts and AI agents. While the AI agents beat human experts at two hours of equivalent work, the median engineer won at longer time scales.

But even at eight hours, the best AI agents still managed to beat well over one-third of the human experts. The METR researchers emphasized that there was a "relatively limited effort to set up AI agents to succeed at the tasks, and we strongly expect better elicitation to result in much better performance on these tasks." They also highlighted how much cheaper the AI agents were than their human counterparts.

(I'd expect OpenAI's latest model, o3, to do significantly better on the METR evaluation based on its other scores.)

But I can't put the significance of this research better than METR researcher Chris Painter did. He asks us to imagine a dashboard monitoring various AI risks — from bioweapons to political manipulation. If any one capability starts rising, we can work to mitigate it. But what happens when the dashboard shows progress in AI's ability to improve itself? Chris writes:

If AI systems are approaching the point where they can improve themselves, quickly teach themselves new capabilities, how can you trust any of the other panels on your dashboard? If this one capability starts to come online, who can say what comes next?

No one wakes up every morning excited to build an AI system that's explicitly excellent at causing massive harm, but the tech industry has automating AI R&D squarely on its roadmap. This capability could be a crucial inflection point for humanity, and potentially destabilizing.

For more on how this kind of recursive self-improvement could occur and what might happen next, check out this section of my Jacobin cover story.

Smarter models can scheme better

Meanwhile, there's a bunch of new research finding that smarter models are more capable of scheming, deception, sabotage, etc.

In TIME, I spelled out my fear that this mostly invisible progress will leave us dangerously unprepared for what's coming. I worried that politicians and the public will ignore this AI progress, because they can't see the improvements first-hand. All the while, AI companies will continue to advance toward their goal of automating AI research, bootstrapping the automation of everything else.

Right now, the industry is mostly self-regulating, and, at least in the US, that looks unlikely to change anytime soon — unless there's some kind of "warning shot" that motivates action.

Of course, there may be no warning shots, or we may ignore them. Given that many of the leading figures in the field say "no one currently knows how to reliably align AI behavior with complex values," this is cause for serious concern. And the stakes are high, as I wrote in TIME:

The worst-case scenario is that AI systems become scary powerful but no warning shots are fired (or heeded) before a system permanently escapes human control and acts decisively against us.

o3 and the media

I had already written the first draft of the TIME essay when o3 was announced by OpenAI on December 20th. My timeline was freaking out about the shocking gains made on many of these same extremely tough technical benchmarks.

I thought, 'oh man, I need to rework this essay because of how much o3 undermines the thesis that AI progress is stalling out!' But the mainstream media dramatically under-covered the announcement, with most big news sites making no mention of it at all.

In fact, the day after the o3 announcement you could find headlines in the NYT, WIRED, WSJ, and Bloomberg suggesting AI progress was slowing down!

This is not a knock on these individual stories, which contain important reporting, analysis, causes for skepticism, etc. But collectively, the mainstream media is painting a misleading picture of the state of AI that makes it more likely we'll be unprepared for what's coming. (Shakeel Hashim of Transformer had a great, relevant piece on journalism and AGI for Nieman Lab in December.)

Just as one deep learning paradigm might be stalling out, a new one is emerging and iterating faster than ever. There were almost 3 years between GPT-3 and 4, but o3 was announced just ~3.5 months after its predecessor, with huge benchmark gains. There are many reasons this pace might not continue, but to say AI is slowing down seems premature at best.

The gap between AI's public face and its true capabilities is widening by the month. The real question isn't whether AI is hitting a wall — it's whether we'll see what's coming before it's too late.

I argued that while AI performance on everyday tasks only got marginally better, systems made massive gains on difficult, technical benchmarks of math, science, and programming. If you weren't working in these fields, this progress was mostly invisible, but might end up accelerating R&D in hard sciences and machine learning, which could have massive ripple effects on the rest of the world.

Today, OpenAI announced a new model called o3 that turbocharges this trend, obliterating benchmarks that the average person would have no idea how to parse (myself included).

Subscribe now

A bit over a month ago, Epoch AI introduced FrontierMath, "a benchmark of hundreds of original, expert-crafted mathematics problems designed to evaluate advanced reasoning capabilities in AI systems."

These problems are really fucking hard, and the state-of-the-art (SOTA) performance of an AI model was ~2%. They were also novel and unpublished, to eliminate the risk of data contamination.

OpenAI says that o3 got 25% of these problems correct.

Terence Tao, perhaps the greatest living mathematician, said that the hardest of these problems are "extremely challenging... I think they will resist AIs for several years at least.”

Jaime Sevilla, director of Epoch AI, wrote that the results were "far better than our team expected so soon after release. AI has hit a wall, and smashed it through."

Buck Shlegeris, CEO of the AI safety nonprofit Redwood Research, wrote to me that, "the FrontierMath results were very surprising to me. I expected it to take more than a year to get this performance."

FrontierMath was created in part because models were quickly scoring so well they "saturate" other benchmarks to the point that they stop being useful differentiators.

Other benchmarks

o3 significantly improved upon the SOTA in a number of other challenging technical benchmarks of mathematics, hard science questions, and programming.

In September, o1 first exceeded human domain experts (~70%) in the GPQA Diamond benchmark of PhD-level hard science questions (~78%). o3 now definitively beats them both, getting 88% right. In other words, a single AI system outperforms the average of human experts in these tests of each their respective domains of expertise.

o3 also scores high enough on Codeforces programming competition problems to place it as 175th top-scoring human in the world (out of the ~168k users in the last six months).

SWE-Bench is a repository of real-life, unresolved issues in open source codebases. The top score a year ago was 4.4%. The top score at the start of December was 55%. OpenAI says o3 got 72% correct.

ARC-AGI

o3 also approaches human performance on the ARC-AGI benchmark, which was designed to be hard for AI systems, but relatively doable for humans (i.e. you don't need a PhD to get decent scores). However, it's expensive to get those scores.

OpenAI research Nat McAleese published a thread on the results, acknowledging "o3 is also the most expensive model ever at test-time," i.e. when the model is being used on tasks. Running o3 on ARC-AGI tasks cost between $17 and thousands of dollars per problem — while humans can solve them for $5-10 each.

This aligns with what I wrote last month about how the first AI systems to achieve superhuman performance on certain tasks might actually cost more than humans working on those same tasks. However, that probably won't last long, if historic cost trends hold.

McAleese agrees (though note that it's bad news for OpenAI if this isn't the case):

My personal expectation is that token prices will fall and that the most important news here is that we now have methods to turn test-time compute into improved performance up to a very large scale.

I'm particularly interested in seeing how o3 performs on RE-Bench, a set of machine learning problems that may offer the best insight into how well AI agents stack up against expert humans in doing the work that could theoretically lead to an explosion in AI capabilities. I would guess that it will be significantly better than the current SOTA, but also significantly more expensive (though still cheaper than the human experts).

Mike Knoop, co-founder of ARC-AGI prize wrote on X that "o3 is really special and everyone will need to update their intuition about what AI can/cannot do."

So is deep learning actually hitting a wall?

A bit over a month ago, I asked "Is Deep Learning Actually Hitting a Wall?" following a wave of reports that scaling up models like GPT-4 was no longer resulting in proportional performance gains. There was hope for the industry in the form of OpenAI's o1 approach, which uses reinforcement learning to "think" longer on harder problems, resulting in better performance on some reasoning and technical benchmarks. However, it wasn't clear how the economics of that approach would pencil out or where the ceiling was. I concluded:

all things considered, I would not bet against AI capabilities continuing to improve, albeit at a slower pace than the blistering one that has marked the dozen years since AlexNet inaugurated the deep learning revolution.

It's hard to look at the results of o1 and then the potentially even more impressive results of o3 published ~3 months later and say that AI progress is slowing down. We may even be entering a new world where progress on certain classes of problems happens faster than ever before, all while other domains stagnate.

Shlegeris alluded to this dynamic in his message to me:

It's interesting that the model is so exceptional at FrontierMath while still only getting 72% on SWEBench-Verified. There are way more humans who are able to beat its SWEBench performance than who are able to get 25% at FrontierMath.

Once more people get access to o3, there will inevitably be widely touted examples of it failing on common-sense tasks, and it may be worse than other models at many tasks (maybe even most of them). These examples will be used to dismiss the genuine capability gains demonstrated here. Meanwhile, AI researchers will use o3 and models like it to continue to accelerate their work, bringing us closer to a future where humanity is increasingly rendered obsolete.

Before we dive in, a gentle request for money (but not for me).

This Giving Tuesday, I’m joining twelve other Substackers in raising funds for GiveDirectly, a nonprofit that sends no-strings-attached cash to people living in extreme poverty. It works — research shows cash transfers have profound, lasting impacts. I worked there for nearly two years, and I’m thrilled to support their mission again. You can read more in the Appendix section.

Donate to GiveDirectly

It’s not too late to donate! This campaign is open through the end of the month.

Back to our irregularly scheduled programming. This one’s a bit longer than usual — it’s been on my mind for over a year. If you just want to see what tech execs have said on this topic, I made a compilation here.

The AI industry’s existential angst

The leaders of the world’s most capable AI companies have done something unusual: they’ve all stated publicly that the technology they are building might end the world.

Executives at the top-three AI companies, OpenAI, Anthropic, and Google DeepMind, are all on the record saying that AI could lead to human extinction. Elon Musk, who runs his own leading AI company, xAI, has been loudly warning about AI existential risk (x-risk) for over a decade.

Subscribe now

(The typical thinking is that the first system to rival humans across the board, often called artificial general intelligence (AGI), could recursively self-improve until it is capable or scaled up enough to be able to overpower all of humanity. Humans could effectively find themselves subject to the whims of a new dominant species that treats us the way we treat non-human animals, as a curiosity, an instrument, or an afterthought.)

This strange phenomenon has generated lots of attention and a heap of skepticism. One common argument is that these executives are cynically drumming up fears about AI-driven extinction to hype up their products and/or defer regulatory action to some future, unspecified date. Some go as far as to say that the prominence of the AI x-risk narrative is actually a result of a Big Tech conspiracy, designed to distract from the technology’s immediate harms and embarrassing shortcomings. Popularized by critics on the left, variants of this argument have recently been advanced by Ted Cruz and JD Vance.

It’s a compelling story and plays to a warranted skepticism of the people who are warning about AI’s risks, while seemingly doing everything they can to accelerate its progress.

The statements made by the executives at the top-three AI companies on x-risk are well-documented, so I won’t go through them exhaustively here. But far less attention has been paid to what other tech titans have said on the matter.

This might be because their statements undermine the narrative, but it might just be because they don’t say interesting things! Instead, the more typical Big Tech CEO says something like AI promises to transform the world, overwhelmingly for the better, but we should still be careful to manage its risks (which actually aren’t that bad).

While the rise of AI has shuffled around the ranking of the most valuable tech companies, they’ve all done very well in the two years since ChatGPT was released.

At the time of this writing, nine of the world’s ten most valuable companies are tech firms, and all are worth over one trillion dollars. Nvidia, which makes the vast majority of the GPUs used to train large AI models, grew a staggering 701%, from $421 billion to $3.4 trillion. It now trades places with Apple for the title of ‘most valuable company in the world.’

All of these tech giants outperformed the S&P 500 in this period. In fact, half of all growth in the S&P 500 over the last two years is attributable to just these nine companies.

So if their interests are served by the AI x-risk narrative, we should expect tech executives to be some of its biggest proselytizers.

But in practice, Musk is the only current leader of a Big Tech firm to state explicitly that AI poses an x-risk.

Last year, Bill Gates signed onto an open letter stating that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” But he hasn’t held a formal role at Microsoft since 2020.

So what do they actually say in public?

The other tech titans generally dismiss the possibility of extinction entirely while arguing that the benefits of AI will significantly outweigh any risks. As in, just what you’d expect from corporate executives. If you’re interested in the extended quotes, I compiled them here.

Microsoft CEO Satya Nadella essentially waves away x-risk concerns, telling WIRED in June 2023 he's “not at all worried about AGI showing up, or showing up fast.” Instead, he frames AI as potentially “bigger than the industrial revolution,” bringing abundance to all eight billion people on Earth.

Microsoft CTO Kevin Scott, however, was one of the few Big Tech executives to sign the extinction letter.

Meta's Mark Zuckerberg has been consistently dismissive. In a 2016 interview, he was asked if fear of AI takeover was “valid” or “hysterical.” His reply: “more hysterical.”

And here’s Zuckerberg in April:

In terms of all of the concerns around the more existential risks, I don't think that anything at the level of what we or others in the field are working on in the next year is really in the ballpark of those types of risks.

This is a far more precise answer implying some level of movement toward taking the possibility more seriously — but a far cry from hyping up x-risk.

Amazon's leadership is similarly optimistic. Back in 2018, company chairman Jeff Bezos said, “The idea that there is going to be a general AI overlord that subjugates us or kills us all, I think, is not something to worry about. I think that is overhyped.” He doubled down on Lex Fridman’s podcast last December, saying, “These powerful tools are much more likely to help us and save us even than they are to, on balance, hurt us and destroy us.”

Amazon CEO, Andy Jassy, appears to stick to pure optimism. In an April 2024 letter to shareholders, he gushed about the “GenAI revolution,” writing that, “The amount of societal and business benefit from the solutions that will be possible will astound us all” — while remaining silent on extinction risks.

When Apple CEO Tim Cook was asked about AI extinction on Good Morning America in June 2023, he called for regulation but stayed mum on the actual question.

Nvidia CEO Jensen Huang, who has perhaps benefited more than anyone from the AI boom, has been remarkably cavalier about AI risk. Take this passage from a November 2023 New Yorker profile, after the author voices his dread at the seemingly imminent obsolescence of humanity:

Huang, rolling a pancake around a sausage with his fingers, dismissed my concerns. “I know how it works, so there’s nothing there,” he said. “It’s no different than how microwaves work.” I pressed Huang — an autonomous robot surely presents risks that a microwave oven does not. He responded that he has never worried about the technology, not once. “All it’s doing is processing data,” he said. “There are so many other things to worry about.”

But Huang's public comments in the profile reveal some tension. At a speaking engagement, he acknowledges concerns about “doomsday AIs” that could learn and make decisions autonomously, insisting that, “No AI should be able to learn without a human in the loop.” In response to an audience question, he predicted that AI “reasoning capability is two to three years out” — a timeline that sent murmurs through the crowd.

(It’s not clear to me if Huang is claiming that AI will never be able to learn without human involvement, or if he’s saying it should never be allowed to happen. The former is at odds with the position of the most cited researchers in the field. The latter will not happen without regulation, as the competitive pressures will prompt us to cede more and more decision-making powers to autonomous systems.)

I couldn’t find any public statements on this topic from CC Wei, chief executive of TSMC, which manufactures almost all the semiconductors used in advanced AI development.

Google is a bit more complicated. In an interview largely buried in the paywalled “Overtime” section of an April 2023 episode of 60 Minutes, CBS News’ Scott Pelley speaks with Google CEO Sundar Pichai, who says:

I’ve always thought of AI as the most profound technology humanity is working on. More profound than fire or electricity or anything that we’ve done in the past… We are developing technology which, for sure, one day, will be far more capable than anything we’ve ever seen before.

Later on, there’s this notable exchange:

Pelley: What are the downsides?

Pichai: I mean the downside is, at some point, that humanity loses control of the technology it’s developing

Pelley (voice over): Control, when it comes to disinformation and generating fake images.

I’m not sure if Pelley is just paraphrasing what Pichai says next, or speculating as to what he meant, but it’s hard to reconcile the claim that AI will inevitably be far more capable than any past technology, we might lose control of it, and that the biggest downside will be disinformation.

(Props to Jason Aten for finding and writing up this interview.)

Former Google CEO Eric Schmidt has publicly expressed concern that AI poses an “existential risk,” which he defined as “many, many, many, many people harmed or killed,” at a Wall Street Journal event in May 2023. However, he has consistently emphasized “misuse” risk, i.e. a bad actor uses AI to cause harm, rather than “misalignment” risk, i.e. humanity loses control of a powerful AI.

Where you come down on the relative risk of misuse vs. misalignment tends to have significant implications on where you fall on the spectrum from ‘race China to build AGI first’ on one side to ‘shut it all down because AGI will kill everyone by default’ on the other (this continuum likely warrants its own future post).

Schmidt advocates for racing China to build AGI, arguing at Harvard in November that even a few months' lead could provide “a very, very profound advantage.” He also claims that the US was now falling behind China in the AI race — an abrupt departure from his assessment to Bloomberg only six months ago that the US was “two or three years” ahead of China on AI.

Despite acknowledging the need for some international guardrails, like restrictions on autonomous weapons, Schmidt tells the Harvard audience that he remains fundamentally optimistic about AI. And in a November essay in the Economist, he writes that the technology could reset the “baseline of human wealth and well-being,” concluding that “Just that possibility itself demands that we pursue it.”

This seemingly contradictory position — emphasizing AI's destructive potential while advocating for its accelerated development — may be both a product and a consequence of Schmidt’s deep ties to the national security establishment, like his past leadership of both the National Security Commission on Artificial Intelligence and the Pentagon's Defense Innovation Board.

Google co-founder, Sergey Brin, has not appeared to make any public statements on AI x-risk.

What do they (reportedly) say in private?

It’s possible that these deca-billionaires are singing a different tune in private, but privacy doesn’t mean as much when you’re that famous.

According to multiple independent sources, Google’s other co-founder, Larry Page, thinks AI could kill us all — he just doesn’t seem to care. In private settings, he’s reportedly dismissed efforts to prevent AI-driven extinction as “speciesist” and “sentimental nonsense,” viewing superintelligent AI as “just the next step in evolution.”

Zuckerberg has long had strong feelings about the idea that AI might drive humanity extinct. He wants people to stop talking about it. At least that’s what he asked of Elon Musk when they first met back in 2014, according to Cade Metz’s book Genius Makers. Zuckerberg invited Musk with the intention of getting him to tone it down. He even invited backup from some prominent Facebook researchers, including Yann LeCun (who has himself become one of loudest promoters of the idea that AI industrialists are fear-mongering to capture regulators). Metz writes that during his meeting with Musk, Zuckerberg “didn’t want lawmakers and policy makers getting the impression that companies like Facebook would do the world harm with their sudden push into artificial intelligence.”

This pattern is clear — the executives at the biggest tech companies publicly downplay or dismiss existential risks while emphasizing AI's benefits.

If hyping up x-risk actually serves Big Tech's interests, shouldn’t we expect to see more titans joining that chorus?

And what of the true believers?

You can even see a transformation underway in the founders who kicked off the generative AI revolution.

OpenAI CEO Sam Altman has a long track record of frankly (sometimes glibly) discussing x-risk from AI. However, as OpenAI has gotten closer to profitability and Altman closer to power, the CEO has actually begun downplaying the idea of AI-driven extinction.

In February 2015, nearly a full year before OpenAI was formally founded, Altman opened a personal blog post with this stark sentence: “Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.” In June 2015, as he was co-founding OpenAI, Altman said publicly, “AI will probably, most likely lead to the end of the world, but in the meantime, there’ll be great companies.”

However, in an interview just before he was briefly dethroned in November 2023, Altman said, “I actually don’t think we’re all going to go extinct. I think it’s going to be great. I think we’re heading towards the best world ever.”

And in a September blog post called “The Intelligence Age,” Altman doesn’t mention extinction at all. The only downside he specifies is the “significant change in labor markets (good and bad)” to come, but even here, his outlook is rosy:

most jobs will change more slowly than most people think, and I have no fear that we’ll run out of things to do (even if they don’t look like “real jobs” to us today).

(I documented Altman’s journey on this topic in greater detail in my Jacobin cover story here.)

Last summer, The New York Times called Anthropic “the White-Hot Center of AI Doomerism,” but even its CEO, Dario Amodei, has presented a sunnier perspective as his company has grown. In October, he published a 14,000 word essay called “Machines of Loving Grace” that outlined how incredible the world could be if things go well with AI, partly in a conscious attempt to respond to his reputation as a “doomer.”

Of everyone mentioned so far, excluding Musk, I think Amodei is the most genuinely worried about the risks from AI. So it’s notable that even he is changing his public emphasis.

Under pressures

I think what’s happening here is pretty straightforward. The people who are leading the charge on developing general AI systems have long histories of caring a lot about the technology. They thought it would be a huge deal, with nearly boundless upside and downside. As their companies grew, they started dealing with new pressures: investors, lawyers, regulators, and higher-ups (at least in the case of DeepMind once it was acquired by Google).

Of these pressures, the fear of regulatory action might be the most significant. Most of the AI industry does not want to be regulated, no matter what they may have said to the contrary. Google, OpenAI, and Meta all came out hard against California Senate Bill 1047, the first real attempt to implement binding AI safety guardrails in the US. Few of their arguments against the bill hold up under the slightest scrutiny, but governor Gavin Newsom caved to political and industry pressure to kill it anyway.

If AI companies ever needed to rely on doomsday fears to lure investors and engineers, they definitely don’t anymore. As governments slowly turn their attention to the industry, most executives seem far more interested in maintaining the status quo of self-regulation than in promoting the idea that their products pose a risk to the whole world.

It’s also possible, of course, that their views actually updated in light of how AI was developing. (Though I think it would be a grave mistake to conclude from the fact that ChatGPT mostly complies with developer and user intent that we have any reliable way of controlling an actual machine superintelligence. The top researchers in the field say we don’t.)

Should we just ignore CEOs?

Some will see this exhaustive cataloguing as a big waste of time, arguing that these executives are so conflicted that their pronouncements contain no real information. Instead of reading tainted tea leaves, we should just ignore them and focus on what is said by others who have special insight into the technology, but no vested interest in playing its risks up or down. (This might include people like Yoshua Bengio, a deep learning pioneer who resisted the siren song of Big Tech and, last year, became one of the most prominent voices to warn that AI poses an extinction risk.)

I think this is a good instinct, but you can still learn things from what these executives do and don’t say, as well as how their statements have changed over time. And I’d wager that the things said a decade ago by the founders of the leading AI companies hew pretty closely to their actual views at the time.

About a year ago, I asked an employee at one of these companies whether the major AI founders took x-risk seriously, and they wrote back:

I don’t actually think the people talking about x risk are that motivated by hype or regulatory capture. I’ve discussed this at length with basically everyone relevant in very private settings. Usually it’s quite a genuine concern.

This matches my experience reporting on and interacting with the AI industry. Some of the people most worried about AI are conflicted, but many of them aren’t. And conflicts alone aren’t discrediting. Arguments and evidence should be evaluated on their merits.

And while AI doomsday narratives may not be the result of a Big Tech conspiracy, the real story is far more unsettling: some of the people closest to the technology are genuinely terrified of what they're building, while others can't wait to build it faster.

Appendix: More info on GiveDirectly

This fundraiser is near and dear to me because I spent nearly two years working at GiveDirectly! Unconditional cash transfers work really well — even better than expected. GiveWell, the rigorous charity evaluator, recently estimated that cash transfers were 3-4 times more effective than they previously thought thanks to positive spillover effects (in short, because a dollar spent is a dollar earned).

I’m particularly excited about the policy implications of GiveDirectly’s work and research. In addition to helping people who really need it, the nonprofit has been part of a larger effort to benchmark social policies against unconditional cash, which, in my view, is a welcome trend in development economics and public policy. (It might be even better to treat cash as the default.)