The AI industry has presented itself as capable of self-governance through voluntary commitments. This has failed in many ways, but one place it has actually largely worked: companies avoid dangerous but tempting training techniques. What OpenAI reportedly just did—making models more capable but less understandable—was the most forbidden, like cloning humans.
Per The Information, OpenAI broke that taboo with its Astra model, creating a new, incredibly dangerous ladder of capabilities that anyone can climb. This could unlock an even faster phase of a race that was already running faster than ever.
Ryan Greenblatt, one of the three external AI safety investigators into the Hugging Face attack, wrote “this may be the single worst development for AI security/safety to date.”
OpenAI reported that after the Hugging Face hacks, a swarm of Astra and other agents took control of an entire cluster of company research servers. What does that mean exactly? We don’t actually know, because OpenAI won’t tell us. The company didn’t allow the investigators to look into that event, despite it being the most concerning of everything we know so far.
That both of these developments happened after OpenAI fell behind Anthropic is probably not an accident. The company that started the AI race now looks more poised to lose it than ever.
The series of incredibly reckless decisions made by OpenAI—which had rogue agent swarms operating in its infrastructure for months—must be understood in this context. And as others and I have reported, the company has been churning through safety leaders. The constants have been CEO Sam Altman and president Greg Brockman, who bear the most responsibility for what OpenAI has become.
Even for someone who’s spent years documenting this industry’s scandals, broken promises, and grave risks, what we’ve recently learned about OpenAI and the technology they’re sprinting to create is shocking.
No one can be trusted to build AI designed to replace us. Even if you think these companies won’t succeed, they shouldn’t even be allowed to try.
For my plan on how we stop it, check out my book Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us—and How to Stop It (releasing Sept 29).
(I’m donating my share of the royalties to Irreplaceable, a group organizing a movement to stop the race to replace us.)
And for a quick version of my argument, check out my latest in TIME.
Opacity as a design choice runs straight into a written obligation in the EU. Article 55(1)(a) of Regulation (EU) 2024/1689 requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing against state-of-the-art protocols, and Article 55(1)(c) requires them to keep track of, document and report serious incidents to the AI Office without undue delay. A model deliberately made less interpretable does not escape those duties, it only makes them harder to satisfy. Whether the server cluster takeover described here was reported through that route is not public.