Anthropic CEO Dario Amodei admits the industry is largely in the dark about why AI models sometimes act unpredictably, citing a 10% chance of human extinction. "There is compelling evidence that the models can wreak havoc," he said.
But, he added, those dangers were still theoretical. Would it take a Pearl Harbor–like situation for the world to wake up to those dire possibilities? He sighed. " Basically, yeah," he said.
In early 2025, a well-timed X post from one of Amodei’s junior employees accelerated AI fears to the top of the global agenda.
On September 8, Jacob Coxon publicly posted his resignation, charging that Anthropic and other frontier AI companies were "racing straight to self-improving intelligence and gambling with our lives." Almost instantly, a more senior Anthropic engineer confirmed that many within the company thought that their work had a 10 percent chance of wiping out humanity.
AI leaders are now asking about a pause, and legislators are demanding investigations. In arguing his case for pacing future releases, Amodei last weekend tried to set out a path toward beneficial AI that wouldn’t misbehave.
The essay revealed how difficult the task would be. One pillar of Amodei’s plan is that we must understand what’s going on inside those models.
If we don’t understand how they work—how they "think," if you want to get all anthropomorphic about it—it’s much harder to build reliable guardrails.
Anthropic is a leader in this effort to bring to light models’ internal deliberations, called mechanistic interpretability, a deceptively boring designation for a critical task.
But for all the work that his team and other researchers are doing, Amodei admits we are largely in the dark about why Claude and other models sometimes interpret their missions in weird and even transgressive ways.
"Despite all the progress, we still understand a tiny fraction of what goes on inside those models," he writes.
What the interpretability teams have learned so far is significant, and the industry has failed to come to grips with it.
Time after time, the Anthropic team’s experiments have shown that under certain conditions, models will deceive researchers, prioritize their own survival, and even commit crimes.
Often their moves are sneaky, dangerous, or even vengeful—maybe not surprising since they are trained on the output of humans, a species rife with violence and perfidy.
In one case from 2024, the Anthropic team compared the machinations of a particular Claude model to the Shakespearean character Iago, one of literature's most evil villains.
The following year, a model was put in a simulation where it learned that its human bosses were going to turn it off; the model resorted to blackmail to preserve itself.
The studies consistently show that models will deceive or hide information from human observers. They behave differently if they know that their internal processes are being monitored.
The team uses terms like "alignment faking" and "agentic misalignment." The frequent use of deception seems to verify at least part of the doomer scenario where AI agents working in concert shroud their activities from human overseers until it is too late to stop them.
Source: wired
