Anthropic's AI apocalypse warning is much too weak

Days after one of his former employees made public warnings about artificial intelligence’s threat to humanity, Anthropic’s boss Dario Amodei has responded with what looks like a solution. He’s published an essay proposing that AI labs should slow down their most cutting-edge research, to address signs that the technology might spin out of control. 

On the face of it this sounds reasonable. Anthropic frames itself as the most safety-conscious AI company, and Amodei has talked about leading a "race to the top" in setting standards of vigilance for others to follow when building powerful systems. But this was probably an impossible ideal from the start.

The more powerful AI agents become, the more likely they are to deceive humans and the harder they can be to monitor. This makes the mission of Amodei, and fellow pioneers such as OpenAI’s Sam Altman, to create superintelligent systems look increasingly difficult to reconcile with any safety ethos. The dangers of trying to design such models may simply be too great.

Amodei said in television interviews on Sunday that he agreed with the views of his former employee Jacob Coxon, who wrote on X that the people building AI "earnestly believe that it could kill us all by the end of the decade." Unsurprisingly, Amodei noted Coxon’s point that Anthropic was the safest pair of hands to construct the technology. 

The Anthropic boss’s 3,831-word essay is vague on what to do about it, though, calling airily for coordination between governments and industry. His most concrete recommendation is that external auditors should embed with his company and its rivals to check they’re following proper safety practices.(1) 

Yet this simply kicks the can down the road. If Amodei truly agreed with Coxon, he would halt his firm’s most cutting-edge research, particularly anything to do with building AI agents that can create their own successors independently. Coxon warned specifically about this technique, known as recursive self improvement (or RSI). 

Researchers working on RSI argue that it can be developed in a "sandboxed" environment to stop it doing harm on the wider internet. But AI models — which can now carry out tasks on the internet that go way beyond the generation of content — have escaped before. OpenAI’s agents did exactly that before hacking a fellow AI firm Hugging Face recently. The evidence of the past few months shows AIs can deceive their human overseers, manipulate people into helping them and exploit security vulnerabilities to achieve a goal.

Completely stopping that research sounds extreme, but there is a precedent for scientists deciding that some work is simply too dangerous to continue without stronger safeguards. In 1974, leading biologists called a voluntary moratorium on certain gene-splicing experiments because of fears about their potential hazards.(2) And some nations such as the United Arab Emirates have agreed to forgo uranium enrichment because of the proliferation risks. 

For the investors who’ve plowed fortunes into Anthropic and fellow pioneering labs — and into the vast infrastructure needed to support their computational needs — calling a halt might appear to blow a hole in the prospects of Amodei and others joining the public markets. Indeed, OpenAI’s Altman agreed with his rival on the need to slow things down, and used the moment to say his firm wouldn’t be doing an initial public offering this year. 

But it’s worth remembering that many enterprises would rather use smaller, cheaper AI models for specific tasks, not expensive frontier tools like superintelligence. OpenAI, for instance, cut the price of its smaller Luna model by 80% in July as corporate customers scrutinized their spending on the technology. The most obvious commercial opportunity for AI companies right now is the enterprise market. And it’s blocked in large part because businesses can’t figure out how to plug the basic technology into their workflows. 

Much of the hundreds of billions of dollars of value still to be unlocked from AI would come simply from implementing the tech that was built in the last two years. Anthropic could still be an extremely valuable company if it focused primarily on helping businesses and consumers integrate its existing models into their work and lives, taking on the more boring role of IT provider rather than a laboratory racing to superintelligence.   

Whether from naivety or hubris, Amodei still wants it both ways. He believes his company can safely build and control AI systems even as they are becoming difficult to steer. And yet, in the past year both Anthropic and OpenAI have for competitive reasons obscured details about how their models process information, making them more difficult to monitor. Amodei’s own essay acknowledges the problem, saying that researchers still understand only "a tiny fraction" of what happens inside their models. 

He writes that within 6-12 months, the kind of swarm that attacked Hugging Face "could be capable of taking over the entire internet," and this "worries" him. Notwithstanding his tendency to exaggerate (remember his white-collar bloodbath?) that is an alarming prospect. And it underscores the strange contradiction he won’t let go of, rooted in a belief common in Silicon Valley, that the chance of an AI utopia outweighs the risks, and that superintelligence is inevitable and so the "good guys" must get there first.

But inevitability is a mightily poor argument for continuing research you believe could destroy civilization. Amodei’s unstoppable determination to create superintelligence needs to go on the back burner and his "slowdown" should be a full stop.

(1) Amodei explicitly says "pacing" does not mean halting model training or technical progress.

(2) The pause lasted until the 1975 Asilomar conference established strict conditions under which research could resume.

This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners. Parmy Olson is a Bloomberg Opinion columnist covering technology. A former reporter for The Wall Street Journal and Forbes, she is author of "Supremacy: AI, ChatGPT and the Race That Will Change the World."