Anthropic is urging a unilateral slowdown on the development of new AI models until rigorous safeguards can be put in place—with a few caveats.
In a blog post published Thursday, the company cited its own internal data as evidence that modern AI systems are nearing the point of “recursive self-improvement”—i.e., being able to refine their capabilities without a human in the loop. “AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond,” the post, which was written by company cofounder Jack Clark and Anthropic Institute lead Marina Favaro, reads. “But full recursive self-improvement also might increase the risks of humans losing control over AI systems.”
The post goes on to suggest that the best way to steer clear of such risks could be a worldwide pause akin to Cold War-era disarmament and non-proliferation pacts, in which countries like the U.S. and Russia agreed to limit their nuclear weapons programs to avoid what came to be known as Mutually Assured Destruction (MAD). “If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” the authors wrote.
If this all sounds familiar, you’re not hallucinating. Back in early 2023, nonprofit organization the Future of Life Institute published an open letter calling on all frontier AI labs to enact a six-month pause on the development of powerful new models. It was signed by Elon Musk, pioneering AI researcher Yoshua Bengio, Apple cofounder Steve Wozniak, and tens of thousands of others. While that letter and its luminous roster of signatories did much to kickstart the public conversation around the risks of an AI industry driven by blind capitalist logic, no firm policies were ever enacted to actually make that six-month pause a reality. Just a few months after it was published, Musk publicly launched his own AI startup to compete with OpenAI, Anthropic, Google, and the other big labs.
A crossroads
It’s worth noting the extremely hedged language that Anthropic uses in its blog post: If it were possible…we think…likely…. It reads as more of a suggestion than a fully committed call to action. That could have something to do with the crossroads at which the company now stands.
Since its inception, Anthropic has carved out a position for itself as a kind of moral conscience for the industry, the one big player that’s devoted to building AI cautiously, and if necessary, slowly, to prevent the kind of rogue AI scenario that sci-fi authors, philosophers, and many leaders within the big AI lab themselves have long feared. That safety-oriented approach has clearly paid off. While its Claude chatbots still trail behind OpenAI’s ChatGPT in app downloads, its tools have become massively popular among web developers and enterprise customers, both of which are key demographics for the industry. That’s one of the big factors behind its recent eclipsing of OpenAI as the world’s most valuable startup.
But it also has a highly anticipated IPO coming up, after which point it will be legally required—like all publicly traded companies—to do what’s best for its shareholders. And at least at this point in the AI boom, stepping on the brakes isn’t widely viewed as a winning long-term financial strategy.
Critics of the slowdown argument usually say that given the huge financial and geopolitical stakes of the AI race, it’s highly unlikely that the U.S., China, and other countries would agree to pause development, and even if they did, it’d be impossible to enforce such a pause. Even worse, if U.S. developers all agreed to pause, it could leave the door open for less ethically minded developers elsewhere to take the lead, to everyone’s detriment. As Clark and Favaro put it in their post, “if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe.”
In order to be effective, Clark and Favaro argue, a slowdown would require not just unilateral agreement among AI developers around the world, but also some kind of global mechanism that could verify that each lab has in fact paused development, and that would trigger an alarm if and when the pause was violated.
This raises more questions than it answers, since it’s not clear what such an oversight system might look like in practice. Again, Cold War-era nuclear agreements can provide a rough template, but it’s by no means a one-to-one match. “Training runs are far easier to conceal than missile siloes,” the post acknowledges. (It should also not be forgotten that despite postwar international agreements against the proliferation of nuclear weapons, nuclear powers like the U.S., China, and Russia have once again been building up their nuclear arsenals.)
The new blog post noted that in the coming months, Anthropic plans to meet with policymakers, its competitors in the industry, and other stakeholders to discuss these challenges, and hopefully work towards building oversight systems that would make a global pause actually implementable. Again, possibly with an eye towards its future shareholders, the company gave what feels like a halfhearted commitment: “If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.”
Read the full article here
