Anthropic calls for global AI pause amid recursive self-improvement risks

45

The Acceleration Toward Recursive Self-Improvement

The core of Anthropic’s alarm lies in a fundamental shift in how AI is built. For years, development relied on human engineers to write code and train models. That era is ending. According to Exame, more than 80% of the code currently being integrated into Anthropic’s development base is written by Claude, the firm’s primary model. This automation has triggered a massive spike in productivity. Internal data cited by the company shows that engineers are currently delivering approximately eight times more code per day than they did in 2024. This surge is driving the industry toward a phenomenon known as recursive self-improvement. In this scenario, AI systems move from being tools used by humans to being agents that can design, develop, and train their own successors. While Anthropic notes that this stage has not been fully reached, the trajectory suggests it could arrive before social, legal, or safety frameworks are prepared to manage it. “If systems are capable of fully building their own successors, the ways to protect them, monitor them, and shape their behavior will become much more important.” Anthropic, via Valor

Geopolitical Tension and the Competition Dilemma

Anthropic’s call for a “pause” is not a plea for a total halt, but for a coordinated, verifiable slowdown. However, the proposal faces immediate friction from the realities of global power politics. As Folha de S.Paulo reported, critics in Washington and Silicon Valley argue that any deceleration could hand a decisive advantage to China in the ongoing technological race. The math of competition creates a “prisoner’s dilemma” for AI labs. G1 highlights a critical concern raised by the company: if only one firm or nation decides to slow down, they risk being left behind by competitors who continue at full speed. This tension is already being codified into policy. A Trump administration executive order released this week has shifted much of the safety burden onto the labs themselves, requesting that they voluntarily submit their most advanced models to government cybersecurity testing before any public release.

Verification and the Nuclear Arms Model

To solve the problem of “cheating” during a pause, Anthropic is looking toward historical precedents in international security. InfoMoney notes that the company is comparing the necessary coordination to the international regulation of nuclear weapons. A successful pause would require:

• Global agreement between major players, specifically the United States and China.
• Verifiable rules that allow peer laboratories to monitor one another.
• Mechanisms to ensure that unprincipled actors do not continue development in secret.

The difficulty lies in the nature of the technology. Unlike nuclear silos, which are physical and difficult to hide, AI training rounds are much easier to conceal. The incentives to “defect” and continue training in secret are immense, as the first entity to achieve true recursive self-improvement could inherit global leadership. “Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while facing competitive and geopolitical pressures.” Anthropic, via Folha de S.Paulo

Anthropic’s Safety Stance and the Mythos Precedent

Anthropic is attempting to leverage its reputation as a “safety-first” laboratory to lead this movement. The company has a history of making controversial decisions based on these principles. For example, it recently refused to allow the United States military to use its models for domestic surveillance or fully autonomous weapon systems. This refusal has already carried consequences. The U.S. government has placed the company on a national security restriction list, with enforcement expected by the end of 2026. The urgency of the company’s warning is underscored by the capabilities of its own tools. Earlier this year, Anthropic’s Mythos model caused significant concern in the banking and software sectors due to its ability to identify and exploit vulnerabilities in existing code. This capability demonstrates that the risks of “uncontrolled” AI are not merely theoretical—they are already manifesting in the software ecosystems that underpin the global economy. As the company prepares for a potential initial public offering, it is attempting to balance its commercial ambitions with a manifesto that calls for a “pause button” to ensure that human governance and social structures can keep pace with the machines they are building.