Fears over the danger of artificial intelligence have literally gone nuclear.
Ahead of a scheduled meeting in Washington next week between U.S. President Donald Trump and his Chinese counterpart, Xi Jinping, experts are calling for a shared framework between the two countries to mitigate the risks of an AI-triggered cyberattack or nuclear strike. They address a concern similar to the one that’s driven multiple AI industry insiders to warn about possible extinction-level events: Humans are handing over too much autonomy over critical systems to AI agents they can neither understand nor control.
In a report published last week, the Brookings Institution’s Melanie Sisson and Fudan University’s Tianjiao Jiang wrote that it was crucial for both countries to ensure that critical military decisions—like when to launch a nuclear retaliatory strike—remain solely in human hands.
“As militaries integrate agentic AI into their operations and AI models advance rapidly in sophistication and capability, new pathways to catastrophic outcomes are emerging,” Sisson wrote in the report. “Presidents Donald Trump and Xi Jinping can lead in governing these dangers by agreeing that only humans, not AI, should make the decision to initiate a cyberattack against each other’s nuclear command, control, and communications systems (NC3) or critical infrastructure.”
Sisson and Jiang suggested that the U.S. and China work together to establish “red lines” around the military use of AI, rules determining what “meaningful human control” looks like in practice, and a dedicated “military hotline for AI,” through which officials in both countries could quickly report “emergency incidents triggered by AI errors or AI cyberattacks.”
Former president Joe Biden reached an agreement with president Xi in late 2024 that neither the U.S. nor China should allow decisions related to the use of nuclear weapons to be handled by AI. But tensions between the two countries have ratcheted up since Trump returned to office just a couple of months later, in large part over their competition to build ever-more powerful AI systems. And the current President has shown no signs that he feels beholden to honoring agreements struck by previous administrations.
The Trump administration has also eagerly embraced AI tools in its ongoing military campaign against Iran, including the use of Palantir’s Maven to gather and parse battlefield information that helped military officials identify targets for future strikes. And while a growing number of AI researchers and ordinary Americans have begun to worry about the possibility of an AI-triggered doomsday, the country’s 80-year-old commander-in-chief has dismissed such fears as a “SICK conspiracy” spread to benefit China, while also posting AI-generated images to social media depicting overseas destruction and himself as a competent, imposing leader.
Fears that a simple miscommunication, technical glitch, or bureaucratic hiccup could trigger an international nuclear exchange are not new. During the height of the Cold War in September 1983, a lieutenant colonel in the Soviet Air Defense Forces named Stanislav Petrov received a warning from an automated early-warning satellite system that five U.S. nuclear warheads were headed directly towards the Soviet Union. If he had alerted his superior officers in the military about the apparently inbound missiles, they very likely would have authorized retaliatory strikes. Instead, in a moment of awe-inspiring level-headedness, he reasoned that if the U.S. were going to launch a full-scale nuclear attack on its Cold War adversary, it would’ve used hundreds of bombs from its vast arsenal, not just five. He phoned his superior officers to report a false alarm. It was later determined that the satellite system had picked up flares of sunlight bouncing off of clouds and interpreted them as incoming warheads.
Petrov may have single-handedly spared the world from armageddon that day. But an AI system entrusted with making similar judgment calls, based purely on analyzing real-time data, might not have made the same decision. Even when handed ostensibly routine, harmless tasks, modern AI agents are prone to all kinds of unpredictable behavior, as was dramatically demonstrated by the recent Hugging Face hack. Multiple insiders from frontier AI labs have come forward over the past week to warn that the capabilities of these agents are advancing much quicker than our understanding of how to ensure they’re aligned with human interests and values.
Earlier this month, Senator Bernie Sanders—one of Capitol Hill’s most vocal critics of the unregulated AI sector—called for an international ban on the development of so-called “superintelligence,” with punishments on par with those in place around the illegal development of nuclear weapons. In July, United Nations Secretary-General António Guterres urged the 193 U.N. member states to unilaterally outlaw the use of autonomous weapons systems. “Some decisions must remain forever human—none more than taking a human life,” Guterres said during his speech, delivered at the inaugural Global Dialogue on AI Governance in Geneva.
Read the full article here
