When it comes to the role of AI in nuclear warfare, one thing nearly everyone can agree on is the importance of keeping a “human in the loop” on the decision to actually use deadly force.
The human people have in mind is presumably someone like Stanislav Petrov, the Soviet military officer who famously, in 1983, made the risky decision not to alert his superiors when a computerized early warning system detected an apparent US missile launch. Petrov believed, correctly as it turned out, that it was false and is sometimes credited today with having “saved the world.” The notion that it should be humans, for all our flaws, who make the ultimate decision to use the world’s deadliest weapons is what motivated the 2024 joint statement between the US and China that AI systems should not be given authority to launch nuclear weapons.
But an alarming incident that came to light in September illustrated the limits of this comforting line of thinking. CNN reported on September 18 that in the midst of the war with Iran, the US military had been preparing to board a Chinese ship in the Middle East believed to be transporting components of a nuclear weapons program. That belief was based on an “entirely false” intelligence report by an analyst using artificial intelligence that misidentified the cargo the ship was carrying. According to the article’s sources, “military planes were in the air” before the error was discovered and one believes the incident “almost started a war” between two rival nuclear superpowers.
This is a very different sort of AI-enabled existential risk scenario from the ones that have dominated headlines over the past few weeks. The concern is not superintelligent sovereign software making decisions that put humans at risk, but humans themselves making decisions with the help of AI that kills other humans.
We don’t know from the CNN report exactly how the analyst used AI to reach the conclusion that the ship was carrying nuclear components, or how the error was eventually discovered. But Jacquelyn Schneider, director of the Wargaming and Crisis Simulation Initiative at Stanford University’s Hoover Institution, said the scenario was all too believable from her early career as a US Air Force intelligence officer.
“Before AI, we always had bad intel officers,” she told Transformer. “A bad intel officer doesn’t brief [colleagues on] the uncertainty behind where they got the data from. They don’t even question where they got the data from. That exists with or without AI, but with AI, it’s so much easier to be confident in the assessment and to have so much less visibility into where the data is coming from.”
For decades, experts have been warning about the paradoxical phenomenon that when systems become more automated, overall errors decrease but the percentage of errors attributable to humans increases. This is known as “automation bias”: human users are both overly trusting of information provided by automated systems and ignore signs of problems if they haven’t been flagged by those same systems. The incident with the analyst’s report on the Chinese ship would appear to be a textbook example of this type of bias at work.
This is a cause for concern as militaries, the US very much included, have been aggressively pushing to integrate AI into surveillance, intelligence gathering and targeting operations. This week, Defense Secretary Pete Hegseth announced the formation of a new Autonomous Warfare Command to oversee the deployment of drones and artificial intelligence.
The Israeli outlet +972 Magazine has reported that in the early days of the Gaza war, Israel Defense Forces personnel often served as a “rubber stamp” for decisions made by an AI bomb targeting system known as Lavender, despite evidence that the system, designed to identify Hamas targets, made errors as much as 10% of the time, contributing to the high numbers of civilian casualties.
There are also ongoing questions about the role Palantir’s Maven Smart System — the AI command-and-control system that has been called the Pentagon’s “everything app” — played in the bombing an elementary school in Minab on the first day of the Iran war. The strike, which killed more than 150 people, mostly children, was likely due to outdated data fed into the system. This week, Bloomberg reported that the bombing was just one of 1,000 targets struck on the first day of the war, in a coordinated barrage enabled by Maven. Palantir responded to the incident with upgrades designed to get the system to re-review its underlying intelligence before making recommendations.
In Gaza and Iran, human overreliance on AI-synthesized intelligence may have already led to civilian casualties. In a US-China scenario, those kinds of errors could cause a world war and a far greater death toll.
For all the very warranted attention in AI safety debates on making sure these systems behave in ways that are aligned with human values and preferences, the bigger risk may come from how the humans working with the systems behave. It was ultimately humans, not AI systems, that chose to greenlight the targets selected by Lavender, the Minab school bombing, and — very nearly — approved boarding a Chinese vessel in a war zone.
For the past few years, Stanford’s Schneider has been conducting war games simulating scenarios in which an errant AI targeting system triggers US-China conflict in the Taiwan Strait — scenarios disconcertingly reminiscent of what recently transpired in the Middle East.
“When you think about the decision tree that gets you to launching a nuclear weapon, the AI integration could be very, very early in the decision-making process,” she said. That’s a much harder problem to address with a blanket policy or international treaty.
The real existential risk from AI may turn out not to be from a rogue program launching a nuclear war on its own, or even advising a national leader to do so, but from military officers or intelligence analysts much lower down in the decision-making hierarchy, under enormous stress and time pressure, relying on AI to make the decisions that start the war in the first place.
Ultimately the partnership between humans and AI could end up being more dangerous than the AI on its own. The idea of keeping a “human in the loop” on any life-or-death decisions involving AI may seem comforting, but it requires that the human in question is actually up to the task.
