What you need to know about autonomous weapons
The nascent tech that’s caused friction between AI companies and the Pentagon
On Friday, Pete Hegseth directed the Department of War to designate Anthropic a supply chain risk for refusing to grant the military unrestricted access to its models. Hours later, OpenAI announced its own deal with the DoW, with red lines that appeared similar to Anthropic’s. Upon closer inspection, however, those red lines start looking pretty thin.
One such red line: “No use of OpenAI technology to direct autonomous weapons systems.”
What is an “autonomous weapon”?
No one can agree on a definition.
Weapons, like all technology, exist on a spectrum — in this case from “fully human-controlled” to “this could kill someone with zero human input.” My relatively low-tech car, for example, is mostly under my control, but features such as cruise control can automate the act of driving at a constant speed down the highway. Many features are automatic, but it’s nowhere close to an autonomous self-driving vehicle.
A 2019 International Committee of the Red Cross (ICRC) report stated that there’s “no clear technical distinction between automated and autonomous systems, nor is there universal agreement on the meaning of these terms.” The US DoD Directive 3000.09, the policy that OpenAI mentions in its agreement with the Department of War, distinguishes between autonomous and semi-autonomous systems, but doesn’t prohibit either. It just requires both to afford “appropriate levels of human judgment over the use of force.”
Do autonomous weapons (by some definition) already exist?
Kind of.
Today, weapons capable of autonomous behaviors like independently spotting and pursuing targets are actively being used in the Russia-Ukraine War. The US’s Replicator program is developing autonomous drone systems for Taiwan to defend against potential Chinese attacks, while China trains its own autonomous drones and robots.
But none of these weapons are fully autonomous yet, meaning humans are still involved at some point(s) along the kill chain — often as a hardware limitation, but sometimes just as a software choice.
How autonomous are they, then?
Take, for example, the Bumblebee, a cutting-edge AI-powered drone operating in Ukraine. Once a human selects a target, these drones, provided by companies led by former Google CEO Eric Schmidt, can carry out the rest of their attack — including a terminal strike — independently.
However, such weapons still face limitations including short battery lives and lower accuracy than skilled humans. “Autonomous weapons with sustained endurance, high flexibility and the ability to discern, identify, rank and pursue multiple categories of targets independent of human action have yet to appear,” the New York Times recently reported.
That’s exactly what the US military and its favorite defense tech builders dream of. Schmidt has described his ideal military vision as “two layers of drones”: one layer for unmanned aerial surveillance, and another for bomb attacks. With a system like this, he said, “it would be essentially impossible to invade a country by land.” And it’s an active work in progress. Earlier this year, for instance, the Pentagon launched a $100m prize challenge to produce “voice-controlled, autonomous drone swarming technology.” Both SpaceX and Anthropic reportedly made pitches.
OpenAI claims its models are “cloud-only” and not deployed on “edge devices”...what does that mean?
In its published agreement with the DoW, OpenAI claims it will enforce its “red line” against fully autonomous weapons systems through “cloud-only deployment” and withholding its models from “edge devices.” The reasoning is that, if a weapon is going to make independent real-time decisions, as one must on the battlefield, the AI model powering it must live within the weapon itself (that’s “the edge”). If the model instead lives in a data center somewhere, the weapon wouldn’t be able to function without being connected to the cloud — and that connection could be easily severed.
Anthropic rejects this premise, The Atlantic reported. The distinction between the cloud and the edge is “less a wall and more of a gradient,” Ross Anderson wrote. “Drones on the battlefield can now be orchestrated through mesh networks that include cloud data centers. And while they’re designed to survive on their own, the military’s impulse will always be to maintain as much connectivity between them and the most powerful models in the cloud; the better the connection, the more intelligent the machine.”
“Even if most kill decisions are carried out on a local machine, most of the decisions leading up to that — the ‘autonomous kill chain’ — involve running powerful algorithms in the cloud first,” wrote Hayden Field at The Verge, based on information from a source with knowledge of AI company negotiations with the Pentagon. “Even if OpenAI’s tech isn’t directly involved in pulling the trigger, it could very well be powering everything leading up to that point, with no guarantee a human oversees the final step.”
Drones of the kind already used in Ukraine often have both: an on-device computer vision model running on a cheap Raspberry Pi makes split-second perceptual judgments like “is that a target?,” while everything else happens in the cloud. A frontier model doesn’t need to be in a drone to help it autonomously kill someone.
OpenAI also said the DoW can’t use its AI systems for “high-stakes decisions that require approval by a human decisionmaker”... what does that actually mean?
The company claims that weapons aided by its models will still have a human in the loop, “where law, regulation, or Department policy requires human control.” OpenAI adds that its deployment architecture, such as “running and updating classifiers,” will help it ensure that these conditions are being met.
But according to The Verge’s source, “that’s not necessarily true.” Classifiers “wouldn’t be able to confirm whether a human reviewed an AI system’s decision to attack a target before the kill strike … and if the government determines an action is legal, then OpenAI’s classifiers wouldn’t be allowed to prohibit the technology from carrying it out.”
Sam Altman has said that one of OpenAI’s most important safety principles is “human responsibility for the use of force, including for autonomous weapon systems,” but without seeing the full DoW contract, it’s hard to say what humans will actually be “in the loop” for. On Tuesday, he reportedly told OpenAI staff the company “do not get to make operational decisions” about what the Pentagon does with its technology.
Are AI companies against lethal autonomous weapons altogether?
No — they just think they’re not ready yet. OpenAI, Anthropic, and the US military all seem to view fully autonomous weapons as an inevitable part of future warfare. In a statement last week, Dario Amodei said that they “may prove critical for our national defense,” and that Anthropic “offered to work directly with the Department of War on R&D to improve the reliability of these systems.” However, Amodei added, “frontier AI systems are simply not reliable enough to power fully autonomous weapons. We will not knowingly provide a product that puts America’s warfighters and civilians at risk.”
Consider the difference between autonomous vehicles and autonomous weapons. Waymo (mostly) works today because it spent years collecting millions of miles worth of driving data, training on that data, then running rigorous evaluations with human oversight. And even then, it only works in limited, carefully mapped locations. The same hasn’t happened for autonomous weapons. General purpose frontier AI models, including Claude and ChatGPT, would likely require similarly extensive training on battleground data to effectively and reliably handle war-like tasks like “independent target pursuit.”
That means there’s currently no foolproof way to predict what these models will be good or bad at in the battlefield. It was funny when GPT-4 couldn’t count the number of ‘r’s in ‘strawberry’. It would be extremely unfunny if the model powering an autonomous drone consistently, or even occasionally, failed to recognize human children.
This could change in the future, but would likely require specifically creating new training data sets, evaluations, and safety benchmarks from scratch. (And the organizations that seem most willing to do this today, such as defense tech startup Anduril, lack the resources to do so at the scale of OpenAI or Anthropic.)
So why are we building them anyway?
In 2015, over 1,000 leading AI and robots researchers — including Elon Musk — signed an open letter urging the UN to ban autonomous weapons. “The key question for humanity today is whether to start a global AI arms race or to prevent it from starting,” they wrote. “Starting a military AI arms race is a bad idea.”
Unfortunately, the arms race has already begun. Spending on autonomous weapons is already estimated to have hit almost $18bn last year despite current limitations, and is expected to keep growing rapidly. It’s a textbook multi-player prisoner’s dilemma: if no one trusts that their opponent will stop, then everyone will keep going.
There have been various attempts in recent years to reach international agreement on autonomous weapons. Currently, 130 countries have in some form indicated their support for a legally binding instrument governing their use. Those that haven’t include Russia, Israel and the US.
If readers want to stay up to date about the process of getting to a legal ban on UN level, give Stop Killer Robots a follow: https://substack.com/@stopkillerrobots?
Negotiations are still ongoing, this year we should at least agree on a definition, a framework for negotiations ánd open formal negotiations!
Just why do we need these? Why are we are so stupid when it comes to technology? At the Open AI protest yesterday, an AI professor spoke. I didn't get his name and I'm unclear what his job exactly entails since AI requires no brain function, but he did say we have a five year window to destroy AI and all it's data centers or it would kill everyone on the planet. Or we could all be enslaved by the billionaires. Tech people say: full speed ahead!