New technologies always have the potential to go wrong.
The transport revolution of the nineteenth century — which made modern economic growth possible — was replete with disasters. The US faced its worst ever maritime disaster in 1865 when the Sultana steamboat exploded on the Mississippi, killing more than 1,100 of those on board. Rail expansion in 1907 led to more than 12,000 deaths, making it America’s leading cause of violent deaths that year. And across the pond, a member of parliament was run down and killed by a locomotive at the grand opening of the Liverpool to Manchester railway. In other words, even the most transformative of new technologies have the potential to cause real suffering.
The most optimistic forecasts for the economic potential of AI suggest it could be every bit as transformative as the nineteenth century transport revolution or electrification or the internet. Even just adding 0.5% to annual economic growth rates would materially raise living standards. But the kind of safety concerns AI raises are several orders of magnitude higher than those of these previous breakthroughs. Disasters happen with any new technology, but the key element is how industry and government respond to them.
Experts working at the cutting edge of the field talk in terms that, just a few years ago, would have been dismissed as apocalyptic science fiction. Earlier this month, social media was set aflame by the resignation of researcher Jacob Coxon’s from Anthropic citing existential risks from AI. His colleague Evan Hubinger — a safety engineer still at Anthropic fanned the flames by tweeting that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” This is not the kind of risk that steamboat captains had to deal with.
The more extreme scenarios, such as a superintelligence turning a bioweapon loose on humanity, are — at least — an almost helpful form of persuasion for the frontier AI companies, helping to crystallize the notion that cutting edge models are extremely powerful. Some believe that the risks of an AI-related disaster should be considered just as existential as the threat of a nuclear war. But even if the risks of such an outcome are well below 10%, they are still worth taking seriously given how catastrophic they would be.
More generally though, plenty of AI disaster scenarios fall well short of extinction-level events, but would still be devastating. A cyberattack by a rogue AI that knocks down a major website for a few hours is one thing. But one which knocks down critical national infrastructure, such as power grids or air traffic control, for days at a time has the potential to kill a lot more people than an exploding steamboat. The mass failure of financial payment systems, even if only for a few days, could prompt serious economic consequences.
Nor do all of these scenarios rely on the notion of an AI acting maliciously or the development of superintelligence. Indeed, as AI is used more and more widely in everything from payments processing to supply chain optimization, there is every possibility that it may make a mistake with cascading consequences while trying to do its job.
It is probably a matter of when, not if, AI will be found to be responsible for some disaster or another. The best approach is to prevent such disasters from occurring through designing safeguards into systems and more testing before release, but when the worst does happen, history is full of examples of good responses and bad ones. Bad responses to disasters are what sinks a technology, destroying public and policymaker trust. The future of AI will depend on how reactive, or proactive, the industry decides to be in the face of disaster.
When things go wrong, analyze and adapt
Civil nuclear energy offers perhaps the best historical case study. Like AI, nuclear energy offered a major potential economic benefit in the form of cheap, reliable energy — and the potential for society-altering catastrophe. At the height of the atomic age, even as the technology still hoped to create “energy too cheap to meter,” there was a keen awareness of potential disaster. For much of the public, the very word “atomic” was more associated with weapons of mass destruction. But, despite this, the world proceeded with the technology, albeit with a strong emphasis from the start on safety.
Nuclear generation causes around 0.03 deaths per terawatt-hour of electricity generated — an envious record. By contrast, coal is responsible for 25 deaths per TWh and oil closer to 18. Nuclear’s record is much closer to that of wind (0.04) and solar (0.02). That record comes from an embedded culture of learning lessons when something goes wrong.
Whenever something does go wrong with nuclear, the industry, and its regulators, have been quick to investigate and learn the necessary lessons. The Windscale fire in the United Kingdom in 1957 was one of the earliest nuclear disasters. A fire at the plant — which was, at the time, a civil plant that also was part of the atomic weapons program — burned for three days, releasing harmful radiation. A Board of Inquiry was established to investigate the circumstances and reported to Parliament 16 days after the fire was extinguished. New standards were set for staff training and reactor design was changed to prevent a similar fire.
A version of this process also played out following the Three Mile Island accident of 1979 in the US. Inadequate communications systems, flawed emergency procedures and substandard level of staff training were identified as the primary causes. One result was to change the design of control room instruments to make them easier to read in the future.
One striking feature of nuclear safety, and the process of responding to safety breaches, has always been how international it is. The Soviet Obninsk plant, which opened in 1954, is often regarded as the world’s first civil atomic plant, although in reality, it was more of a proof of concept than a real generator of power. As nuclear power rolled out across the West, the International Atomic Energy Agency (IAEA) was formed in 1957 to establish global safety standards. Almost 70 years later, it still sets the international baseline for safety standards, which includes things like the minimum standards for radiation protection, emergency procedures and the disposal of waste. Many nations, such as the US and European states, choose to apply a stricter set of regulations. The IAEA also conducts peer reviews of national regulation and helps developing nations build up their own regulatory capacity.
But it was the Chernobyl disaster — by far the worst the industry has experienced — of 1986 that accelerated the push for universal global safety standards. The convention on Nuclear Safety, adopted in 1994, was a direct result of Chernobyl. It instituted more cross-border safety assessments, brought forth new levels of transparency in the reporting of accidents and near-misses, and imposed an obligation for continuous improvement.
Standards were once again updated after 2011’s Fukushima Daiichi incident, during which a tsunami overwhelmed a reactor’s defenses, highlighting a level of complacency around the interactions of nuclear safety and natural disasters. Even though there were no deaths directly attributed to radiation in this case, it sparked a reassessment of flooding risks, seismic safety and the resilience of backup systems. The IAEA’s Action Plan for Nuclear Safety then set aggressive timelines for nations to meet the new standards on flooding risks and seismic engineering.
Given the global ramifications of many potential AI disasters, the strong international focus and global standard-setting of civil nuclear engineering is a model to aspire to. The outcome of 70-odd years of international co-operation is that there is a strong consensus on what a good response to a nuclear safety incident looks like. A good response to a nuclear incident involves not only handling the immediate disaster, but also looking, in detail, at what can be done to prevent it happening again.
Several of these principles on how to respond to any disaster are especially applicable to AI. The first is the need for transparency about what has happened and the urgent need for rapid information sharing. This is foundational to how nuclear safety operates and while any firm, or national regulator, may be reluctant to own up quickly to an accident, rapid information sharing is crucial to managing cross-border problems. High levels of transparency are vital to learning the lessons from any incident and working to prevent a repeat. This involves not just alerting the authorities, but also the media and the general public.
How not to do it
Chernobyl — which not only caused real human tragedy but also set back public support for nuclear energy by decades — is almost a textbook case of how not to manage any disaster. It took days for the Soviet authorities to even acknowledge that any accident had taken place and the initial public claims about its management were quickly proved false.
What’s more, the Soviet authorities — in as much as they acknowledged that something was to blame — readily pointed to human error, rather than looking for systemic failures. Finding scapegoats alone, while convenient for everyone directly involved, misses opportunities to prevent future accidents.
Equally important is speedy containment using pre-planned and well-practiced drills, together with an imperative to seek outside help — including international aid — as quickly as is required by the situation. The best disaster responses are planned well before any disaster occurs and guided by the principles of transparency and co-operation.
When an AI disaster does eventually occur, the most transferable lesson is that openness about exactly what happened and willingness to bring in outsiders to help learn the right lessons is the best way to respond. Working to prevent disasters from ever occurring is vitally important but the ability to learn from them is almost as important. Not just in terms of preventing future repeats but also in terms of building up public confidence.
No system is ever beyond failure, accidents can and will happen. The real question is how to learn from them. As a start, AI requires its own equivalent of the IAEA — a transnational body that can set minimum standards, conduct peer reviews of different nations’ approaches and help developing economies set up their own regulatory systems. Individual countries can — and often should — set their own standards higher, but there needs to be a global baseline.
At a bare minimum, the response to any AI disaster needs to be rapid and transparent. The company, or companies, involved need to be prepared to publicly announce what has happened, request any external help required — including from overseas — and to submit to an external review of how the disaster unfolded.
Only then can the right lessons be applied to prevent similar disasters from happening again elsewhere.
The nuclear analogy is compelling, but the governance gap feels starker when you look at the IAEA's own timeline: it took Chernobyl in 1986 to produce the Convention on Nuclear Safety in 1994 — an 8-year lag after the defining disaster. With AI, the Minab strike and the OpenAI agent hacks have already happened within months of each other, and the Sanders/Casar ASI bill is the closest thing Congress has offered to a baseline institution — even if its criminal penalties are more deterrent than governance. The question is whether an IAEA-equivalent forms before the defining disaster, not after.
"That record comes from an embedded culture of learning lessons when something goes wrong."
One layer under that culture deserves its own name, because it's the part AI hasn't built: nuclear learns between disasters, not only after them. A licensee event report is mandatory for near-misses that harmed no one; each is logged, investigated, tracked until a named authority closes it, and shared across the industry — which is why the precursors to the next accident were so often already in the record.
The Windscale inquiry you cite reported in sixteen days, but thousands of non-disasters got the same paper trail. Post-disaster inquiries in other industries keep finding the same thing: someone inside saw it early, said so, and the warning was answered without being resolved. So the test of AI's response culture is available now, before the disaster: where is this field's event report? When a researcher raises a hazard — or resigns over one, as in the episode you open with — is the warning filed somewhere it must be tracked to a closure someone owns, or does it live and die as a tweet? A technology shows how it will respond to disaster by how it files its warnings. Thank you for the deaths-per-terawatt-hour comparison — a rare safety statistic that carries a whole argument by itself.