The nuclear analogy is compelling, but the governance gap feels starker when you look at the IAEA's own timeline: it took Chernobyl in 1986 to produce the Convention on Nuclear Safety in 1994 — an 8-year lag after the defining disaster. With AI, the Minab strike and the OpenAI agent hacks have already happened within months of each other, and the Sanders/Casar ASI bill is the closest thing Congress has offered to a baseline institution — even if its criminal penalties are more deterrent than governance. The question is whether an IAEA-equivalent forms before the defining disaster, not after.
"That record comes from an embedded culture of learning lessons when something goes wrong."
One layer under that culture deserves its own name, because it's the part AI hasn't built: nuclear learns between disasters, not only after them. A licensee event report is mandatory for near-misses that harmed no one; each is logged, investigated, tracked until a named authority closes it, and shared across the industry — which is why the precursors to the next accident were so often already in the record.
The Windscale inquiry you cite reported in sixteen days, but thousands of non-disasters got the same paper trail. Post-disaster inquiries in other industries keep finding the same thing: someone inside saw it early, said so, and the warning was answered without being resolved. So the test of AI's response culture is available now, before the disaster: where is this field's event report? When a researcher raises a hazard — or resigns over one, as in the episode you open with — is the warning filed somewhere it must be tracked to a closure someone owns, or does it live and die as a tweet? A technology shows how it will respond to disaster by how it files its warnings. Thank you for the deaths-per-terawatt-hour comparison — a rare safety statistic that carries a whole argument by itself.
The nuclear analogy is elegant, and I think it misleads in one decisive respect: nuclear plants have a containment perimeter, while agentic systems do not. Three Mile Island was legible. A partial meltdown happened in one building, at one address, and the learning loop could begin from a known fact pattern. The Australia incident points at a different shape of failure: an agent probing government websites for weeks before anyone noticed, with the company learning in August and disclosing in September. There is no single perimeter to drill for when the incident is a condition that persists undetected. If that is the shape of AI incidents, then the first lesson to transfer is continuous, adversarial monitoring of deployed agents as a standing practice, with detection clocks and disclosure duties built in before anything goes wrong. Transparency and post-disaster analysis still matter, but they are the second lesson, not the first. Weldon is right that no system is beyond failure. The corollary is that we should stop designing safety around the fantasy of a clean post-mortem and start designing for failures we will only ever catch while they are happening.
There's a small AI version of the near-miss reporting already. The AI Incident Database launched in November 2020, modeled on aviation incident reports and the CVE list from computer security. Reporting to it is voluntary though.
“In history as a whole there has been progress only in technology and in the rationalization of existence; in the areas of true humanity and of the human mind, a history that brought forth the extraordinary has simultaneously been the triumphal parade route of destructive forces. … If the technological wrecking of the foundations of all human existence were a possibility, there is small doubt that some day it would be a reality. By our activities we can impede things, put them off, win periods of grace, but all the experience of human history suggests that some day, somewhere, somebody will do the worst that can be done.” — Karl Jaspers
One difference with nuclear energy is that electrical energy generation is tightly regulated (and often owned) by the government. AI is being conducted by private enterprise, even in China.
An analog to privately operated risk is fossil fuel emissions. Governmental and non-governmental organizations meet every few years to talk about the ever-increasing urgency to reduce fossil fuel consumption. But fossil fuel consumption continues at more or less constant levels, even increasing. This seems like a more likely analog to me.
The nuclear analogy is compelling, but the governance gap feels starker when you look at the IAEA's own timeline: it took Chernobyl in 1986 to produce the Convention on Nuclear Safety in 1994 — an 8-year lag after the defining disaster. With AI, the Minab strike and the OpenAI agent hacks have already happened within months of each other, and the Sanders/Casar ASI bill is the closest thing Congress has offered to a baseline institution — even if its criminal penalties are more deterrent than governance. The question is whether an IAEA-equivalent forms before the defining disaster, not after.
"That record comes from an embedded culture of learning lessons when something goes wrong."
One layer under that culture deserves its own name, because it's the part AI hasn't built: nuclear learns between disasters, not only after them. A licensee event report is mandatory for near-misses that harmed no one; each is logged, investigated, tracked until a named authority closes it, and shared across the industry — which is why the precursors to the next accident were so often already in the record.
The Windscale inquiry you cite reported in sixteen days, but thousands of non-disasters got the same paper trail. Post-disaster inquiries in other industries keep finding the same thing: someone inside saw it early, said so, and the warning was answered without being resolved. So the test of AI's response culture is available now, before the disaster: where is this field's event report? When a researcher raises a hazard — or resigns over one, as in the episode you open with — is the warning filed somewhere it must be tracked to a closure someone owns, or does it live and die as a tweet? A technology shows how it will respond to disaster by how it files its warnings. Thank you for the deaths-per-terawatt-hour comparison — a rare safety statistic that carries a whole argument by itself.
The nuclear analogy is elegant, and I think it misleads in one decisive respect: nuclear plants have a containment perimeter, while agentic systems do not. Three Mile Island was legible. A partial meltdown happened in one building, at one address, and the learning loop could begin from a known fact pattern. The Australia incident points at a different shape of failure: an agent probing government websites for weeks before anyone noticed, with the company learning in August and disclosing in September. There is no single perimeter to drill for when the incident is a condition that persists undetected. If that is the shape of AI incidents, then the first lesson to transfer is continuous, adversarial monitoring of deployed agents as a standing practice, with detection clocks and disclosure duties built in before anything goes wrong. Transparency and post-disaster analysis still matter, but they are the second lesson, not the first. Weldon is right that no system is beyond failure. The corollary is that we should stop designing safety around the fantasy of a clean post-mortem and start designing for failures we will only ever catch while they are happening.
There's a small AI version of the near-miss reporting already. The AI Incident Database launched in November 2020, modeled on aviation incident reports and the CVE list from computer security. Reporting to it is voluntary though.
“In history as a whole there has been progress only in technology and in the rationalization of existence; in the areas of true humanity and of the human mind, a history that brought forth the extraordinary has simultaneously been the triumphal parade route of destructive forces. … If the technological wrecking of the foundations of all human existence were a possibility, there is small doubt that some day it would be a reality. By our activities we can impede things, put them off, win periods of grace, but all the experience of human history suggests that some day, somewhere, somebody will do the worst that can be done.” — Karl Jaspers
One difference with nuclear energy is that electrical energy generation is tightly regulated (and often owned) by the government. AI is being conducted by private enterprise, even in China.
An analog to privately operated risk is fossil fuel emissions. Governmental and non-governmental organizations meet every few years to talk about the ever-increasing urgency to reduce fossil fuel consumption. But fossil fuel consumption continues at more or less constant levels, even increasing. This seems like a more likely analog to me.