AI cyberrisk might be a bit overhyped — for now at least
Experts say key factors currently limit the risk of catastrophic harm from AI-enabled cyberattacks — as far as we know
What should we make of Anthropic’s disclosure that it has identified hackers using Claude to “commit large-scale theft and extortion of personal data” and by North Koreans to obtain remote working jobs at US firms, revealed in its most recent threat intelligence report released in late August?
Whenever any technology gets into the hands of white hat users deploying it for legitimate means, it’s only a matter of time before more nefarious individuals put the same tech to work for black hat measures. “It’s not really hypothetical,” says Juraj Jánošík, head of AI at cybersecurity firm ESET. “We’ve seen it’s already happening.”
Around the same time as Anthropic disclosed its threat intelligence report, ESET identified what it claimed was the first AI-powered ransomware, PromptLock. The self-sustaining system could generate malware and communications with victims designed to extort them.
These are troubling developments. But beyond scammers and cybercriminals, how do those who are tracking AI development — and how bad actors are using it — feel about AI-related cyberrisk of a more catastrophic, market-moving and world-changing level? Is an AI incursion into a nuclear facility, or a power grid, a risk today?
Some are relatively sanguine about the potential risks of the current crop of AI. “In our team we’re on board with the more cyber-chill takes for various reasons,” says John Halstead, research fellow at the Centre for Governance of AI. “One being that the base rate of damages from cyberattacks just doesn’t seem all that big in the grand scheme of things.”. Halstead points out that potentially the most economically costly cyberattack in history was NotPetya, which caused $10 billion in damages. While significant, that’s not world-changing. “And usually, the very large ones are more in the hundreds of millions of dollars range,” he adds.
Bruce Schneier, a security technologist and fellow at the Berkman Klein Center for Internet & Society at Harvard University, also isn’t too concerned about cyberrisk from AI — yet. He points out both attackers and defenders are experimenting, but AI’s net impact on cyber operations is still unclear. “AI is doing all sorts of things, and I don’t think we know really what, where or really how it’s going to shake out,” he says. But overall, “I’m not worried.”.
There is currently a fair amount of uncertainty about whether the current cutting edge of AI is up to the job when it comes to cracking into those most important bits of infrastructure. “It’s a very high capability bar,” says Halstead. “The sort of tasks involved in carrying out a cyberattack are very diverse, and require lots of autonomous action in these novel and documented environments.” That’s something Jánošík agrees with. “We are far from a standalone AI model, which is fully capable of doing its own thing without human interaction giving it a purpose,” he says.
But not everyone is quite so content with the current state of affairs. “AI is often used to assess risk and quantify risk and for risk mitigation, but we are being exposed to situations and technology and interaction between systems that is unprecedented, and that we’ve never been exposed to before,” says Carissa Véliz, associate professor at the Institute for Ethics in AI at the University of Oxford.
That unprecedented nature means we’re ill-equipped to understand or quantify the potential risks — in large part because we might not know what they could be, she says. “We’ve never had LLMs before, and we’ve never used them the way we’re using them, so there isn’t a database for it, and there is a risk that we might feel safer than we are.”
Véliz acknowledges that areas of high national security importance, such as vital networks and systems — including those around nuclear facilities — are likely to have more checks and balances than others. “However, I don’t know whether there are enough measures in place so that we can be certain that they are careful enough,” she says.It’s not just the known knowns that could imperil us, she reckons. It’s the unknown unknowns, too.
She makes the point that it’s important to get this right, first time. “We can’t just wait for it to go wrong and then correct it,” she explains, “because it can go really, very wrong.”
That’s an overly pessimistic view according to others, in part because of the role AI can play in bolstering resilience to attacks. Schneier foresees a future where AI is as much of a help in preventing the worst cyberrisks as it is a worry. “We can imagine over 10 years of the future when AI vulnerability finding is built into every compiler, and bugs are fixed before the product’s released,” he says. That’s doubly likely for high value infrastructure that could be subject to attack, where sending it offline would materially affect economies and societies.
At the moment, Schneier believes time is on the defenders’ side. “Can you set an AI on a general task of, ‘Break into this network’ and have it do so?” he asks. Early experiments in doing so suggest that “the results aren’t great,” he explains. “They’re not zero, but they’re not great.”
He also points out that every development on the offensive side by hackers is usually quickly met with a commensurate defensive measure once those subject to attack learn how it works. “In the long run, this benefits the defender, because both the attackers and defenders can find vulnerabilities,” he says. “And when the defender finds them, they fix it.”
A problem Schneier does forsee is that while that’s the current situation, things change quickly in both AI and cyber. “It’ll get better,” Schneier says of AI’s capabilities to launch attacks. “We know how this game is played.”
He says we’re in an odd interregnum where we haven’t yet gamed out all the potential risks and taken actions to mitigate them. “The interim is kind of scary,” says Schneier, “but in the long run, this particular capability benefits the defenders more.”
At this moment the interplay involved in getting into the most secure system is something that stymies the best human hackers. “Getting to that world is very close to AGI,” Halstead says. That may remain an insurmountable challenge for the AI we have today, if perhaps not beyond what’s over the horizon.
 | A guest post by
|