Black October1 is being followed by an even Blacker November. I don’t know if this is Russians, Iranians, bored teenagers, or autonomous AI swarms that broke out of confinement, but it sort of doesn’t matter. We’ve passed some sort of threshold where 40 years of insecure practices are going to catch up with all of us.
I go back and forth on what I think this means long term. I mean it’s clear in the short term we are going to be in for a rough ride. Those 40 years of insecurity are not going to be cleared up quickly.
It’s quite something to see that all of our carefully constructed “responsible disclosure” practices are being torn up2 — and rightly so. When vulnerabilities were rare, it made sense to slow down a bit, coordinate disclosures across vendors and make sure we can get the mitigations out effectively, without alerting any bad guys to a new zero-day.
These days you have to assume that if any single person has found a vulnerability with AI, then very soon, so will anybody else interested in the same topic.3 If you try and coordinate disclosure, you’re just slowing down defenders while the attackers go full blast at every new vulnerability.4
If that’s the long-term status, that’s a really hard world for us all to live in. The frontier labs have noticed the problem. OpenAI in particular is devoting “a significant amount of [their] frontier compute”5 to this problem — the trouble is, it’s all shutting the stable door after the horse has bolted. Last week they found nearly 1000 previously unreported Linux kernel vulnerabilities and found fixes for them.6 Which, you know, great — but those are 1000 vulnerabilities which are already out there and deployed. This is the problem we already had, they’ve just shown it to us.
There is a way to view this positively. We could quite reasonably now ask OpenAI et al to go across every common piece of software that exists, audit for vulnerabilities and fix them. In fact, someone will do this whether or not we ask them to. And, we can demand of our LLMs that every new piece of software written is free of vulnerabilities. That’s a quite reasonable expectation as well; it’s hard to prove a negative, but these days it’s actually quite hard to get Claude to output code containing vulnerabilities.
Maybe if that happens, and then we wait a year or two for install cycles, then by 2030 we could be looking at a future world completely free of software vulnerabilities. I’ve spoken to a couple of very senior cybersecurity researchers who believe this wholeheartedly. They’re going to batten down the hatches for the next couple of years, in the expectation that a golden age of computer security will dawn within the next decade.7
I don’t think I agree. Or rather, yes, I could probably buy into the idea that 5 years from now, nobody is writing software with vulnerabilities like we see today. But:
But worse; ultimately all our systems work in the real world, and physics doesn’t respect our security boundaries. You can sniff CPU activity by listening to the sounds of a CPU fan even today.10 What could an AI do in 10 years’ time, equipped with a broad range of sensors? On this view, if the AI can perform arbitrarily complex physical interrogations of a system, ultimately there is no defence.
This is a restatement of the AI doomers’ viewpoint from a pure cybersecurity perspective — once AI is loose in the world, certain battles are already lost.
I’m not sure I believe that either though. I see two competing exponential growth curves here; one for the attacker, one for the defender. Which dominates long-term? The only honest answer is “I don’t know”. I’m very mistrustful of the “golden age of security” view; I think that involves radically underestimating the breadth of the risk landscape. I maybe object a bit less to the doomer view which at least shows more imagination.
But I think both sides of the argument underestimate the incentives involved and the power of negative feedback loops. Cybersecurity was already an arms race; I think AI massively accelerates the arms race on both sides, but wherever defenders or attackers gain an advantage, there will be resources and capital thrown at the other side, and very often, that means that you end up with some dynamic equilibrium balancing out.
I do think the advantage lies with the attackers for the next few years;11 I do think that in today’s sense “software cybersecurity” will be solved on the same timescale. There’s an enormous amount of work to be done in securing our existing systems, and there will be a lot of money lost to criminals12 as we play catch-up. We will win this software security battle in the end. But the war is just going to migrate to different battlefields; we will have to keep on fighting.