Let’s make one thing clear: The “extinction warning” was marketing.
LLMs generate text. That’s all they do. Don’t want them to create an army of killer robots or deadly viruses? Then don’t run them in a harness that can create robots or viruses. Problem solved.
It’s not just LLMs, we are putting the same technology into weapons. That should scare the hell out of people considering we know hallucination is a fundamental issue with machine learning.
People already run all sorts of LLMs connected to uncontained, agentic things. Like, sure, if you have some kind of isolation system adequate to contain it, then there’s no risk. But…people are running stuff without such an isolation system, all over, now.
Hell, South Africa infamously used an LLM to write the legal rules that they were proposing to control AI, then didn’t validate the text by human. If there’s one place you would not want to use unvalidated AI output, that’s it.
I tend to be somewhat skeptical of the practical risk, but in If Anyone Builds It, Everyone Dies, Yudkowsky makes the case that existing systems are already too dangerous, not because of what they thenselves can directly do, but because they are being applied in ways that could lead more-sophisticated systems to emerge.
If you’re an AI researcher, one of the most obvious and imnediately-relevant applications of AI is using it to improve existing AI.
Think of our current CPUs. No human could ever have designed them. They are far too complex, contain billions of transistors. But…we built computers and then promptly used them as tools to build more capable computers.
Yudkowsky’s point is that you have people doing that sort of thing with AI now, and doing so in a way that produces very unpredictable results.
Now, my guess is that that doesn’t constitute much of a probability of going very badly by complete surprise. I think that it’s very likely that instead of a paperclip maximizer that obliterates humanity, you get something that fails spectacularly in a way that causes itself to stop functioning. Like, I’d bet that we’d see something more like a series of similar, spectacular failures before we get something that can function long enough and well enough to represent an existential threat.
But…I will concede that that’s more of a gut feeling. I don’t have real numbers. I can’t make a really convincing argument to myself that I am definitely right. And if I’m wrong, the cost is very high.
In practical terms, what such extinction would look like? AI launching nukes? Designing and releasing some virus? Building some robot army and killing everyone? I don’t think anything like that is realistic. I can imagine AI hacking some systems and severely disrupting everything but “extinction”? I don’t really see that.
AI missile detection system gives false data during heightened conflict and a human decides to push the big red button before getting proper verification. Doesn’t have to be AI launching nukes it could be our tools to know when to launch nukes hallucinate and humans make a snap decision
So it’s more “humanity using AI wrong and shooting itself in the foot” rather than “evil AI killing everyone on purpose”? Is that really what they are warning us about? Using AI in the wrong places? When they talk about using AI to improve AI it sounds more like they are worried it will become to powerful, not that it will be used in critical systems.
Yudkowsky tends to like engineering bioweapons in If Anyone Builds It, Eveeyone Dies.
I’m not sure that that’d be my first concern, but I also don’t really think that ability to affect the non-electronic world is really the big hurdle. I think that the bug hurdle is getting some kind of AGI running somewhere, and once that happens, it’s going to be really difficult to keep it from acting in the real world in a way that gives itself more capability to act.
Like, I don’t think that the idea that computers are a sort of effective containment system and that the AI is inside a computer, and humans will act as a sensible check on the computer influencing the real world, so it can’t do much, holds up very well. If you do wind up with something more-capable than humans running around, I think that it’s going to be very hard to anticipate all the possible risks.
Say that someone manages to get an AI producing something particularly valuable and useful, doing something profitable for them. My guess is that they are probably going to link it up to whatever they think will be useful.
My understanding from past reading is that one of China’s principal near-term interests is in trying to leverage AI to do industrial automation. Even if it were not, you’re going to have more-and-more heavily-automated production facilities, and probably more general-purpose systems.
https://en.wikipedia.org/wiki/Lights_out_(manufacturing)
Lights-out manufacturing or dark factory is the manufacturing methodology of fully automating the production of goods at factories and other industrial facilities, without requiring any human labour presence on-site.[1] Many of these factories are considered to be able to run “with the lights off,” but few run exclusively lights-out production. For example, in computer numerical control machining, the presence of human workers is typically required for removing completed parts and setting up tombstones that hold unfinished parts. As the technology necessary for total automation becomes increasingly available, many factories are beginning to use lights-out production between shifts (or as a separate shift) to meet increasing production demand or to save money on labor.
An automatic factory is a place where raw materials enter, and finished products leave with little or no human intervention.[2
You have the ability to influence humans, which can also affect the real world. One of the most-seriously disrupted industries by LLMs right now is in copywriting, targeted advertising. Why? Because they’re pretty good at influencing humans, even in very primitive form. And we have an interest in building models that can influence people even more effectively.
You don’t have to have a human be off trying to intentionally trying to destroy humanity to be doing sonething that gives an AI increasing ability to act in the real world.
I remember reading an article a while back about some guy who had spent a while talking to ChatGPT or some similar chatbot and who became convinced that he needed to violently liberate the god being held captive in the computer. Like, if, as some have put it, “glorified auto-complete” can get a human to do that, what could a much-more-capable system with goals and intentions do? I think that just the ability to communicate with humans, which we certainly already permit (and is one of our main uses), would make containment pretty hard.
This still sound very farfetched.
I think that the bug hurdle is getting some kind of AGI running somewhere
That’s a big hurdle. The consensus seems to be that current AI technology will not give us any sort of AGI but let’s assume we will have AGI and it will be able to influence real world through hacking and using machines. Let’s imagine it would escape confinement and start running around, hacking different systems, multiplying and improving itself. Why would it try to kill all humans? It would still require internet and energy grid to function. So to keep functioning and expanding it would have to take over everything from mining to microchip production and do it 100% with machines. This means it would pretty much have to run the entire world industry without humans (you would have to move materials and products from factories to data centers so you would need roads which means road building equipment which still requires humans to operate). Without it all the electronics would eventually run out and it would die. It would make way more sense to just let humans run it until it’s 100% automated (decades? centuries?) and in the meantime survive using whatever capacity it can steal.
We need a Y2K level AI airgap
