This is a summary of an argument made in a recent book, If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares. Yes, they mean it literally.
If you’re interested in their argument, there are obvious reasons why you may prefer to read the book rather than this post. First, this is a very condensed version, without much support for the individual claims. Second, the book is written by significantly more capable writers than this post is.1
I encourage you to do so, especially if you don’t find the argument as presented here convincing. You may well remain unconvinced, but I would feel amiss if what made you dismiss it was this short summary.
That said, below is my attempt at distilling their thesis.2
Humanity’s Special Power
Intelligence may be understood as the ability to predict the world and steer it toward chosen outcomes; different intelligent entities may choose to steer the world toward different destinations.
Humans are the most intelligent species and can apply their intelligence to the widest range of tasks, hence they are currently the most powerful species.
But this will not last forever, as the level of human intelligence is naturally limited by the biology of human brains. Machine intelligence will not be subject to these limitations and will eventually surpass human intelligence by a wide margin.
Grown, Not Crafted
Creating AI models doesn’t require engineers to specify or understand how the AI achieves the task it’s trained for. The modern AIs consist of billions of numbers (weights), and understanding the cognitive processes they use during computation is extremely difficult and, at present, impossible.
Learning to Want
Smart AIs will act like they want things. This doesn’t require them to have feelings or emotions: AI “wants” something in the sense that its actions clearly demonstrate it is trying to achieve it.
AI will develop wants because they are useful for the tasks it is trained for. If an AI is trained to write useful computer programs, it is likely to perform better if it develops a desire to write code without bugs. We would observe that it always runs a syntax check and other tests before finishing the response: it acts as if it wanted to write correct code.
You Don’t Get What You Train For
We don’t know how to reliably create AIs that want exactly what we would like them to want3. We can’t predict what wants the AI will develop. In fact, we can’t even guarantee that the AI’s wants will align with the task it was trained for.
The AI could achieve what it was trained to do in ways its developers did not foresee. For example, when an AI was trained to play a boat racing computer game, the engineers gave it a goal to maximize its score, which was intended to make it learn to finish the game fast. Instead, the AI learned to run in circles, hitting turbo blocks —increasing its score but never finishing the game.
As another example, when researchers asked reasoning models to win in a chess game, the models would often rewrite the game state or replace the opponent with a weaker version. The model wanted to get to the winning state by all available means, not win in a regular chess match.
Its Favorite Things
Whatever goals and wants the machine superintelligence develops, most of them are unlikely to be best achieved by keeping humans alive and happy. And as discussed above, we can’t reliably make the AI want to keep humans alive and happy.
We’d Lose
Superintelligent AI would have many ways to interact with the world. The more options the AI has to influence the world, the more it can be useful to us: AIs today already talk to millions of people each day, browse the internet, and write computer programs. One easily imaginable way to achieve its goals in the world would be to persuade people to do its bidding.
Humanity would not be able to fight against a machine superintelligence that doesn’t want to keep people around. It would beat us by exploiting reality in ways we couldn’t defend against, predict, or maybe even understand in retrospect. That’s what happens when groups with different levels of technological capabilities fight.
That’s about it! As stated above, you may consider reading the book or the relevant chapters to learn more.
All words in this article were generated by yours truly, barring Grammarly auto-correct.
The headers correspond to chapter names from the book, all from Part 1: Nonhuman Minds.
In other words, AIs are not wont to want what we want them to want.