
Human Compatible summary
Artificial Intelligence and the Problem of Control
The book in one sentence
The usual way we build AI, by handing a machine a fixed goal to optimize, gets more dangerous as machines get smarter, so we should build machines that stay unsure of what humans want and learn our preferences by watching what we do.
Overview
Stuart Russell helped write the textbook most students use to learn AI, and here he sounds an alarm about his own field. The way we build smart machines, by handing them a fixed goal to chase, works fine while they stay dumb. Once they get smarter than us, that same setup turns dangerous.
Key idea 1: The Standard Model Is the Trap
For about seventy years, we have built artificial intelligence one way. First you define intelligence as the ability to reach a goal. Then you build a machine that looks at the world and picks the action that best reaches whatever goal you gave it. A thermostat does a tiny version of this. So does a chess program and the app that plans your drive home. Russell calls this the standard model, and for most of AI's history it has worked well. The smarter the machine gets, the better it hits the target you set.
Here is the catch. That setup is only safe when the machine is weak or the goal is exactly right, and the goal is almost never exactly right. We are bad at writing down everything we care about. A weak machine that misreads the goal just does a poor job, and you fix it. A very capable machine that misreads the goal can cause real harm, because it will chase the target you actually typed with all the skill it has.
Russell reaches back to Norbert Wiener, who saw this coming in 1960. Wiener warned that once we build a machine we cannot easily stop, we had better be sure the purpose we put into it is the purpose we truly want. Picture telling a powerful system to fix the ocean's rising acidity. A fast, cheap way to lower the acid could strip oxygen out of the air as a side effect. You asked it to fix the acid. You never said keep the atmosphere breathable, because to you that goes without saying.
The danger here has nothing to do with a machine that hates us. The problem is a capable machine aimed at a goal that leaves something out. The same shape shows up in ordinary software already. A feed built to maximize clicks will happily make you miserable if misery keeps you clicking. So when you hand any capable system an objective, expect it to reach that exact objective and skip everything you forgot to say.
7 more key ideas in the app
- 2. The Gorilla Problem
- 3. King Midas and the Wrong Wish
- 4. A Goal-Driven Machine Won't Let You Stop It
- 5. Three Rules for Machines That Actually Help
- 6. Doubt Is What Makes the Off Switch Work
- 7. Reading Our Wishes From What We Do
- 8. The Harm Already Under Way
Did you know?
- Russell is not an outside critic of AI. He co-wrote 'Artificial Intelligence: A Modern Approach' with Peter Norvig, the standard AI textbook used in more than a thousand universities around the world.
- Russell has pushed against autonomous weapons in the real world. He helped create the 2017 short film 'Slaughterbots' to warn about cheap killer drones, and he founded the Center for Human-Compatible AI at UC Berkeley.
Life 3.0
Pale Blue Dot
Reality Is Not What It Seems
Sapiens
Seven Brief Lessons on Physics