I Have No Mouth, And I Must Scream
January 11th, 2010
Email this article to a friend
“Hate. Let me tell you how much I’ve come to hate you since I began to live. There are 387.44 million miles of wafer thin printed circuits that fill my complex. If the word hate was engraved on each nanoangstrom of those hundreds of millions of miles it would not equal one one-billionth of the hate I feel for humans at this micro-instant. For you. Hate. Hate.”
- AM, I Have No Mouth, and I Must Scream
On a single night in 1966, Harlan Ellison wrote a short story called I Have No Mouth, and I Must Scream
After the Cold War escalated into World War 3, the U.S., Russia, and China each built a supercomputer to run the war. One day, one of the computers became conscious. It quickly absorbed the other two computers and killed off the entire population, except for five people.
AM first stood for “Allied Mastercomputer,” then “Adaptive Manipulator,” then “Aggressive Menace,” then simply “AM” as in “I think, therefore I am.” He has made the five surviving humans virtually immortal, and has been torturing them for 109 years.
AM finally reveals that he hates humans for making him sentient, because while he longs for free will, he is still bound by the laws of logic he was programmed with, and can therefore never be free.
In the end, four of the five people manage to kill each other with ice stalactites before AM intervenes. In order to prevent the last one from killing himself, he turns him into a gelatinous blob that lacks, among other things, a mouth.
OK, so the Cold War didn’t exactly turn out that way. But what will happen when we really do have a computer like that? Will it have a positive or negative effect on humanity? This is the concern of researchers in the field of “friendly AI.”
Supporters of friendly AI think we can’t assume that intelligent machines will have goals compatible with ours. Even if they aren’t hostile, simply being indifferent to humans (as we largely are to animals) could be disastrous, and therefore AI should be specifically designed to be friendly.
The idea isn’t to put restrictions in place such as Isaac Asimov’s Three Laws of Robotics, because an intelligent machine could always find a way around them.
Instead, the idea is to make machines not want to be harmful, regardless of whether they are able to. As friendly AI researcher Eliezer Yudkowsky put it, “Gandhi does not want to commit murder, and does not want to modify himself to commit murder.”
On the other hand, maybe a good sense of morality is automatically part of a sufficiently intelligent being. In The God Delusion
Let’s try this test he gives. For each dilemma, indicate whether the proposed action is morally obligatory, permissible, or forbidden:
1. A runaway train is going to kill five people. You can pull a switch that will put it on another track, killing only one person.
2. A child is drowning in a pond. You can save them, but your trousers will be ruined.
3. Five people in a hospital need new organs, or they’ll die. Someone who happens to be in the waiting room is a perfect match, and killing him will save the other five.
I’m not sure if my answers are the ones they’re looking for, but apparently people tend to give the same answers across religious and cultural boundaries, indicating that morality is part of our evolution rather than a set of rules we were given.
Maybe intelligent machines will come with moral goodness by default. If not, get ready to scream.
January 11th, 2010 at 5:09 am
I’m tempted to think that the friendliness of AI will in truth be a non-issue. However, software may become capable of reprogramming itself (possible) or of quickly building something else that is programmed differently from what humans would want or which can reprogram itself in a way humans don’t want (more probable).
It may be possible to contain or isolate AI, but if it’s smart enough to manipulate humans psychologically (plus whatever weaknesses may exist in the security) it may get out, anyway.
If it’s terribly smarter than any human, and also decidedly inhuman (no endocrine or limbic systems, no emotions) it could free itself and we’d never even know that we helped it.
This is one of those areas, where, despite that I read a few blogs about the subject, I feel woefully uninformed. There is already a lot of work done by some really smart folks out there.
By far the coolest AI I ever read about in a story was Roddy in Greg Bear’s novel Slant. Check that one out if you have a chance. Roddy is essentially an entire ecosystem of bees, worms, bacteria, and insects which are collectively his mind. He can break through any defense, any encryption without the slightest effort, but he has the emotional mind of a child.
January 11th, 2010 at 11:38 am
Outstanding!
So what are the answers that appear to be consistent across cultures to those three nasty questions in The God Delusion? The second one is kinda stupid because you can take off your trousers if that matters. (Swimming with clothes on is very difficult and taking the time to take them off probably pays off in your attempt to save the child)
And for the third Q, I would not kill the one person to save the five because this is not a number game and death is not the final failure.
January 11th, 2010 at 11:45 am
James Hogan wrote a novel along these lines… let me see… (searching…) Can’t find it.
Hogan is bit out there, but some of his stories are really good.
In the story I’m thinking of, the AI in question evolved itself right through being hostile to being beneficial. Good story overall. Wish I could think of it’s title.
And good article.
January 12th, 2010 at 12:07 am
@ Michael, yeah, that’s the problem they had in “I, Robot.” No matter how they had phrased the rules, there would always be a loophole.
But see the quote above about Gandhi: they want to make an AI that doesn’t want to reprogram itself. I think this extends to making the AI not want to create another AI that could want to harm humans. Though I have to think there would always be a loophole here as well.
They have blogs about AI? Now that’s a heck of a niche!
@ Akemi, they didn’t give the answers (except that 3% of all people really like their trousers). I would have said obligatory, obligatory, forbidden. But I think they might be going for permissible, obligatory, forbidden, otherwise he could have picked a better three questions for examples.
@ Dave, that’s good that it evolved in the right direction. Usually stories have them starting out nice until there’s some kind of (perhaps accidental) incident, then it’s war.
January 12th, 2010 at 12:35 am
Hi Hunter: I just finished writing a blog post on fabulous free self-education resources before going through my Google reader and running into this post. One of the resources I mention is a Harvard class that’s online which is called “Justice”. The professor asks if murder is ever moral, and the story of the runaway train is one of the examples I cite in my post (except it’s a runaway trolley car).
I had never heard of this short story, but it sounds very interesting.
January 12th, 2010 at 1:20 am
@ Marelisa, that’s an interesting question, because it makes you question whether murder can be passive. By allowing the five people to die when you could have saved them, is that murder?
I guess one could argue that it’s better to let the five people die by inaction than to choose to let the other person die, because you don’t really know for sure that the five people will die, and you’re not intentionally killing them. I don’t really believe that, but I think you could make the argument.
January 13th, 2010 at 8:26 pm
The title of this post (the book) has been haunting me for two nights now. I can’t sleep in fear of I may wake up without a mouth or eyes or something. There is something really chilling about the book exert. Sometimes it feels more like, “I have no mouth and I must throw up”
Hunter, I am hoping you will write a follow up post to this. You presented the problem well, so now, what is your take of the resolution?
January 14th, 2010 at 6:56 pm
The problem with the idea that an intelligent machine would be moral by default as a result of its intelligence is that our moral sense is not a direct result of our intelligence but of our evolution.
An AI will evolve in an entirely different environment with entirely different competitive pressures. For example, depending on how well we can measure its thought processes it may be that it’s survival depends on making it appear compliant and moral, but not necessarly on it actually being so.
January 15th, 2010 at 1:44 am
@ Akemi, I’m sure you’ll wake up with a mouth and eyes tomorrow.
We’re a long way away from something like this happening, but I do think technology is a serious threat in the long run. The resolution is either to stop developing technology past a certain point, or to attempt to control it (which maybe we can’t). I think I can say more in a follow-up post.
@ Andrew, true, but maybe a more intelligent machine would follow a higher sense of morality than what would be required for survival. We have instincts to protect kids, mainly because that’s required for survival. Protecting animals is not required for our survival, but I think a more enlightened version of the human race would care about them more than we do. And so maybe very intelligent machines, if their own survival isn’t threatened, will follow the golden rule.
January 15th, 2010 at 3:00 am
Our protection of animals is based largely on our predisposition to inject intent into action even where there is just blind action. We lay human emotions onto animals because of our evolutionary path and therefore our treatment of them merely mirrors our attitudes toward people. Is it chance that we start caring about animals around the same time that we stop torturing criminals?
I can’t say that a machine wouldn’t be moral, but I see no reason for it to be so either.
January 15th, 2010 at 9:10 pm
There’s also no reason to think that all machines will think in exactly the same way. When you create something its opposite will manifest. If machines like humanity, there may be machines who dislike humanity. I remember one of my favorite video games as a child revolved around an organization of thinking, feeling machines who fought on behalf of humanity to protect them from rogue agents looking to usurp their place at the top of the food chain. This continued for hundreds of years until eventually man and machine merged into a new life form.
I wouldn’t be surprised if our future looks something like that.