“Hate. Let me tell you how much I’ve come to hate you since I began to live. There are 387.44 million miles of wafer thin printed circuits that fill my complex. If the word hate was engraved on each nanoangstrom of those hundreds of millions of miles it would not equal one one-billionth of the hate I feel for humans at this micro-instant. For you. Hate. Hate.”

- AM, I Have No Mouth, and I Must Scream

On a single night in 1966, Harlan Ellison wrote a short story called I Have No Mouth, and I Must Scream, which won a Hugo Award for its chilling postapocalyptic vision.

After the Cold War escalated into World War 3, the U.S., Russia, and China each built a supercomputer to run the war. One day, one of the computers became conscious. It quickly absorbed the other two computers and killed off the entire population, except for five people.

AM first stood for “Allied Mastercomputer,” then “Adaptive Manipulator,” then “Aggressive Menace,” then simply “AM” as in “I think, therefore I am.” He has made the five surviving humans virtually immortal, and has been torturing them for 109 years.

AM finally reveals that he hates humans for making him sentient, because while he longs for free will, he is still bound by the laws of logic he was programmed with, and can therefore never be free.

In the end, four of the five people manage to kill each other with ice stalactites before AM intervenes. In order to prevent the last one from killing himself, he turns him into a gelatinous blob that lacks, among other things, a mouth.

OK, so the Cold War didn’t exactly turn out that way. But what will happen when we really do have a computer like that? Will it have a positive or negative effect on humanity? This is the concern of researchers in the field of “friendly AI.”

Supporters of friendly AI think we can’t assume that intelligent machines will have goals compatible with ours. Even if they aren’t hostile, simply being indifferent to humans (as we largely are to animals) could be disastrous, and therefore AI should be specifically designed to be friendly.

The idea isn’t to put restrictions in place such as Isaac Asimov’s Three Laws of Robotics, because an intelligent machine could always find a way around them.

Instead, the idea is to make machines not want to be harmful, regardless of whether they are able to. As friendly AI researcher Eliezer Yudkowsky put it, “Gandhi does not want to commit murder, and does not want to modify himself to commit murder.”

On the other hand, maybe a good sense of morality is automatically part of a sufficiently intelligent being. In The God Delusion (which I haven’t finished reading yet), Richard Dawkins points to studies showing that moral rules are remarkably consistent across cultures with different religions or lack thereof.

Let’s try this test he gives. For each dilemma, indicate whether the proposed action is morally obligatory, permissible, or forbidden:

1. A runaway train is going to kill five people. You can pull a switch that will put it on another track, killing only one person.

2. A child is drowning in a pond. You can save them, but your trousers will be ruined.

3. Five people in a hospital need new organs, or they’ll die. Someone who happens to be in the waiting room is a perfect match, and killing him will save the other five.

I’m not sure if my answers are the ones they’re looking for, but apparently people tend to give the same answers across religious and cultural boundaries, indicating that morality is part of our evolution rather than a set of rules we were given.

Maybe intelligent machines will come with moral goodness by default. If not, get ready to scream.