Comments on: I Have No Mouth, And I Must Scream

By: Illustro Cado

Illustro Cado — Sat, 16 Jan 2010 02:10:11 +0000

There’s also no reason to think that all machines will think in exactly the same way. When you create something its opposite will manifest. If machines like humanity, there may be machines who dislike humanity. I remember one of my favorite video games as a child revolved around an organization of thinking, feeling machines who fought on behalf of humanity to protect them from rogue agents looking to usurp their place at the top of the food chain. This continued for hundreds of years until eventually man and machine merged into a new life form.

I wouldn’t be surprised if our future looks something like that.

By: Andrew

Andrew — Fri, 15 Jan 2010 08:00:07 +0000

Our protection of animals is based largely on our predisposition to inject intent into action even where there is just blind action. We lay human emotions onto animals because of our evolutionary path and therefore our treatment of them merely mirrors our attitudes toward people. Is it chance that we start caring about animals around the same time that we stop torturing criminals?

I can’t say that a machine wouldn’t be moral, but I see no reason for it to be so either.

By: Hunter Nuttall

Hunter Nuttall — Fri, 15 Jan 2010 06:44:12 +0000

@ Akemi, I’m sure you’ll wake up with a mouth and eyes tomorrow. We’re a long way away from something like this happening, but I do think technology is a serious threat in the long run. The resolution is either to stop developing technology past a certain point, or to attempt to control it (which maybe we can’t). I think I can say more in a follow-up post.

@ Andrew, true, but maybe a more intelligent machine would follow a higher sense of morality than what would be required for survival. We have instincts to protect kids, mainly because that’s required for survival. Protecting animals is not required for our survival, but I think a more enlightened version of the human race would care about them more than we do. And so maybe very intelligent machines, if their own survival isn’t threatened, will follow the golden rule.

By: Andrew

Andrew — Thu, 14 Jan 2010 23:56:14 +0000

The problem with the idea that an intelligent machine would be moral by default as a result of its intelligence is that our moral sense is not a direct result of our intelligence but of our evolution.

An AI will evolve in an entirely different environment with entirely different competitive pressures. For example, depending on how well we can measure its thought processes it may be that it’s survival depends on making it appear compliant and moral, but not necessarly on it actually being so.

By: Akemi - Yes to Me

Akemi - Yes to Me — Thu, 14 Jan 2010 01:26:24 +0000

The title of this post (the book) has been haunting me for two nights now. I can’t sleep in fear of I may wake up without a mouth or eyes or something. There is something really chilling about the book exert. Sometimes it feels more like, “I have no mouth and I must throw up”

Hunter, I am hoping you will write a follow up post to this. You presented the problem well, so now, what is your take of the resolution?

By: Hunter Nuttall

Hunter Nuttall — Tue, 12 Jan 2010 06:20:42 +0000

@ Marelisa, that’s an interesting question, because it makes you question whether murder can be passive. By allowing the five people to die when you could have saved them, is that murder?

I guess one could argue that it’s better to let the five people die by inaction than to choose to let the other person die, because you don’t really know for sure that the five people will die, and you’re not intentionally killing them. I don’t really believe that, but I think you could make the argument.

By: Marelisa

Marelisa — Tue, 12 Jan 2010 05:35:19 +0000

Hi Hunter: I just finished writing a blog post on fabulous free self-education resources before going through my Google reader and running into this post. One of the resources I mention is a Harvard class that’s online which is called “Justice”. The professor asks if murder is ever moral, and the story of the runaway train is one of the examples I cite in my post (except it’s a runaway trolley car).

I had never heard of this short story, but it sounds very interesting.

By: Hunter Nuttall

Hunter Nuttall — Tue, 12 Jan 2010 05:07:43 +0000

@ Michael, yeah, that’s the problem they had in “I, Robot.” No matter how they had phrased the rules, there would always be a loophole.

But see the quote above about Gandhi: they want to make an AI that doesn’t want to reprogram itself. I think this extends to making the AI not want to create another AI that could want to harm humans. Though I have to think there would always be a loophole here as well.

They have blogs about AI? Now that’s a heck of a niche!

@ Akemi, they didn’t give the answers (except that 3% of all people really like their trousers). I would have said obligatory, obligatory, forbidden. But I think they might be going for permissible, obligatory, forbidden, otherwise he could have picked a better three questions for examples.

@ Dave, that’s good that it evolved in the right direction. Usually stories have them starting out nice until there’s some kind of (perhaps accidental) incident, then it’s war.

By: Dave Doolin

Dave Doolin — Mon, 11 Jan 2010 16:45:51 +0000

James Hogan wrote a novel along these lines… let me see… (searching…) Can’t find it.

Hogan is bit out there, but some of his stories are really good.

In the story I’m thinking of, the AI in question evolved itself right through being hostile to being beneficial. Good story overall. Wish I could think of it’s title.

And good article.

By: Akemi - Yes to Me

Akemi - Yes to Me — Mon, 11 Jan 2010 16:38:30 +0000

Outstanding!
So what are the answers that appear to be consistent across cultures to those three nasty questions in The God Delusion? The second one is kinda stupid because you can take off your trousers if that matters. (Swimming with clothes on is very difficult and taking the time to take them off probably pays off in your attempt to save the child)
And for the third Q, I would not kill the one person to save the five because this is not a number game and death is not the final failure.