Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.

Friendly AI Discussion with James Hughes

James Hughes was gracious enough to post a comment on my recent post on the disagreements between pro-Friendly AI and anti-Friendly AI transhumanists, so I thought I would repost it here along with my response. Hughes said:

Michael

If your example of a purely selfless creature is a worker drone then we are indeed talking past one another on several levels.

I do believe it is possible for their to be expert systems which facilitate human communication and decision-making without imposing any goals of their own.

I do not believe that is what is intended when your group talks about “artificial general intelligence” which is supposed to be not only self-aware at a human level, but inconceivably more complex and powerful.

Your proposal is that if you start with “kernel code” that is as selfless as a worker drone or iPhone app that it will remain so when it becomes godlike.

I don’t buy it, and neither do most other people who hear the idea. It is, as I’ve said, a form of displaced religious faith in the purity and immutability of good code. In the beginning was the Code, and the Code was good…

Here is my response:

James,

Say that you place some amount of probability on a hard takeoff from the first superintelligence, say 5%.

Say that you aren’t sure that the superintelligence will lead to a hard takeoff, but to be “conservative”, you assume that it will, so you take as many precautions as you can.

You nominally have two choices: AI or IA?

I tentatively welcome either transition as long as the first superintelligence has human interests deeply in mind.

I so happen to think that AI superintelligence is probably easier than IA superintelligence, so it is in my best interest to maximize the probability that said AI superintelligence at least starts off human-friendly.

Even if we have no long-term control, we have control over the starting point.

I applaud anyone who is interested in making human-friendly IA superintelligence, but I don’t see that strong a movement in that direction, currently. Many people in SIAI are interested in IA and keep a close eye on it, so it’s the best place to be for those concerned about both IA and AI superintelligence.

Maybe the “Code” will fail, and will lead to our destruction. The goal of the Friendly AI movement is to increase our understanding as much as possible and promote the creation of seed AI with human-friendly initial motivations. If there is some cosmic force that automatically transforms human-friendly motivations into human-unfriendly motivations during the self-improvement process, then we are doomed either way. But, if human-friendly motivations give rise to self-modification choices that preserve the human-friendly utility function, then we will be in good shape.

The question boils down to: which would you rather have the first superintelligence be, AI or IA? Either you think the question doesn’t matter all that much, or you may have some preference. My preference is for AI, for a lot of reasons, but it’s unfair to imply that we are traitors to the human race just because we are working towards an AI Singularity. The very reason we want an AI Singularity to begin with is that we consider it the easiest way to preserve human values across the transition.

The background perspective for all of this is Bostrom’s "Future of Human Evolution" paper.

Your proposal is that if you start with “kernel code” that is as selfless as a worker drone or iPhone app that it will remain so when it becomes godlike.

I don’t buy it, and neither do most other people who hear the idea.

Why not? We know why humans get more selfish when they get power: because humans are programmed to pass on their genes, be subservient when weak, and destroy their enemies when they have the chance. Hence beta males in chimp clans sometimes band together and kill the alpha. The “more power = more selfishness” connection makes sense for Darwinian organisms, but we have specific mental routines that drive this behavior. Why do you think these mental routines would emerge de novo in an AI specifically uninterested in them? Wouldn’t they have to be deliberately programmed in? Otherwise, where would they come from? Remember that the AI has complete control over its own source code — it can enforce tyrannical control over its own mental content. Do you think it would just sublimely slip into another state of mind without even knowing it?

Are you familiar with the idea of the Blank Slate? I think that you, Mike Treder, and some others in H+ might have a Blank Slate view of intelligence, where mental properties unique to human minds are assumed to be properties of minds-in-general.

Another disagreement of ours seems to be around the ethics of building a selfless superintelligent Transition Guide to begin with. We don’t see it as ethically troublesome to build a selfless superintelligence, but you seem to imply that it’s both 1) unethical, and 2) extremely difficult. If it’s so difficult as to be impossible, why bother with condemning it ethically? Please clarify.