It’s safe to say that most human ideas on personal (non)identity are confused.
It’s presumably safe to say that the ideas of a SuperIntelligence won’t be confused.
Maybe a SuperIntelligence will be ultra-Parfittian [ see Derek Parfit's "Reasons and Persons": http://en.wikipedia.org/wiki/Reasons_and_Persons ] and do without any notion of an enduring metaphysical ego altogether.

Either way, might not the term “human-friendliness” be question-begging?
For example, we recognize it is extremely friendly to help toddlers develop into something radically different i.e. mature adult human beings. So is it “unfriendly” for an AGI to help to help adult humans develop into something radically different too i.e. mature post-human superbeings? Admittedly one trouble here is that the idea of becoming “post-human” is a little outside our comfort zone. Would a post-human really be “me”? But is my ancestral toddler namesake really “me” either? Today one often-invoked criterion of personal identity is “memory”. But if, for example, posthuman consciousness is naturally richer than human peak experiences, why would a post-human want to remember such ancestral states? By analogy, would you want to remember what it was like last time you were exceedingly bored?
Also, recall how morally queasy we are about cases where, say, congenitally deaf parents want to have a congenitally deaf child – so-called “elective disability”. By analogy, why should a “friendly” SuperIntelligence be constrained to (re)create dumb and malaise-ridden human beings when our matter and energy could be reprogrammed into, for example, blissfully intelligent post-human “smart angels” instead?
A final point. History suggests that humans have a strong genetic predisposition to behave in extraordinarily unpleasant and unfriendly ways both to each other and members of other species. A proposed human-friendly AI might “lock in” our nasty genetic malware for ever under the aegis of this supposed Superintelligence. By contrast, a true SuperIntelligence would presumably be able to transcend anthropocentric biases and grasp all possible perspectives. At the very least, IMO, an impartial friendliness to all sentient beings dictates changing human nature so it no longer reflects the genetic inclusive fitness of one species of African ape. A failure adequately to grasp all possible perspectives is arguably as much a cognitive as a moral limitation – and one I think we should all strive to overcome.

Say that you place some amount of probability on a hard takeoff from the first superintelligence, say 5%.

Say that you aren’t sure that the superintelligence will lead to a hard takeoff, but to be “conservative”, you assume that it will, so you take as many precautions as you can.

You nominally have two choices: AI or IA?

I tentatively welcome either transition as long as the first superintelligence has human interests deeply in mind.

I so happen to think that AI superintelligence is probably easier than IA superintelligence, so it is in my best interest to maximize the probability that said AI superintelligence at least starts off human-friendly.

Even if we have no long-term control, we have control over the starting point.

I applaud anyone who is interested in making human-friendly IA superintelligence, but I don’t see that strong a movement in that direction, currently. Many people in SIAI are interested in IA and keep a close eye on it, so it’s the best place to be for those concerned about both IA and AI superintelligence.

Maybe the “Code” will fail, and will lead to our destruction. The goal of the Friendly AI movement is to increase our understanding as much as possible and promote the creation of seed AI with human-friendly initial motivations. If there is some cosmic force that automatically transforms human-friendly motivations into human-unfriendly motivations during the self-improvement process, then we are doomed either way. But, if human-friendly motivations give rise to self-modification choices that preserve the human-friendly utility function, then we will be in good shape.

The question boils down to: which would you rather have the first superintelligence be, AI or IA? Either you think the question doesn’t matter all that much, or you may have some preference. My preference is for AI, for a lot of reasons, but it’s unfair to imply that we are traitors to the human race just because we are working towards an AI Singularity. The very reason we want an AI Singularity to begin with is that we consider it the easiest way to preserve human values across the transition.

The background perspective for all of this is Bostrom’s Future of Human Evolution paper.

Your proposal is that if you start with “kernel code” that is as selfless as a worker drone or iPhone app that it will remain so when it becomes godlike.

I don’t buy it, and neither do most other people who hear the idea.

Why not? We know why humans get more selfish when they get power: because humans are programmed to pass on their genes, be subservient when weak, and destroy their enemies when they have the chance. Hence beta males in chimp clans sometimes band together and kill the alpha. The “more power = more selfishness” connection makes sense for Darwinian organisms, but we have specific mental routines that drive this behavior. Why do you think these mental routines would emerge de novo in an AI specifically uninterested in them? Wouldn’t they have to be deliberately programmed in? Otherwise, where would they come from? Remember that the AI has complete control over its own source code — it can enforce tyrannical control over its own mental content. Do you think it would just sublimely slip into another state of mind without even knowing it?

Are you familiar with the idea of the Blank Slate? I think that you, Mike Treder, and some others in H+ might have a Blank Slate view of intelligence, where mental properties unique to human minds are assumed to be properties of minds-in-general.

Another disagreement of ours seems to be around the ethics of building a selfless superintelligent Transition Guide to begin with. We don’t see it as ethically troublesome to build a selfless superintelligence, but you seem to imply that it’s both 1) unethical, and 2) extremely difficult. If it’s so difficult as to be impossible, why bother with condemning it ethically? Please clarify.

If your example of a purely selfless creature is a worker drone then we are indeed talking past one another on several levels.

I do believe it is possible for their to be expert systems which facilitate human communication and decision-making without imposing any goals of their own.

I do not believe that is what is intended when your group talks about “artificial general intelligence” which is supposed to be not only self-aware at a human level, but inconceivably more complex and powerful.

Your proposal is that if you start with “kernel code” that is as selfless as a worker drone or iPhone app that it will remain so when it becomes godlike.

I don’t buy it, and neither do most other people who hear the idea. It is, as I’ve said, a form of displaced religious faith in the purity and immutability of good code. In the beginning was the Code, and the Code was good…

Second, we have never to my knowledge disallowed a comment on the IEET site from Michael Anissimov. If a comment he submitted did not get through, it’s more likely a captcha problem than a moderation issue.

Third, we do practice firm moderating of comments and do so unapologetically. Although we are happy to engage alternative points of view, we draw the line when commenters are: a) blatantly irrational or anti-scientific, like those who spew global warming denialist nonsense; b) unproductively argumentative, acting like trolls as opposed to engaging in constructive debate; or c) engaging in personal attacks, name-calling, being disrespectful, etc.

I hope that makes clear how we conduct our business.