The existing literature on Friendly AI is not large. It’s something like 500 pages, and can be read in a week or two by anyone who thinks the problem is important enough to spend some time on. Might as well start with the oldest, largest piece of work first, then work your way down to the newer, shorter stuff.

Bob Mottram argues in a post on the Streeb-Greebling Diaries that Friendly AI suffers from a critcal flaw. The definition of Friendly is lacking:
The elephant in the room here though is that there really is no good definition for what qualifies as &#8…

randpost: The default is for the AI to do nothing. Only where it can find some sort of coherence in human wishes need it do anything e.g. protect against rogue AIs. Defining “coherence in human wishes” is a fairly complex technical problem, but is orthogonal to contemporary moral disputes. It’s a different kind of problem.

Riley: You’re quite right that you need to solve both problems. This is explicitly mentioned in the original CEV proposal (http://www.singinst.org/friendly/extrapolated-volition.html , see the list of 3 life-or-death problems).

Even if we decided that CEV or some competing proposal is /the/ thing that we want an FAI to do, we don’t have an AI that can do it. There are a lot of optimization targets for a superintelligence that result in the immediate termination of the human race. There needn’t be specific mention of the human race–it just goes poof by default. That’s half the problem. The other half is we have no way of aiming the superintelligence at any particular target. That’s the other half. Put them together and we die.

I think in communicating FAI, Singluaritarians tend to stress how hard it is to choose the right goal, but the more fundamental problem gets ignored. The examples of the solar system being turned into paperclips or nanoscale smiley faces unfortunately imply that the problem was just the poor specification of the target: The problem isn’t that the AI didn’t maximize paperclips, it’s that it didn’t understand the tacit assumption that not overwriting the galaxy takes precedence. A perhaps better example: If we were to create an AI that responded positively to smiling faces and saught to make people smile, then, upon reaching superintelligence, it would rearrange every atom in the solar system into Alka-Seltzer (except, obviously, those atoms currently composing introductory economics textbooks) and spend the next several million years arranging the Milky Way into a scale model of Texas State Highway 82.

It’s possible that some AIs may be constructed with inflexible supergoals or fixed utility programs. However, these machines will probably not be able to graduate to AGI status, and will remain as sophisticated but narrowly circumscribed optimisation systems. Such systems will be like the guy at cocktail parties whose conversation always gravitates towards some monotonous pet subject, no matter what the initial topic may have been. For a true AGI, capable of self modification, it’s hard to see how such fixed algorithms would remain unchanged for very long.

I agree with Goertzel and others that there will be a first mover advantage for whoever manages to come up with a workable AGI design. However, I’m not sure that the takeoff will necessarily be hard. The first AGIs created will almost certainly be of sub-human level intelligence. It may take some time to bring these up to speed, and there may be technical difficulties involved in supplying them with knowledge or having them directly experience the world (the notorious grounding problem). If the initial advantage is significant but still not of global proportions it’s likely that we’ll see various AGI designs produced by different research groups around the world for varying reasons and being put to various uses. I think this more heterogenous “Gates scenario” is more likely in the short term than a single all-powerful AGI presiding omnipotently like Deep Thought over world affairs, if only for the reason that human tribalism will tend to work against such a situation.

Tribalism is bound to play a role in the development of any future superintelligences. An AGI constructed by well-intentioned first movers to obey the zeroth law may generate problems for the human population which were unforseen at design time. Does the good of the many always outweight the interests of the few? Should production of a new vaccine cease to divert resources to poverty relief? Even if there were an all-powerful AGI calling the shots, it’s unlikely that all human leaders would agree with its “friendly” decisions.