See Ben Goertzel’s summary of a discussion at the post-Singularity Summit workshop, Eleven Ways to Avoid an Extremely Bad Singularity. A great post, nice to see closer analysis of the issue after so many summaries and popular articles. (In the comments, Ben says: First, I just want to clarify one thing that was stated in my original H+ blog post (you know this, but it may not be clear to your readers): that blog post of mine was a summary of some ideas tossed around in a discussion in the post-Summit workshop … it wasn’t a disquisition on my own views, nor an endorsement of the ideas mentioned!) Here is my general reaction:

I consider 1, “Human-enforced fascism”, relatively unlikely but certainly possible, especially with molecular nanotechnology (MNT) or something analogous. With sufficiently fast and flexible MNT, a small group could probably take over the world. Luckily, very few people think that MNT is possible anytime soon and it may be that it isn’t possible period. I certainly hope so, because it would make everything unnecessarily complicated. I like the fact that I seem able to understand the broad outlines of a world without MNT.

“Friendly” AGI fascism, 2, is also a concern, but less than than the above, in my opinion. It might be extremely difficult to limit AGI intelligence to a certain level once it gets going. The scenario of someone creating a superintelligent AGI and somehow being able to control it I usually just call “magic”, and the respective user “magician”. This is probably impossible, and simply leads to a classic paperclip scenario.

3, “AGI and/or upload panspermia”, is more of an optional side action than a strategy for avoiding a bad Singularity. It basically involves sending AGIs or human uploads out in every direction as fast as you can. I don’t consider it a very good strategy for running away from an unfriendly AI because you eventually have to stop and colonize somewhere, and that’s when they get you. I also very much doubt that you will be able to build a spaceship that travels at near the speed of light without randomly exploding or bathing you in cosmic rays unless you yourself are extremely superintelligent and near-omniscient anyway. At that point you’ve already passed the last relevant hurdle between humanity and a beneficial Singularity. (Unless you yourself are malevolent or selfish.)

4 is creating a virtual AI world sandbox. Great idea, I just hope that programmers don’t get lazy because they think they have a “leakproof” situation. David Chalmers talked about this a lot in his talk at the Summit, and I think he’s simply wrong about the relative difficulty of containing and/or fooling an AGI. Only a fool tries to fool a superintelligence.

5 is “Build an oracular question-answering AGI system, not an autonomous AGI agent”. I doubt this would work — there is no solid difference between question-answering and taking real actions in the world. For instance, an AGI would need to be allotted some intelligence-gathering action-potential to answer questions. Preventing these actions from profoundly impacting the world would be difficult. This falls into the family of containment/magic scenarios whose plausibility is very speculative. Since the whole human race (including you) goes bye-bye if you mess up, it’s probably a bad idea to even think in these terms. Just Say No to Constructing Superintelligent AI Oracles.

6 is “Create upgraded human uploads or brain-enhanced humans first”. Good luck. Historical examples of individual humans getting ultra-powerful usually involve them going slightly crazy and conquering everything around them. We can actually talk about Friendly AI and come up with some consensus on what it should be, then implement it, but with intelligence enhancement you are entirely subject to the whims and beliefs of some random dude or gal. With Friendly AI you can make an entity that is almost entirely unbiased with respect to human cultures and institutions, with an enhanced human you can’t. There is no one I trust enough, including myself, to do this. The first substantially enhanced human intelligence is more likely to be the sort of person that can claw their way to the top of some transhumanist-engineer social group, not the type of person bred from birth to do it. (Which would be more appropriate, and still might fall short.) If we had to pick anyone to upgrade, I’d choose an emotionally stable 10-year-old boy or girl, because this is around as smart and developed humans get before being hit by a wave of pubescent cognitive changes, including lust, adult-like social competition, and more Machiavellian crap that makes people start thinking in zero-sum terms. Power and lust in men leads to rape, and rape leads to chaos. I don’t know how it works with women.

7 is CEV, Coherent Extrapolated Volition. I personally think that this is the best idea out there. I disagree with Ben that this isn’t well-definable in practice, though it certainly isn’t well-defined as it stands. It is clearly fleshed out enough that you could have someone (a team of AGI programmers) specify it in more detail until they have something. Still, would it be well-specified enough to work? It looks like a better idea than most others that have been proposed. The one-sentence version is, “In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.” Coding this as a goal system sounds like a pain in the ass in terms of the technical challenges, but perhaps necessary if we want to avoid scenario 6. It would be nice to have a toy system of this that actually worked on a limited scale. I have no idea what that would cost, probably a lot. (A lot of the costs would be creating an underlying intelligence powerful enough to model human preferences in detail and “interpret them as we wish that interpreted”.) Its value and convincingness would obviously scale with the underlying intelligence. Places like SecondLife would be the test grounds, and hopefully we would have lots of pre-human AGI behavior data before ascending to truly general intelligence.

8 is “Individual Extrapolated Volition” (IEV). Unfortunately, human beings are inherently social beings, so you have to model social interaction in the extrapolation process. Not a minute doesn’t go by for most of us that we aren’t submerged in human artifacts. Only a man naked in the wilderness is truly alone. The individual aspect is slightly appealing due to the fact that it would be apparently easier than CEV, but wouldn’t it be “stealing the Singularity” to have just one person’s volition extrapolated? I feel morally uncomfortable about the idea. I’m not sure whether it’s rational or irrational, but CEV “feels” better to me. Of course, if it’s technically much more difficult, we might be forced to pick IEV whether we like it or not.

9 is “Make a machine that puts everyone in their personal dream world” is what I call an “Evermore scenario” because it is the basis of the plot of the video game Secret of Evermore. It also happens in the truly terrifying P.K. Dick story Eye in the Sky. Anyone thinking about messing around with advanced AI should read the latter. In that book you get to see what would happen if fundamentalist Christians got their hands on the AI, as well as other scenarios. Basically, I think this is a bad idea. I think it should be strongly discouraged, in fact. Social connections and consensus and the need for stakeholder politics (to steal a line from Dale Carrico) could go away after the Singularity if we wanted them to, but that would lead to scenario #9, which I think is a soulless one. The only way to make politics go away would be to make the connections between people go away. You could do that if you surrounded yourself with puppets and zombies. After living in a world like this for a few years, a lot of people would probably simply forget about the real one and start “whacking off” (metaphorically or literally) for decades on end. Not a happy scenario. A beautiful world can’t be created out of one man’s imagination.

10 is “Engineer a very powerful nonhuman AGI that has a beneficial goal system”. Various people have various thoughts on this, which I have consolidated as a self-styled Friendly AI librarian. Feel free to mention any links not in that list. Essentially, while I think various people have good ideas, including Ben, Eliezer’s ideas really stand out for me in their rigor and plausibility. People might say, “of course you would say that, you work for SIAI”, but the thing is, if I started to change my mind, it would make a load of sense to talk about it, because without a good Friendly AI theory, we might all be screwed. I’d rather live to 1000 than please an intellectual figure or social group. Still, Eliezer has avoided many obvious pitfalls that practically everyone else seems to fall into. Ben’s concept of Voluntary Joyous Growth and other thoughts are interesting but even more poorly specified than CEV. Am I missing a major paper here? Ben also references Joyous Growth on SL4 here. I really hate mailing lists — why don’t people talk on blogs more so that there is more of a public record and discussion? Most mailing lists are not private anyway.

CEV is exciting to me because it “solves” the problem of morality by avoiding it almost entirely. Instead of trying to specify the necessary parameters for Friendliness acquisition directly (before I just said “a morality” here, which Ben pointed out was a strawman. Instead, now I am saying “90% of all non-careful Friendly AI programmers” would believe that directly programming the “necessary parameters for Friendliness acquisition” would work better than letting coherent human preferences dictate those parameters) it just points to human minds and says, “suck the morality out of that thing, then average it with others”. The approach of using an extrapolition algorithm inherently makes it difficult if not impossible to manipulate it in directions that you personally like, at the expense of other people’s goals. With most other goal systems, including Joyous Growth, it seems to rely on us trusting the programmer to provide good “basic values”. With CEV, the programmer is forced only to use what can be extracted from real human preferences. No one gets more than one “vote”. It is the most democratic scheme yet presented for an AI goal system, for those who get all excited by the word “democratic”. I think that some things are best done democratically and some aren’t. Providing goal system content for an AGI whose goal is to manifest our collective volition is definitely something best done democratically.

11, “Let humanity die the good death” through gradual intelligence enhancement seems like another one of those optional side issues and not really a way to avoid a bad Singularity. The way it is phrased here seems especially distracting and likely to confuse people new to this stuff. (More than they already are.)