To make everything as open and obvious as possible, I created a small boxes-connected-by-arrows chart to explain my beliefs on what the Singularity is about and what mankind should do about it:

We can call these nodes 1 through 8, reading left-to-right from the top.

What observations can we make right away? Well, it’s interesting how all the ideas at the top are relatively non-mainstream, non-widespread, controversial, and none of them are interdependent. You can have a hard takeoff without superintelligence, for instance, and seed AI without any of the other boxes. You can argue in favor or against any one of these boxes as a profession (if you’re a tenured philosopher), or just as a hobby.

Say we annihilate the box that says seed AI is likely before 2030. That partially ameliorates my concern/worry, but not really, because then I still have to worry about self-bootstrapping BCI-augmented humans and/or uploads.

However, there is one box that does contribute a lot to the concern/worry, and that’s the far right one, Box 4. In my original vision, there was no box 4, and there was no worry. I believed that any sufficiently intelligent agent would become friendly, discovering the “objective truth about morality”. That’s the present position of Peter Voss, unless I’m mistaken.

After what seemed like forever, the big picture in Box 4 was presented on Overcoming Bias in late January, but the pieces of this view have been floating around for over a decade. I forget how I picked it up originally, but I know that reading How the Mind Works by Steven Pinker helped. A particularly good presentation is given by Joshua Greene, director of the Moral Cognition Lab at Harvard, in his doctoral dissertation, “The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It”.

Where do the other boxes come from? Every box has dozens or hundreds of references I’ve absorbed over a decade of reading, but I can point out the salient ones for people in a hurry. Box 1 is sketched by the brain emulation arguments given in Ray Kurzweil’s The Singularity is Near, and a shorter version is given by Nick Bostrom in “How long until superintelligence?” This one doesn’t hugely matter — even if seed AI comes in 500 years, the impact is so enduring and absolute it’s worth putting attention towards now. Even if it comes about in 70 years (2080), that’s still roughly within my expected lifetime, taking into account life extension based on historical progress.

Box 2 comes mainly from the AI Advantage, which I originally encountered in Creating Friendly AI, which was subsequently reinforced by arguments in Levels of Organization in General Intelligence and dozens of other sources. Since last summer, myself and other SIAI volunteers/interns/employees have been building more detailed, flexible, academia-friendly models of the situation here, which I overviewed in a post late last month. These models accommodate both slow-as-mud takeoffs and near-instantaneous takeoffs depending on which parameters you set. Lots of interesting debate and thought will center around this box in the coming decade.

Box 3 is another fun and oddly controversial one. You’d think that after humanity being dethroned from its central place in the cosmos about ten times since the beginning of the Enlightenment, that would be enough for us to acknowledge that qualitative superintelligence is plausible, but apparently it isn’t. Pundits like J. Storrs Hall and others are able to look at humans, then look at animals, and say that a similar intelligence gulf couldn’t exist between us and another hypothetical being. I’ll surely be forced to argue the points in this box for years to come, but like box 4, and unlike boxes 1 and 2, I see this as a losing battle for the opposition, and thereby somewhat less interesting. I consider the incredulity around the plausibility of superintelligence to be a temporary and fragile thing — the right fictional exposition, whether in book or movie form, will destroy this anthropocentric conceit. “Understand” by Ted Chiang is a nice try, as is the much shorter and funnier “X17″ by Eliezer Yudkowsky.

Blah, down to my most hated box, the accursed Box 5. Eliminating box 5 by pursuing a solution is my present interim goal in life. Unfortunately, I have met people where all four boxes on top are present but they don’t lead to box 5. A number of reasons can be put forth for this, the most prominent being focused exclusively on your own life and close friends and not caring if all of humanity is snuffed out as long as it’s quick. Or perhaps a lack of emotional valence — if you have a long-time commitment to political activism, then being worried about self-improving AI is boring because it doesn’t invoke evpsych-derived obsessions with political intrigue. So, you avoid following your beliefs to their logical conclusions because the conclusions are too disturbing or have actionable implications that contradict your prior plans.

Box 5 is more open and easily observed while boxes 1-4 are not immediately obvious, leading some intrepid Internets Psychologists to write in their own made-up ideas for the upper boxes. James Hughes has published a paper on his, and made a big deal about it at the 2007 Singularity Summit, which, by the way, received front-page coverage in my city newspaper (the Chronicle). Basically, because he doesn’t understand one or more of the top boxes, he can’t imagine how box 5 could derive from boxes 1-4, so he makes up his own boxes that seem to make more sense as sources to generate concern, like run amok Millennialism. I’m not sure how to feel about this. Sort of bored, really, because it discourages debates about boxes 1-4.

Box 6 happens when the worry and concern temporarily abates and you actually think about what to do. All sorts of fun ideas can emerge from there, and many of them have never been published. My brain contains a large catalog. Make up your own. When I see people spontaneously generating plans here, I see that they finally understand my point. More and more such plans (usually in the form of solutions to Friendly AI) have been popping up here in the last decade, some by Ph.Ds like Tim Freeman and Matt Mahoney, and some by cranks, like Arthur T. Murray, who plan to have the Singularity all wrapped in time for 2012. Discussions with other singularitarians made me originally realize that the space of possible actions is quite huge, if you only pause to think about it. Some cranks have inevitably proposed answers like “destroy everything”. Others have proposed government regulation, which is silly because no legislature will consider superintelligence plausible prior to its creation.

Box 7 is the current main plan, embodied by the Singularity Institute and all the support behind it. I like the general idea, but it must be emphasized that it is entirely incomplete and needs more work immediately. My support for the plan may be withdrawn at any time, based on how it evolves. Other contributors to the SL4 and AGI lists have come up with specific implementation plans for this, but the most interesting ideas (in my view) come from the 30-odd person, mathematics-oriented community of SIAI interns and volunteers. This is composed of names you may remember from the peanut gallery on Overcoming Bias, including Michael Vassar, Marcello Herreshoff, Anna Salamon, and many others. People like Matt Mahoney, Peter Voss, Bill Hibbard, J. Andrew Rogers, Pei Wang, Jürgen Schmidhuber, Marcus Hutter, Steve Omohundro, Moshe Looks, Richard Loosemore, Tim Freeman, and a small set of others (which can be found lurking on Ben Goertzel’s AGI list) offer interesting counterpoints here. Ray Kurzweil’s solution, “the free markets will do it!” demonstrate that he lost box 4 somewhere along the way. Maybe he will find it eventually.

Box 8 involves coming up with some way to enhance human intelligence as a stepping stone to the long-term fix of Friendly AI. Originally I just dismissed this idea out of hand, based on the biological complexity of human minds, the preexisting optimization conducted by evolutionary processes, difficulty of securing funding and government approval, difficulty of noninvasive testing procedures, etc. Today, I’m still extremely skeptical, but have become vaguely less skeptical due to smart folks presenting me with decent arguments. I have the feeling that lots of people are holding back some of their ideas on this because they don’t want to be seen in public discussing them, plus they think they might be valuable. I warn them that without a public discussion of possible engineering approaches, the community will languish due to insufficient exchange of ideas. You may think that you and your 6 smart friends are a sufficient group to discuss it with, but believe me, your little group is not the only one thinking about it in a serious, writing-up-tentative blueprints way. There are at least 20 more where that came from, and all you little cells staying quiet is just delaying progress by years.