Bob Mottram isn’t impressed by the Singularity Institute’s grant proposals for our $100,000 Singularity Research Challenge:

It’s kind of sad how SIAI seems to have become obsessed with “AI risks” and human extinction. Perhaps they always were from the beginning, but it’s just my perception of them that was at fault. There’s certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.

The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.

To clarify what is happening, what Bob Mottram considers “spurious notions of risk”, we consider “deadly serious notions of risk”, so this is the main source of disagreement. Here was my response:

Our Uncertain Future project is pioneering probabilistic futurism in AI and WBE studies, and has received thumbs-up from several academics including Bela Nagy, who manages the Santa Fe Institute Performance Curve Database.

A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider “nice” or “common sense” is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values. See the Fun Theory sequence on Less Wrong, for instance.

SIAI believes that AGI is an extremely difficult endeavor and deserves far more theory-level work than programming in the dark or working towards narrow AI tasks that drain away our attention at the expense of the Singularity itself.

Basically, if you consider an intelligence explosion plausible, SIAI’s activities make sense, and if you don’t, they don’t. It’s not a matter of marketing, just disagreement on which tasks are the most important for humanity to face right now. We consider clarifying decision theory and creating a reflective decision theory to be a major priority, for instance, and spend time on that accordingly.

To clarify further, in 2009 SIAI grew large enough to break into several loose divisions. This is excellent, because the Singularity Institute is one of the most important organizations on the planet and is one of the only barriers standing between humanity and extinction from unFriendly AI. However, it makes the task of explaining what we do all the more complicated. It so happens that I am paid to explain it, but sometimes I get discouraged because I discuss the organization constantly on this blog, occasionally several posts per day, and there is still a great amount of confusion about what our organization does and believes. Perhaps I ought to run an SIAI Video Q&A in the vein of Eliezer’s recent Less Wrong Q&A.

What happened in the last year is that Anna Salamon and Steve Rayhawk joined us and created the Visiting Fellows program, under Anna’s leadership. (Anna, Steve, and myself were only recently added to our staff page.) This entity is only peripherally related to SIAI’s central AI project, which was more or less put on hold for two years while Eliezer Yudkowsky wrote the Less Wrong sequences. As our 2009 accomplishments document states, Eliezer worked with Marcello Herreshoff (his profile can be found here) on Friendly AI over the Summer.

So, think of SIAI as having three branches — 1. Administrative/PR, which consists of President Michael Vassar, Media Director Michael Anissimov (aka me), and Chief Compliance Officer Amy Willey, 2. AGI research that constitutes serious progress towards seed AI, which makes up years of Eliezer’s past work, Eliezer’s future work after he finishes putting together his rationality book, Marcello Herreshoff’s intermittent work, and contributions from Peter de Blanc, Nick Hay, and others (since 2006), including Anna Salamon and Steve Rayhawk, and 3. the Visiting Fellows Program, which including Visiting Fellows and various volunteers.

The goal of the Visiting Fellows Program is to put together extremely smart people concerned about reducing existential risk and have them pursue academic projects that make the best use of their respective strengths. Branch #3 also serves as a filtering mechanism for 2. The thing is, starting a true AGI project would be very expensive, not so much in money but in terms of intelligence, philosophy, computer science, and math knowledge required. Consolidating the necessary personnel will not be easy.

Why are we concerned about “AI risks” and human extinction? Well, this is why, among other reasons. SIAI is not about pursuing intermediate AI commercial benefits — our organization only exists to pursue the Singularity and minimize AI risk. Writing illustrating this point has been produced in substantial quantities since our founding in 2000. SIAI is mostly a bunch of utilitarians.

Would readers be interested in a Lulu book putting together a lot of information about the Singularity Institute in one place? Only about 0.5% of my blog readers ever comment, so I feel like I’m talking to a vast sea of silent lurkers all the time. Seriously, it’s weird.

In general, our approach turns off people like Bob Mottram, but inspires praise from people like Alan Darwst. In particular, Mr. Darwst writes:

Among the utilitarians I’ve met over the years, a sizable fraction have come to the conclusion that the optimal destination for utilitarian funding is organizations that research speculative futuristic scenarios and the philosophical / scientific / methodological questions that such research requires. In particular, many of these utilitarians have named the Singularity Institute for Artificial Intelligence (SIAI) as a good example of such an organization, so I’ll focus on it here, but the discussion can apply more broadly.

The way the Singularity goes is a matter of life and death for humanity. Unfriendly AI programmed to value anything besides a very specific set of Homo sapiens-characteristic values will probably overlook our material preservation. From the perspective of most possible minds, humans are just another arrangement of atoms. We don’t have any inherent moral value. “Moral value” is an “imaginary” thing that only exists among the tiny space of minds-in-general with explicit moral philosophies.

If we had the ability to build AGI today, our planet would not last the year, because we haven’t solved Friendly AI. If we could build a seed AI now, we wouldn’t know how to specify its goals in a way that doesn’t eliminate us all completely. We are clueless. We can’t create a utility function that is consistent under reflection and preserves individual humans when a tremendous amount of optimization pressure is applied to fulfilling it. We need a mathematical model of value that leaves us alive even when the unimaginable power of superintelligence is channeled into it. I think that Coherent Extrapolated Volition is a good enough solution that it would work, but it needs to be specified in much more detail. That’s exactly what one of our grant proposals is about.

These grant proposals deserve funding now. We are about to walk into a minefield and we don’t even have a map. We need to throw everything at the problem — people, money, attention, everything.