I recently heard from a colleague at SIAI that a major potential donor had visited the Visiting Fellows Program and was considering whether to fund any of our grant proposals. Some email initially gave me the impression that he was going to argue that we ought to spend more time directly on AI research and less on publishing academic papers, but I was surprised to learn it was the opposite. He said that we were too small to get very far with AI research on our own (our annual budget is in the $500,000/year range, and our finance information for past years is available on Guidestar), so it makes sense to put a lot of focus on academic papers and idea promotion, or so he argued.

I bring this up because I am sympathetic to SIAI supporters who argue that AGI research is a must. However, the constant question is, “does this research contribute to AGI in general or specifically Friendly AI?” We really want the latter, and really don’t want the former. (According to our current views, most AGI designs would lack the necessary values to be human-friendly and would probably view human beings as raw materials to achieve other ends. Engineering human-friendliness would require special, dedicated effort.)

A major AGI project would be expensive. I believe Eliezer Yudkowsky once said that $5M/year would be a reasonable number for such a project in its mid-stages. I’m not sure if $5M/year would be necessary to start (probably not), but certainly more than $200K a year would make sense. Thinking of AGI as a mighty pyramid, it seems fruitless to build it on such a weak foundation. Others may disagree, but I should point out that selection effects at the present time dictate that every AGI project is necessarily a small one. (Except maybe Numenta, if you consider that AGI.) Therefore, every AGI designer currently working on AGI believes that AGI can be done on a small budget.

Instead of working exclusively on Friendly AI and wasting all of our money because we get only 0.1% of the way to the goal with our current financial and human resources, why not rally some support until we actually have a foundation worth building on? It makes sense to me. Some people are disappointed that Eliezer took the majority of two years off of Friendly AI research to write the Overcoming Bias/Less Wrong sequences and prepare his book, but I know that SIAI probably wouldn’t be at its current level of resources if he never did. Most people who have gone through many of the posts have been very impressed, and we’ve acquired a lot of new allies thanks to that. A lot of interesting decision theoretical investigations have taken place on Less Wrong which are explicitly AGI-relevant, and the site has created a larger community with some star members who are being cultivated as potential Friendly AI programmers. There might not have even been a Visiting Fellows Program if it weren’t for Less Wrong.

SIAI has proven its ability to take relatively small amounts of money and generate huge quantities of discussion, analysis, and attention around the Singularity and closely related ideas. In terms of publicity per dollar spent, I would think we are in the 99th percentile of effectiveness for non-profits. The existence of SIAI has contributed in a major way to the rising profile of the Singularity in recent years (along with Ray Kurzweil), and to a Singularitarian like myself, I consider that very important. We are spreading ideas, making connections, acquiring allies, and some of us are doing AGI-relevant research. Our 2009 accomplishments document lists the major items. As always, we are open to feedback, and I encourage you to email myself or Anna Salamon at our singinst.org addresses if you have any comments or questions.

According to my current understanding, launching a positive Singularity offers the greatest good for the greatest number at this point in history. It doesn’t seem to me that the Singularity will go well by default — I think it will go wrong by default. So, strong and immediate intervention is necessary. At this point in time, SIAI seems like the best organization for the job, and we have received endorsements from many dedicated utilitarians, such as Alan Darwst.