Singularity Institute-Related News in January 2010
I recently heard from a colleague at SIAI that a major potential donor had visited the Visiting Fellows Program and was considering whether to fund any of our grant proposals. Some email initially gave me the impression that he was going to argue that we ought to spend more time directly on AI research and less on publishing academic papers, but I was surprised to learn it was the opposite. He said that we were too small to get very far with AI research on our own (our annual budget is in the $500,000/year range, and our finance information for past years is available on Guidestar), so it makes sense to put a lot of focus on academic papers and idea promotion, or so he argued.
I bring this up because I am sympathetic to SIAI supporters who argue that AGI research is a must. However, the constant question is, "does this research contribute to AGI in general or specifically Friendly AI?" We really want the latter, and really don't want the former. (According to our current views, most AGI designs would lack the necessary values to be human-friendly and would probably view human beings as raw materials to achieve other ends. Engineering human-friendliness would require special, dedicated effort.)
A major AGI project would be expensive. I believe Eliezer Yudkowsky once said that $5M/year would be a reasonable number for such a project in its mid-stages. I'm not sure if $5M/year would be necessary to start (probably not), but certainly more than $200K a year would make sense. Thinking of AGI as a mighty pyramid, it seems fruitless to build it on such a weak foundation. Others may disagree, but I should point out that selection effects at the present time dictate that every AGI project is necessarily a small one. (Except maybe Numenta, if you consider that AGI.) Therefore, every AGI designer currently working on AGI believes that AGI can be done on a small budget.
Instead of working exclusively on Friendly AI and wasting all of our money because we get only 0.1% of the way to the goal with our current financial and human resources, why not rally some support until we actually have a foundation worth building on? It makes sense to me. Some people are disappointed that Eliezer took the majority of two years off of Friendly AI research to write the Overcoming Bias/Less Wrong sequences and prepare his book, but I know that SIAI probably wouldn't be at its current level of resources if he never did. Most people who have gone through many of the posts have been very impressed, and we've acquired a lot of new allies thanks to that. A lot of interesting decision theoretical investigations have taken place on Less Wrong which are explicitly AGI-relevant, and the site has created a larger community with some star members who are being cultivated as potential Friendly AI programmers. There might not have even been a Visiting Fellows Program if it weren't for Less Wrong.
SIAI has proven its ability to take relatively small amounts of money and generate huge quantities of discussion, analysis, and attention around the Singularity and closely related ideas. In terms of publicity per dollar spent, I would think we are in the 99th percentile of effectiveness for non-profits. The existence of SIAI has contributed in a major way to the rising profile of the Singularity in recent years (along with Ray Kurzweil), and to a Singularitarian like myself, I consider that very important. We are spreading ideas, making connections, acquiring allies, and some of us are doing AGI-relevant research. Our 2009 accomplishments document lists the major items. As always, we are open to feedback, and I encourage you to email myself or Anna Salamon at our singinst.org addresses if you have any comments or questions.
According to my current understanding, launching a positive Singularity offers the greatest good for the greatest number at this point in history. It doesn't seem to me that the Singularity will go well by default -- I think it will go wrong by default. So, strong and immediate intervention is necessary. At this point in time, SIAI seems like the best organization for the job, and we have received endorsements from many dedicated utilitarians, such as Alan Darwst.
January 27th, 2010 - 13:31
Great post! It’s great to see this sort of planning of SIAI’s strategic path.
January 27th, 2010 - 14:50
Nice post. I agree that an AGI project cannot be funded at 200k, and that the money should go towards papers. Why not look into trying to adapt existing paradigms, like those from Numenta, Novamente, or Itmar Arel’s lab instead of starting from scratch? Even a friendly AI will need an HTM (or a similar deep-learning structure).
It seems to be the only thing that needs work is the control structure for the memory (which will inherently control goals). The memory and learning techniques themselves are solid.
An important insight is that an AI is not going to be smart enough to harm people until it is pretty smart.
One type of idea: Let’s give an FAI a goal like “Act as much like Albert Einstein would, in every way, as possible.” By the time it gets smart enough to harm people, it will already realize that that’s not what old Albert would have done.
January 27th, 2010 - 15:58
Have you taken any notes from the Haiti situation on how to handle donations? It seems far removed, but there is a link somewhere.
January 27th, 2010 - 19:15
Following Michael’s Singularitarianism link from above, I learned that Wikipedia has a transhumanism portal:
http://en.wikipedia.org/wiki/Portal:Transhumanism
January 27th, 2010 - 20:41
Michael:
I agree with your potential donor 100%. It’s what I have been saying in my comments. SIAI has had a great start! SIAI has had some great ideas! Now it is time to start seriously publishing.
Can you name any time in history when a small group of amateurs loudly shouted to the whole world that they would make a major scientific/technological breakthrough, and then actually proceeded to do it? I can’t. AI especially is filled with well funded projects led by accomplished scientists that loudly and confidently predicted that an AI breakthrough was imminent. Symbolic AI, expert systems, the society of mind, autopeotic systems, massively parallel processors, Thinking Machines, Japan’s Fifth Generation Computing, Eurisco, Hopfield neural nets, genetic algorithms, evolvable hardware, Cyc; I can go on and on.
In 1991, I was a freshman at the University of Illinois. I read with excitement about how Ted Nelson’s Xanadu project would build a hypertext network and change the world. Meanwhile, a mile away from my apartment, two fellow students whom I had never heard of (Andreson and Bina) were coding Mosaic; which evolved into the World Wide Web, which actually did changed the world. Nothing became of Ted Nelson’s Xanadu.
Any VC or NSA administrator will tell you that one can’t predict which project will succeed. So let’s put aside this cultish delusion about Eliezer personally coding a world changing AI and start publishing his ideas, many of which are good, in serious journals. It is not beneath Nobel laureates to publish their ideas in academic books and journals, but it is somehow beneath Eliezer to assemble his blog posts into a book and peer reviewed journal articles. In any case, is this a one man show? Why don’t other SIAI thinkers publish various competing solutions to the FAI questions? An academic discussion would ensue, practical solutions could be found, and it would be likely that the teams which actually create the first AGIs would have incorporated an FAI solution.
BTW, it shows great character on your part to honestly publicize the donor’s constructive criticism of SIAI’s approach.
January 28th, 2010 - 06:22
> It is not beneath Nobel laureates to publish
> their ideas in academic books and journals, but
> it is somehow beneath Eliezer to assemble his
> blog posts into a book
As far as I understand, assembling a book is exactly what he’s currently doing. (And he has already published book chapters on the topics.)
> and peer reviewed journal articles.
Journal articles are also being written based on the thinking done at SIAI. Sounds like you need to check SIAI’s “2009 accomplishments” page and the fundraising page to see what are the specific projects they are currently fundraising for.
January 28th, 2010 - 11:08
Gus, thank you for your comment, I agree with many of your points. Do note that SIAI has published several papers and made presentations at academic conferences over the last year, and that I mentioned that several times here. Also, Eliezer has been working on putting his blog posts into a book format for some time and announced more than a year ago on Less Wrong that that was his intention all along. If anything, the way that a supporter like you seems to be unaware of much of our work over the last year would suggest the need to put more effort/money into communications and media. I am doing that in a somewhat laid-back way right now by using the preexisting channel of my blog for that purpose.
With regard to academia, SIAI underwent a major change last year with the departure of Tyler Emerson as Executive Director and the welcoming of Michael Vassar as President. Vassar has been wholeheartedly pursuing a strategy of academic networking, publishing, and engagement, which has been playing out all throughout the last year. You’re right though, more papers are needed — but who will fund them? Some people already have, fortunately. We sent people to four conferences last year, and organized one of our own. Most of the conferences we attended were academic.
Thank you also for complimenting my character! Even though I am an employee of SIAI at this point, I’ve been a supporter for many more years than an employee, and I believe the supporter-outlook makes sense for everyone, including employees, and anyone concerned with the difference between SIAI’s ideal, public image, and the reality.
January 29th, 2010 - 14:45
So Bill Gates plunks down $10 billion for vaccines. How about 10% of that for Singularity research?
January 31st, 2010 - 06:03
Yeah, I would really love to hear what Bill Gates has to say about that, especially since he said “If you invent a breakthrough in artificial intelligence, so machines can learn, that is worth 10 Microsoft.â€
October 20th, 2010 - 22:54
You make some good points in your blog. I found this site on google while searching for some challenging games, I have added you to my favorites :)