Accelerating Future

Cross-posted from SIAI blog:

Thanks to generous contributions by our donors, we are only $11,840 away from fulfilling our $100,000 goal for the 2010 Singularity Research Challenge. For every dollar you contribute to SIAI, another dollar is contributed by our matching donors, who have pledged to match all contributions made before February 28th up to $100,000. That means that this Sunday is your final chance to donate for maximum impact.

Funds from the challenge campaign will be used to support all SIAI activities: our core staff, the Singularity Summit, the Visiting Fellows program, and more. Donors can earmark their funds for specific grant proposals, many of which are targeted towards academic paper-writing, or just contribute to our general fund. The grants system makes it easier to bring new researchers into the fold on a part-time basis, widening the pool of thinkers producing quality work on Artificial Intelligence risks and other topics relevant to SIAI’s interests. It also provides transparency so our donor community can directly evaluate the impact of their contributions.

Human-level and smarter Artificial Intelligence will likely have huge impacts on humanity, but only a tiny number of researchers are working to understand how to ensure those impacts are good ones. The role of the Singularity Institute is to fill that void, bringing scholarship and science to bear on challenging questions. Instead of just letting the chips fall where they may, help the Singularity Institute increase the probability of a positive Singularity by contributing financially to our research effort. We depend completely on donors like you for all funding.

2010 marks the 10th year since SIAI’s founding. With your help, SIAI will still exist in 2015, 2020, 2025… however long it takes to get to a positive Singularity. Thank you for your support!

No Responses »

New staff bios have been added to Singularity Institute’s team page: short bios for Research Fellows Anna Salamon and Steve Rayhawk, Media Director Michael Anissimov, and Chief Compliance Officer Amy Willey. Check them out, and feel free to ask if you’re interested in knowing more about what each staff member does. Here’s the lineup:

Cross-posted from SIAI blog.

2 Responses »

I recently heard from a colleague at SIAI that a major potential donor had visited the Visiting Fellows Program and was considering whether to fund any of our grant proposals. Some email initially gave me the impression that he was going to argue that we ought to spend more time directly on AI research and less on publishing academic papers, but I was surprised to learn it was the opposite. He said that we were too small to get very far with AI research on our own (our annual budget is in the $200,000/year range, and our finance information for past years is available on Guidestar), so it makes sense to put a lot of focus on academic papers and idea promotion, or so he argued.

I bring this up because I am sympathetic to SIAI supporters who argue that AGI research is a must. However, the constant question is, “does this research contribute to AGI in general or specifically Friendly AI?” We really want the latter, and really don’t want the former. (According to our current views, most AGI designs would lack the necessary values to be human-friendly and would probably view human beings as raw materials to achieve other ends. Engineering human-friendliness would require special, dedicated effort.)

A major AGI project would be expensive. I believe Eliezer Yudkowsky once said that $5M/year would be a reasonable number for such a project in its mid-stages. I’m not sure if $5M/year would be necessary to start (probably not), but certainly more than $200K a year would make sense. Thinking of AGI as a mighty pyramid, it seems fruitless to build it on such a weak foundation. Others may disagree, but I should point out that selection effects at the present time dictate that every AGI project is necessarily a small one. (Except maybe Numenta, if you consider that AGI.) Therefore, every AGI designer currently working on AGI believes that AGI can be done on a small budget.

Instead of working exclusively on Friendly AI and wasting all of our money because we get only 0.1% of the way to the goal with our current financial and human resources, why not rally some support until we actually have a foundation worth building on? It makes sense to me. Some people are disappointed that Eliezer took the majority of two years off of Friendly AI research to write the Overcoming Bias/Less Wrong sequences and prepare his book, but I know that SIAI probably wouldn’t be at its current level of resources if he never did. Most people who have gone through many of the posts have been very impressed, and we’ve acquired a lot of new allies thanks to that. A lot of interesting decision theoretical investigations have taken place on Less Wrong which are explicitly AGI-relevant, and the site has created a larger community with some star members who are being cultivated as potential Friendly AI programmers. There might not have even been a Visiting Fellows Program if it weren’t for Less Wrong.

SIAI has proven its ability to take relatively small amounts of money and generate huge quantities of discussion, analysis, and attention around the Singularity and closely related ideas. In terms of publicity per dollar spent, I would think we are in the 99th percentile of effectiveness for non-profits. The existence of SIAI has contributed in a major way to the rising profile of the Singularity in recent years (along with Ray Kurzweil), and to a Singularitarian like myself, I consider that very important. We are spreading ideas, making connections, acquiring allies, and some of us are doing AGI-relevant research. Our 2009 accomplishments document lists the major items. As always, we are open to feedback, and I encourage you to email myself or Anna Salamon at our singinst.org addresses if you have any comments or questions.

According to my current understanding, launching a positive Singularity offers the greatest good for the greatest number at this point in history. It doesn’t seem to me that the Singularity will go well by default — I think it will go wrong by default. So, strong and immediate intervention is necessary. At this point in time, SIAI seems like the best organization for the job, and we have received endorsements from many dedicated utilitarians, such as Alan Darwst.

9 Responses »

If you haven’t picked up this month’s GQ magazine, do it soon. There is a feature on the Singularity Summit and Singularity Institute. (I also hear there is a piece by Carl Zimmer on the Singularity in Playboy but I haven’t picked it up yet.) Seeing community names like Rick Schwall (an SIAI donor and supporter) in a national magazine sure is a trip. According to the National Magazine Awards, circulation is somewhere between 500,000 and 1,000,000 and is up in recent years.

Here is the Singularity portion (I removed the magazine cover due to copyright concerns and complaints from the comments section):

Really freaky, mmhmmm! Freaky like our ancestral past or Pandora freaky, I hope.

H/t to Gus K. for pointing out the article earlier this month.

7 Responses »

I forgot to post this back on 11/24/09 when it was originally sent out to our mailing list.

Dear Friends of SIAI,

I’d like to share with you my picture of SIAI’s most recent year, of the global situation we are addressing, and of the work we can do with your help.

To put it bluntly, we are a small group of intelligent, ambitious, but as yet mostly inexperienced people who are working to increase the odds of an eventual positive Singularity and to decrease the odds of human extinction. Given our size and the size of the world, we’ve been reaching out to larger bodies of capable people. 2009 saw a number of successes in this regard: our Singularity Summit drew 800 attendees, including top academics who have remained in conversation about AI risks; Eliezer Yudkowsky finished a long-running series of posts laying out the concepts needed for thinking about AI risks, and started the successful online community Less Wrong; and we established a Visiting Fellows Program, which brought 14 researchers to Silicon Valley to do research and writing around AI risks and seeded ongoing focused full-time work.

Successes are useful in themselves. They are also indicators of what else can be achieved. In light of recent bridge-building to outside researchers and actors, we are fairly confident that we can turn even relatively small increases in donations into greater odds of an eventual positive singularity (basically: by continuing to do research and outreach in contact with capable people who may care). We wrote up many of the details of how we can do that on a new grant proposals site — a site that lets you see exactly what proposed projects we’re thinking of, and how many dollars would let us do what to increase the odds of a positive long-run future.

If you care about humanity’s long-run future, and if you think SIAI is one of the more promising groups on the scene today, you’ll probably want to check out our proposed projects. Donate if you find a project worth funding or, if you don’t, share your thoughts on what else it is that would be worth funding. SIAI’s mission is ambitious but far from impossible; and spread across our staff and donor network we may have the seeds of a winning strategy.

Also, a Challenge Campaign is starting today, thanks to the generous support of Edwin Evans, Rolf Nelson, Henrik Jonsson, Jason Joachim, and Robert Lecnik. Until February 28, 2010, each dollar donated to SIAI will be matched on a 1:1 basis, up to $100,000, for a total of $200,000. In this context, especially, every donation counts.

Please take a look, and consider putting your dollars into the effort that, as far as I can tell, can turn your dollars into more good than any other today.

http://singinst.org/challenge

Best regards,

Michael Vassar,
President, SIAI

P.S.: I’m not kidding when I say I’d love to hear your thoughts. Do email me.

P.P.S.: We’re still taking applications for our Visiting Fellows program. If you’re interested in working directly with SIAI, check out our program description, and send in an application.

5 Responses »

Bob Mottram isn’t impressed by the Singularity Institute’s grant proposals for our $100,000 Singularity Research Challenge:

It’s kind of sad how SIAI seems to have become obsessed with “AI risks” and human extinction. Perhaps they always were from the beginning, but it’s just my perception of them that was at fault. There’s certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.

The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.

To clarify what is happening, what Bob Mottram considers “spurious notions of risk”, we consider “deadly serious notions of risk”, so this is the main source of disagreement. Here was my response:

Our Uncertain Future project is pioneering probabilistic futurism in AI and WBE studies, and has received thumbs-up from several academics including Bela Nagy, who manages the Santa Fe Institute Performance Curve Database.

A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider “nice” or “common sense” is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values. See the Fun Theory sequence on Less Wrong, for instance.

SIAI believes that AGI is an extremely difficult endeavor and deserves far more theory-level work than programming in the dark or working towards narrow AI tasks that drain away our attention at the expense of the Singularity itself.

Basically, if you consider an intelligence explosion plausible, SIAI’s activities make sense, and if you don’t, they don’t. It’s not a matter of marketing, just disagreement on which tasks are the most important for humanity to face right now. We consider clarifying decision theory and creating a reflective decision theory to be a major priority, for instance, and spend time on that accordingly.

To clarify further, in 2009 SIAI grew large enough to break into several loose divisions. This is excellent, because the Singularity Institute is one of the most important organizations on the planet and is one of the only barriers standing between humanity and extinction from unFriendly AI. However, it makes the task of explaining what we do all the more complicated. It so happens that I am paid to explain it, but sometimes I get discouraged because I discuss the organization constantly on this blog, occasionally several posts per day, and there is still a great amount of confusion about what our organization does and believes. Perhaps I ought to run an SIAI Video Q&A in the vein of Eliezer’s recent Less Wrong Q&A.

What happened in the last year is that Anna Salamon and Steve Rayhawk joined us and created the Visiting Fellows program, under Anna’s leadership. (Anna, Steve, and myself were only recently added to our staff page.) This entity is only peripherally related to SIAI’s central AI project, which was more or less put on hold for two years while Eliezer Yudkowsky wrote the Less Wrong sequences. As our 2009 accomplishments document states, Eliezer worked with Marcello Herreshoff (his profile can be found here) on Friendly AI over the Summer.

So, think of SIAI as having three branches — 1. Administrative/PR, which consists of President Michael Vassar, Media Director Michael Anissimov (aka me), and Chief Compliance Officer Amy Willey, 2. AGI research that constitutes serious progress towards seed AI, which makes up years of Eliezer’s past work, Eliezer’s future work after he finishes putting together his rationality book, Marcello Herreshoff’s intermittent work, and contributions from Peter de Blanc, Nick Hay, and others (since 2006), including Anna Salamon and Steve Rayhawk, and 3. the Visiting Fellows Program, which including Visiting Fellows and various volunteers.

The goal of the Visiting Fellows Program is to put together extremely smart people concerned about reducing existential risk and have them pursue academic projects that make the best use of their respective strengths. Branch #3 also serves as a filtering mechanism for 2. The thing is, starting a true AGI project would be very expensive, not so much in money but in terms of intelligence, philosophy, computer science, and math knowledge required. Consolidating the necessary personnel will not be easy.

Why are we concerned about “AI risks” and human extinction? Well, this is why, among other reasons. SIAI is not about pursuing intermediate AI commercial benefits — our organization only exists to pursue the Singularity and minimize AI risk. Writing illustrating this point has been produced in substantial quantities since our founding in 2000. SIAI is mostly a bunch of utilitarians.

Would readers be interested in a Lulu book putting together a lot of information about the Singularity Institute in one place? Only about 0.5% of my blog readers ever comment, so I feel like I’m talking to a vast sea of silent lurkers all the time. Seriously, it’s weird.

In general, our approach turns off people like Bob Mottram, but inspires praise from people like Alan Darwst. In particular, Mr. Darwst writes:

Among the utilitarians I’ve met over the years, a sizable fraction have come to the conclusion that the optimal destination for utilitarian funding is organizations that research speculative futuristic scenarios and the philosophical / scientific / methodological questions that such research requires. In particular, many of these utilitarians have named the Singularity Institute for Artificial Intelligence (SIAI) as a good example of such an organization, so I’ll focus on it here, but the discussion can apply more broadly.

The way the Singularity goes is a matter of life and death for humanity. Unfriendly AI programmed to value anything besides a very specific set of Homo sapiens-characteristic values will probably overlook our material preservation. From the perspective of most possible minds, humans are just another arrangement of atoms. We don’t have any inherent moral value. “Moral value” is an “imaginary” thing that only exists among the tiny space of minds-in-general with explicit moral philosophies.

If we had the ability to build AGI today, our planet would not last the year, because we haven’t solved Friendly AI. If we could build a seed AI now, we wouldn’t know how to specify its goals in a way that doesn’t eliminate us all completely. We are clueless. We can’t create a utility function that is consistent under reflection and preserves individual humans when a tremendous amount of optimization pressure is applied to fulfilling it. We need a mathematical model of value that leaves us alive even when the unimaginable power of superintelligence is channeled into it. I think that Coherent Extrapolated Volition is a good enough solution that it would work, but it needs to be specified in much more detail. That’s exactly what one of our grant proposals is about.

These grant proposals deserve funding now. We are about to walk into a minefield and we don’t even have a map. We need to throw everything at the problem — people, money, attention, everything.

26 Responses »

On Sunday, January 3rd, I did an interview on KUSP (Central Coast Public Radio) in Santa Cruz, California, a National Public Radio affiliate. I talked to Rick Kleffel for an hour about the Singularity, the Singularity Institute, what we do, anthropomorphism, Friendly AI, and the like. It was for his “Talk of the Bay” radio program. Here is the audio archive.

4 Responses »

Over the past week and a half that the Singularity Research Challenge has been launched, we’ve received some nice support, including a post by Razib Khan at the Gene Expression blog and an explicit donation recommendation by Alan Darwst, author of Utilitarian-Essays.com and a well-regarded figure in the online utilitarian community. Here is a post by Alan on Felicia, the utilitarian community, that goes into why research charities like SIAI offer a very high return on philanthropic investment.

Don’t it just let be Alan and Razib — you too can make a blog post about the Singularity Research Challenge, right at this very moment!

No Responses »

In the context of our 2010 Singularity Research Challenge, Steven over at Black Belt Bayesian has a collection of “reasons to invest in reducing existential risk that you might not have considered before”.

No Responses »

Today is January 1st, 2010, the 10th anniversary of the online publishing of “The Singularitarian Principles” by Eliezer Yudkowsky. This document is a handy set of common sense advice for anyone who considers the possible creation of superintelligence a big deal in utilitarian terms (or otherwise). The work is divided into four “definitional principles”, which form of the central definition of the term “Singularitarian” (as it was defined at the time), and “descriptive principles”, which “aren’t strictly necessary to the definition, but which form de facto parts of the Singularitarian meme.”

The definitional principles are:

1. Singularity
2. Activism
3. Ultratechnology
4. Globalism

The descriptive principles are:

1. Apotheosis
2. Solidarity
3. Intelligence
4. Independence
5. Nonsuppression

The “Singularity” principle refers to believing in “some fundamental change in the rules” in the future. Looking back on this from the vantage point of 2010, I think the term “Singularity” as defined here (”defined many different ways”) is far too vague to be useful. Yudkowsky probably should have anticipated how diluted the meaning of the word would become and the need to define it more narrowly for it to be useful. Other than that, I think the rest of the document essentially makes sense, and even after ten years it is not obsolete by any means.

The activism principle says, “A Singularitarian is someone who believes that technologically creating a greater-than-human intelligence is desirable, and who works to that end.” Aside from #1, this is the most important principle in my eyes. Kurzweil’s definition of a Singularitarian is someone who “understands the Singularity” and has reflected on its consequences for their own life, but to my eyes this is excessively broad and meaningless. Am I an environmentalist just because I understand the environment and have reflected on its consequences for my life? No. Movements are defined by activism and change, not silent reflection alone. Kurzweil is making the definition of a Singularitarian vague so he can sell it to a wider audience, but I doubt that this definition is useful for actually inspiring humanity to have a positive impact on the Singularity, which it must do or perish.

The principle of “ultratechnology” is that “the “Singularity” is a natural, non-mystical, technologically triggered event”, and “What distinguishes the Singularitarians is that we want to bring about a natural event, working through ultratechnologies such as AI or nanotech, without relying on mystical means or morally valent effects.” What this basically says is that we are scientific materialists who refuse to believe in woo. Pretty basic. The few challengers that try to connect us with mysticism over the past decade have largely failed, in my opinion. A more useful avenue of approach has been to claim that the human brain is extremely complex, it must be copied exactly to produce intelligence, and Singularitarians are overconfident of the timescales on which this can be achieved. To help clarify the situation, we created the Uncertain Future modeling application. This application will allow us to make our assumptions explicit in debates, and force us to think about these assumptions carefully.

Globalism means that we want the benefits of a Singularity to extend to everybody. As far as I can tell, most Singularitarians of the Singularitarian Principles variety take this for granted. The benefits of the Singularity would be so large that it would take everyone to fully enjoy them. Programming an AI to benefit some small group rather than humanity in general would probably be a technical hassle in the long run anyway (though I could be wrong about that), and likely terribly unstable, not to mention idiotic and evil. Speaking for the Singularity Institute, we currently have strongly utilitarian and globalist bent. I worry more about our activism than our globalism.

For the descriptive principles, visit the original page, which I strongly recommend if you care to learn more about the Singularity and Singularitarians.

25 Responses »

As I mentioned in my last post, the Singularity Institute (SIAI) has launched a 2010 Singularity Research Challenge to raise funds for Singularity research. Our organization is worth giving money to because the Singularity is a matter of life and death for our entire species, and we may only have a few decades remaining to deal with it. Our group is the most dedicated to maximizing the probability of a positive outcome, and has the intelligence and skill to produce detailed ideas and attract major media attention. We achieve a huge amount with our money. Nearly everyone at the Singularity Institute, including myself, takes a salary significantly lower than our market value given our education and experience, because we personally care about this issue a whole lot.

We have a network of several dozen young academics, mostly aged 20-30, who are devoted to performing research and writing papers on the topic of the Singularity if given the proper support and infrastructure. (For a snapshot of 2009’s Visiting Fellows, along with names and bios, see this page.) Most of these people have degrees or are working on degrees from schools like Stanford, Harvard, Carnegie Mellon, and Yale. For instance, SIAI volunteer Tom McCabe is at Yale and sometimes-SIAI Research Associate Marcello Herreshoff is at Stanford. Many of these youngsters are astonishing geniuses, performing extremely well at both formal and informal tests of intelligence, and it is clear that their life goal consists of having a positive impact on the Singularity.

What we lack is sufficient funding to sponsor all the research we want to sponsor. Visit the website of the Challenge and scroll down to the grant proposals to see some of our ideas for work in 2010. This list is part of an effort at SIAI being spearheaded by Research Fellow Anna Salamon to increase transparency and accountability as we expand. Anna is a competent, energetic manager who leads much of SIAI’s research wing, which is located in Santa Clara, CA. It was under her leadership that the SIAI Visiting Fellows Program was founded, and the grants proposal thing was her idea, which myself, SIAI Visiting Fellows, and volunteers duly fleshed out.

Other than room and board for researchers, SIAI doesn’t have much overhead. There are some administrators, such as President Michael Vassar and our Chief Compliance Officer Amy Willey, and one PR nerd, yours truly. Our goal is to convert dollars into existential risk mitigation more effectively than any other group. Of course, your evaluation of this will vary depending on how much of a risk you believe the Singularity could pose to humanity. Among those who believe that the Singularity could be a risk and it ought to be studied and dealt with, the Singularity Institute often gets high marks. When we receive constructive criticism from supporters, we keep it in mind in everything we do, and frequently question ourselves.

One unfortunate element that holds us back at this point is that the majority of our supporters are in their twenties or early thirties and therefore have limited personal wealth. However, some of SIAI’s dedicated supporters have already secured jobs in the tech sector with decent earning potential. As the Singularity Institute’s support base grows and matures, we will certainly possess greater resources, but we don’t want to wait. We want to fund as much Singularity research as we can, right now. Those considering donating larger amounts, say $1,000 or more, should feel free to get in touch with Anna Salamon (her email address is on the challenge page) and talk about what sort of research you would consider worthwhile to fund.

Discussion about the Singularity, and the Singularity Institute itself, are here to stay. We will still be around in 2020, 2030, 2040 — however long it takes to get to a positive Singularity. We are in it for the long haul. The question is, are you with us? If a robotic or AI advance you see in the news in the next decade impresses you with the speed of mankind’s progress towards real artificial intelligence, and has you concerned about the implications, you may regret not donating to our organization now, in 2009 or 2010. If time travel were possible, then the Singularity Institute is the kind of organization that people would be sent back in time to help. By sheer dumb luck, however, you happen to be one of the few people who has heard of the Singularity Institute this early in the game. Wouldn’t it be silly not to take advantage of that fact?

6 Responses »

Here is a summary of the Singularity Institute’s 2009 accomplishments that myself and other SIAI staff and friends compiled recently in preparation for our 2010 Singularity Research Challenge, where every dollar donated up to $100,000 will be matched. You can also select which research you choose to support, if you like. We compiled almost 20 grants to choose from. Without further adieu, here is the summary, and be sure to visit SIAI’s website for proper formatting and links:

2009 has been a year of growth and new horizons for the Singularity Institute for Artificial Intelligence (SIAI). We achieved a number of milestones relevant to our mission — pursuing dialogue, research, and activism to promote a beneficial Singularity. The response we’ve received has been considerable — SIAI is more high-profile and frequently-mentioned now than it has ever been.

Our key accomplishments in 2009 were holding the Singularity Summit in New York, hiring three new employees (Michael Vassar, Michael Anissimov, and Amy Willey), establishing a continuous SIAI Visiting Fellows Program, delivering eight presentations across four conferences, improving cooperation with allied organizations such as the Future of Humanity Institute, and establishing the Less Wrong web community, which receives thousands of visitors per day and fosters many high-quality discussions on philosophical and practical issues related to decision theory and rationality. The Uncertain Future, an interactive web application for quantitatively modeling future possibilities such as human-level AI, human intelligence enhancement, and global catastrophic risk, was also released as a beta version in December.

In April, Eliezer Yudkowsky completed two years of posting sequences on Less Wrong (which will be edited into a book on rationality and Singularity-relevant topics like reductionism and decision theory), drafting strategy documents for improving internal organization and long-term planning. Throughout the year, we continued consolidating SIAI staff, Visiting Fellows, volunteers, and interns in the San Francisco Bay Area. SIAI Visiting Fellow Peter de Blanc revised a paper on unbounded utility functions. The Singularity Institute received media coverage for its work in The New York Times, Popular Mechanics, Popular Science, Forbes, and many other venues. An article by SIAI President Michael Vassar, “Machine Minds”, made it into the Forbes special “The AI Report”.

The Singularity Institute’s long-term mission is to maximize the probability of a beneficial Singularity, through dialogue, research, and activism. All of our activities are ultimately chosen to further this purpose. The Singularity Institute particularly focuses on the possibility of a Singularity through artificial general intelligence, but also analyzes other potential pathways, including whole brain emulation and human cognitive enhancement.

To summarize our major accomplishments over the past year:

1. Singularity Summit 2009 in New York. Our fourth annual Singularity Summit was the first Singularity-focused conference ever held on the East Coast. Held October 3-4, the Singularity Summit featured 25 excellent speakers on topics including biotechnology, futurism, decision theory, artificial intelligence, quantum computing. the scientific method, cognitive ability, philosophy, computer science, and even synthetic neurobiology. Over 800 people attended, and the conference attracted reporters from over two dozen news organizations, including the New York Times. Coverage was provided by Popular Mechanics, Popular Science, Forbes, and many other media venues. Speakers this year included venture capitalist Peter Thiel, Wired magazine contributing editor Gary Wolf, AI researchers Juergen Schmidhuber, Marcus Hutter, and Itamar Arel, SIAI employees Anna Salamon, Ben Goertzel, and Eliezer Yudkowsky, philosopher David Chalmers, futurist Ray Kurzweil, Stephen Wolfram of Mathematica and Wolfram Alpha fame, and many others. Videos from the Summit are online at Vimeo. After the Summit, SIAI held an in-depth workshop, which allowed the speakers and SIAI staff to share ideas and brainstorm about the risks and benefits of a possible Singularity.

2. Hiring of new employees. Early in the year, Executive Director Tyler Emerson departed the Singularity Institute and his role was filled by a new President, Michael Vassar. Mr. Vassar holds a B.S. in biochemistry from Penn State and an MBA from Drexel University, and was previously Founder and Chief Strategist at Sir Groovy, an online music licensing firm. Prior to that, he held positions with Aon, the Peace Corps, and the National Institute of Standards and Technology. Throughout the year, he participated in numerous interviews and podcasts on behalf of SIAI, including interviews at Accelerating Future, The Futurist, Future Blogger, and h+ magazine.

Two new research fellows, Anna Salamon and Steve Rayhawk, were hired by SIAI in late 2008. Salamon and Rayhawk had previously participated in the 2008 SIAI Summer Program, which was led by Salamon. Salamon holds degrees in mathematics from UC Santa Barbara and Great Books from St. John’s, and Rayhawk holds a degree in mathematics from UC Santa Barbara. Salamon and Rayhawk are both focusing on applying computational Bayesian decision theory to problems in technological forecasting, risk management policy, and social epistemology, and form the core of our Visiting Fellows Program, bringing visiting scholars up to speed on the work that SIAI does. In early 2009, SIAI also hired a Media Director, Michael Anissimov, responsible for compiling, distributing, and promoting SIAI media materials including our writing, websites, and videos, and communicating the activities of SIAI to the public. Anissimov is author of Accelerating Future, a popular blog focused on science and futurism. Most recently, in December, SIAI hired Amy Willey, who holds a law degree from New York University, as Chief Compliance Officer.

With the addition of these new employees, SIAI brought its total full-time employee count to six, including Research Fellow Eliezer Yudkowsky, who has worked for SIAI since he co-founded the organization in 2000.

3. In 2009, SIAI established a Visiting Fellows Program, based in Silicon Valley. The program began with SIAI’s 2009 Summer Fellows, brought together to work on challenging projects in decision theory, philosophy, technological forecasting, heuristics and biases, and planning for the Singularity Summit 2009. Primarily graduate students, the Fellows came from educational backgrounds in mathematics, computer science, and physics, with the remainder ranging from philosophy to economics and biochemistry. They attend or hold degrees from universities including Harvard, Stanford, Yale, Cambridge, Carnegie Mellon, Auckland University, Moscow Institute of Physics and Technology, and the University of California-Santa Barbara. Fellows traveled to Silicon Valley from throughout the United States and from Russia, Belgium, the Netherlands, Sweden, Australia, New Zealand, the United Kingdom, and Canada. Some of these researchers stayed on past the summer or joined shortly thereafter to work with SIAI as volunteers or Visiting Fellows on a more extended basis. Some of the work that came out of the Visiting Fellows Program has been presented in papers and talks at venues like the European Conference on Computing and Philosophy, the Asia-Pacific Conference on Computing and Philosophy, and a Santa Fe Institute conference on forecasting. The Visiting Fellows Program has been instrumental in fostering a devoted community of Singularity Institute supporters making useful contributions towards SIAI’s ultimate goal, and SIAI recently put out a fresh call for new SIAI Visiting Fellows.

4. SIAI researchers, volunteers, and Visiting Fellows presented the following nine talks and papers throughout 2009:

* “Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions”, by Steve Rayhawk, Anna Salamon, Tom McCabe, Rolf Nelson, and Michael Anissimov. (Presented at the European Conference of Computing and Philosophy in July ‘09 (ECAP))
* “Arms Control and Intelligence Explosions”, by Carl Shulman (Also presented at ECAP)
*“Machine Ethics and Superintelligence”, by Carl Shulman and Henrik Jonsson (Presented at the Asia-Pacific Conference of Computing and Philosophy in October ‘09 (APCAP))
*“Which Consequentialism? Machine Ethics and Moral Divergence”, by Carl Shulman and Nick Tarleton (Also presented at APCAP);
*“Long-term AI forecasting: Building methodologies that work”, an invited presentation by Anna Salamon at the Santa Fe Institute conference on forecasting;
*“Shaping the Intelligence Explosion” and “How much it matters to know what matters: A back of the envelope calculation”, presentations by Anna Salamon at the Singularity Summit 2009 in October;
* “Pathways to Beneficial Artificial General Intelligence: Virtual Pets, Robot Children, Artificial Bioscientists, and Beyond”, a presentation by SIAI Director of Research Ben Goertzel at Singularity Summit 2009;
* “Cognitive Biases and Giant Risks”, a presentation by SIAI Research Fellow Eliezer Yudkowsky at Singularity Summit 2009;
* “Convergence of Expected Utility for Universal Artificial Intelligence”, a paper by Peter de Blanc, an SIAI Visiting Fellow.

Many more talks and papers are in the works for 2010, including a talk by SIAI Media Director Michael Anissimov at the Foresight 2010 conference in January.

5. One of the primary goals of the Singularity Institute in 2009 was to strengthen our ties to academia and allied organizations, which was accomplished through talks, papers, and direct dialogue. SIAI researchers and representatives built closer ties to organizations such as the Future of Humanity Institute at Oxford University, Santa Fe Institute, American Association for Artificial Intelligence, Foresight Institute, and many others. SIAI researcher Anna Salamon was invited to give a talk at an exclusive conference on technological forecasting held by the Santa Fe Institute. The Singularity Institute has been using videoconferencing, blogs, and mailing lists to keep in contact with our supporters and collaborators around the globe. SIAI more than tripled its representatives through the Visiting Fellows program, allowing it to better interface with a larger network.

6. 2009 saw the founding of the Less Wrong web community. Less Wrong was founded as a rationalist community to “systematically improve on the art, craft, and science of human rationality”. Thousands of people visit the site every day, with hundreds participating regularly in the comments sections. Less Wrong grew out of Overcoming Bias, a blog co-authored by SIAI Research Fellow Eliezer Yudkowsky and George Mason University economist Robin Hanson. Yudkowsky wrote extensively on Overcoming Bias from 2007-2009, and his posts have been ported over to Less Wrong, where they are organized into sequences that address topics such as reductionism, determinism, human rationality, metaethics, mathematics, and many others.

Less Wrong is important to the Singularity Institute’s work towards a beneficial Singularity in providing an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence. At the same time, it has gathered a community that can provide rapid feedback and significant progress on such problems. For instance, Less Wrong participants Wei Dai and Vladimir Nesov proposed decision algorithms that can deal with a certain classes of problems where Bayesian updating seems to lead decisionmakers astray. This work was closely related to decision theory work done in-house at SIAI, namely Eliezer Yudkowsky’s timeless decision theory, an algorithm that computes the counterfactual consequences of possible actions using an extension of Judea Pearl’s formalism of causal networks to logical uncertainties, and additional work by Anna Salamon and Steve Rayhawk. These developments have received positive attention from Gary Drescher and philosopher David Chalmers, and will be written up for peer review in the coming year.

Besides providing a home for an intellectual community dialoguing on rationality and decision theory, Less Wrong is also a key venue for SIAI recruitment. Many of the participants in SIAI’s Visiting Fellows Program first discovered the organization through Less Wrong.

7. This year Eliezer Yudkowsky finished his posting sequences on Less Wrong, which attracted thousands of enthusiastic readers and came to serve as the seed of a new community. Yudkowsky used the blogging format to write the substantive content of a book on rationality, enabling that work to be read and receive feedback as it progressed. Throughout the summer, Yudkowsky engaged in Friendly AI research with Marcello Herreshoff, a Stanford mathematics student who previously spent his gap year working for SIAI. Yudkowsky is now converting his blog sequences into the planned rationality book, which he hopes will significantly assist in attracting and inspiring talented individuals to effectively work towards the aims of a beneficial Singularity and reduced existential risk.

8. In December, a subset of SIAI researchers and volunteers finished improving The Uncertain Future web application to officially announce it as a beta version. The Uncertain Future represents a new kind of futurism — futurism with heavy-tailed, high-dimensional probability distributions. The purpose is to provide a tool for use by futurists and the informed public to input probability distributions over quantitative questions like, “how much computing power would be necessary to implement neuromorphic AI?”, combining them into a “picture of the future according to you”. Another goal of the project is to provide an alternative to the futurist methodologies of storytelling and scenario building, which dominate the field even though they often cause futurists to overestimate the probability of precise, vivid stories at the expense of a wider space of neglected possibilities.

22 Responses »