Singularity Institute for Artificial Intelligence 2009 Accomplishments Saturday, Dec 26 2009
friendly ai and futurism and SIAI 12:02 pm
Here is a summary of the Singularity Institute’s 2009 accomplishments that myself and other SIAI staff and friends compiled recently in preparation for our 2010 Singularity Research Challenge, where every dollar donated up to $100,000 will be matched. You can also select which research you choose to support, if you like. We compiled almost 20 grants to choose from. Without further adieu, here is the summary, and be sure to visit SIAI’s website for proper formatting and links:
2009 has been a year of growth and new horizons for the Singularity Institute for Artificial Intelligence (SIAI). We achieved a number of milestones relevant to our mission — pursuing dialogue, research, and activism to promote a beneficial Singularity. The response we’ve received has been considerable — SIAI is more high-profile and frequently-mentioned now than it has ever been.
Our key accomplishments in 2009 were holding the Singularity Summit in New York, hiring three new employees (Michael Vassar, Michael Anissimov, and Amy Willey), establishing a continuous SIAI Visiting Fellows Program, delivering eight presentations across four conferences, improving cooperation with allied organizations such as the Future of Humanity Institute, and establishing the Less Wrong web community, which receives thousands of visitors per day and fosters many high-quality discussions on philosophical and practical issues related to decision theory and rationality. The Uncertain Future, an interactive web application for quantitatively modeling future possibilities such as human-level AI, human intelligence enhancement, and global catastrophic risk, was also released as a beta version in December.
In April, Eliezer Yudkowsky completed two years of posting sequences on Less Wrong (which will be edited into a book on rationality and Singularity-relevant topics like reductionism and decision theory), drafting strategy documents for improving internal organization and long-term planning. Throughout the year, we continued consolidating SIAI staff, Visiting Fellows, volunteers, and interns in the San Francisco Bay Area. SIAI Visiting Fellow Peter de Blanc revised a paper on unbounded utility functions. The Singularity Institute received media coverage for its work in The New York Times, Popular Mechanics, Popular Science, Forbes, and many other venues. An article by SIAI President Michael Vassar, “Machine Minds”, made it into the Forbes special “The AI Report”.
The Singularity Institute’s long-term mission is to maximize the probability of a beneficial Singularity, through dialogue, research, and activism. All of our activities are ultimately chosen to further this purpose. The Singularity Institute particularly focuses on the possibility of a Singularity through artificial general intelligence, but also analyzes other potential pathways, including whole brain emulation and human cognitive enhancement.
To summarize our major accomplishments over the past year:
1. Singularity Summit 2009 in New York. Our fourth annual Singularity Summit was the first Singularity-focused conference ever held on the East Coast. Held October 3-4, the Singularity Summit featured 25 excellent speakers on topics including biotechnology, futurism, decision theory, artificial intelligence, quantum computing. the scientific method, cognitive ability, philosophy, computer science, and even synthetic neurobiology. Over 800 people attended, and the conference attracted reporters from over two dozen news organizations, including the New York Times. Coverage was provided by Popular Mechanics, Popular Science, Forbes, and many other media venues. Speakers this year included venture capitalist Peter Thiel, Wired magazine contributing editor Gary Wolf, AI researchers Juergen Schmidhuber, Marcus Hutter, and Itamar Arel, SIAI employees Anna Salamon, Ben Goertzel, and Eliezer Yudkowsky, philosopher David Chalmers, futurist Ray Kurzweil, Stephen Wolfram of Mathematica and Wolfram Alpha fame, and many others. Videos from the Summit are online at Vimeo. After the Summit, SIAI held an in-depth workshop, which allowed the speakers and SIAI staff to share ideas and brainstorm about the risks and benefits of a possible Singularity.
2. Hiring of new employees. Early in the year, Executive Director Tyler Emerson departed the Singularity Institute and his role was filled by a new President, Michael Vassar. Mr. Vassar holds a B.S. in biochemistry from Penn State and an MBA from Drexel University, and was previously Founder and Chief Strategist at Sir Groovy, an online music licensing firm. Prior to that, he held positions with Aon, the Peace Corps, and the National Institute of Standards and Technology. Throughout the year, he participated in numerous interviews and podcasts on behalf of SIAI, including interviews at Accelerating Future, The Futurist, Future Blogger, and h+ magazine.
Two new research fellows, Anna Salamon and Steve Rayhawk, were hired by SIAI in late 2008. Salamon and Rayhawk had previously participated in the 2008 SIAI Summer Program, which was led by Salamon. Salamon holds degrees in mathematics from UC Santa Barbara and Great Books from St. John’s, and Rayhawk holds a degree in mathematics from UC Santa Barbara. Salamon and Rayhawk are both focusing on applying computational Bayesian decision theory to problems in technological forecasting, risk management policy, and social epistemology, and form the core of our Visiting Fellows Program, bringing visiting scholars up to speed on the work that SIAI does. In early 2009, SIAI also hired a Media Director, Michael Anissimov, responsible for compiling, distributing, and promoting SIAI media materials including our writing, websites, and videos, and communicating the activities of SIAI to the public. Anissimov is author of Accelerating Future, a popular blog focused on science and futurism. Most recently, in December, SIAI hired Amy Willey, who holds a law degree from New York University, as Chief Compliance Officer.
With the addition of these new employees, SIAI brought its total full-time employee count to six, including Research Fellow Eliezer Yudkowsky, who has worked for SIAI since he co-founded the organization in 2000.
3. In 2009, SIAI established a Visiting Fellows Program, based in Silicon Valley. The program began with SIAI’s 2009 Summer Fellows, brought together to work on challenging projects in decision theory, philosophy, technological forecasting, heuristics and biases, and planning for the Singularity Summit 2009. Primarily graduate students, the Fellows came from educational backgrounds in mathematics, computer science, and physics, with the remainder ranging from philosophy to economics and biochemistry. They attend or hold degrees from universities including Harvard, Stanford, Yale, Cambridge, Carnegie Mellon, Auckland University, Moscow Institute of Physics and Technology, and the University of California-Santa Barbara. Fellows traveled to Silicon Valley from throughout the United States and from Russia, Belgium, the Netherlands, Sweden, Australia, New Zealand, the United Kingdom, and Canada. Some of these researchers stayed on past the summer or joined shortly thereafter to work with SIAI as volunteers or Visiting Fellows on a more extended basis. Some of the work that came out of the Visiting Fellows Program has been presented in papers and talks at venues like the European Conference on Computing and Philosophy, the Asia-Pacific Conference on Computing and Philosophy, and a Santa Fe Institute conference on forecasting. The Visiting Fellows Program has been instrumental in fostering a devoted community of Singularity Institute supporters making useful contributions towards SIAI’s ultimate goal, and SIAI recently put out a fresh call for new SIAI Visiting Fellows.
4. SIAI researchers, volunteers, and Visiting Fellows presented the following nine talks and papers throughout 2009:
* “Changing the frame of AI futurism: From storytelling to heavy-tailed, high-dimensional probability distributions”, by Steve Rayhawk, Anna Salamon, Tom McCabe, Rolf Nelson, and Michael Anissimov. (Presented at the European Conference of Computing and Philosophy in July ‘09 (ECAP))
* “Arms Control and Intelligence Explosions”, by Carl Shulman (Also presented at ECAP)
*“Machine Ethics and Superintelligence”, by Carl Shulman and Henrik Jonsson (Presented at the Asia-Pacific Conference of Computing and Philosophy in October ‘09 (APCAP))
*“Which Consequentialism? Machine Ethics and Moral Divergence”, by Carl Shulman and Nick Tarleton (Also presented at APCAP);
*“Long-term AI forecasting: Building methodologies that work”, an invited presentation by Anna Salamon at the Santa Fe Institute conference on forecasting;
*“Shaping the Intelligence Explosion” and “How much it matters to know what matters: A back of the envelope calculation”, presentations by Anna Salamon at the Singularity Summit 2009 in October;
* “Pathways to Beneficial Artificial General Intelligence: Virtual Pets, Robot Children, Artificial Bioscientists, and Beyond”, a presentation by SIAI Director of Research Ben Goertzel at Singularity Summit 2009;
* “Cognitive Biases and Giant Risks”, a presentation by SIAI Research Fellow Eliezer Yudkowsky at Singularity Summit 2009;
* “Convergence of Expected Utility for Universal Artificial Intelligence”, a paper by Peter de Blanc, an SIAI Visiting Fellow.
Many more talks and papers are in the works for 2010, including a talk by SIAI Media Director Michael Anissimov at the Foresight 2010 conference in January.
5. One of the primary goals of the Singularity Institute in 2009 was to strengthen our ties to academia and allied organizations, which was accomplished through talks, papers, and direct dialogue. SIAI researchers and representatives built closer ties to organizations such as the Future of Humanity Institute at Oxford University, Santa Fe Institute, American Association for Artificial Intelligence, Foresight Institute, and many others. SIAI researcher Anna Salamon was invited to give a talk at an exclusive conference on technological forecasting held by the Santa Fe Institute. The Singularity Institute has been using videoconferencing, blogs, and mailing lists to keep in contact with our supporters and collaborators around the globe. SIAI more than tripled its representatives through the Visiting Fellows program, allowing it to better interface with a larger network.
6. 2009 saw the founding of the Less Wrong web community. Less Wrong was founded as a rationalist community to “systematically improve on the art, craft, and science of human rationality”. Thousands of people visit the site every day, with hundreds participating regularly in the comments sections. Less Wrong grew out of Overcoming Bias, a blog co-authored by SIAI Research Fellow Eliezer Yudkowsky and George Mason University economist Robin Hanson. Yudkowsky wrote extensively on Overcoming Bias from 2007-2009, and his posts have been ported over to Less Wrong, where they are organized into sequences that address topics such as reductionism, determinism, human rationality, metaethics, mathematics, and many others.
Less Wrong is important to the Singularity Institute’s work towards a beneficial Singularity in providing an introduction to issues of cognitive biases and rationality relevant for careful thinking about optimal philanthropy and many of the problems that must be solved in advance of the creation of provably human-friendly powerful artificial intelligence. At the same time, it has gathered a community that can provide rapid feedback and significant progress on such problems. For instance, Less Wrong participants Wei Dai and Vladimir Nesov proposed decision algorithms that can deal with a certain classes of problems where Bayesian updating seems to lead decisionmakers astray. This work was closely related to decision theory work done in-house at SIAI, namely Eliezer Yudkowsky’s timeless decision theory, an algorithm that computes the counterfactual consequences of possible actions using an extension of Judea Pearl’s formalism of causal networks to logical uncertainties, and additional work by Anna Salamon and Steve Rayhawk. These developments have received positive attention from Gary Drescher and philosopher David Chalmers, and will be written up for peer review in the coming year.
Besides providing a home for an intellectual community dialoguing on rationality and decision theory, Less Wrong is also a key venue for SIAI recruitment. Many of the participants in SIAI’s Visiting Fellows Program first discovered the organization through Less Wrong.
7. This year Eliezer Yudkowsky finished his posting sequences on Less Wrong, which attracted thousands of enthusiastic readers and came to serve as the seed of a new community. Yudkowsky used the blogging format to write the substantive content of a book on rationality, enabling that work to be read and receive feedback as it progressed. Throughout the summer, Yudkowsky engaged in Friendly AI research with Marcello Herreshoff, a Stanford mathematics student who previously spent his gap year working for SIAI. Yudkowsky is now converting his blog sequences into the planned rationality book, which he hopes will significantly assist in attracting and inspiring talented individuals to effectively work towards the aims of a beneficial Singularity and reduced existential risk.
8. In December, a subset of SIAI researchers and volunteers finished improving The Uncertain Future web application to officially announce it as a beta version. The Uncertain Future represents a new kind of futurism — futurism with heavy-tailed, high-dimensional probability distributions. The purpose is to provide a tool for use by futurists and the informed public to input probability distributions over quantitative questions like, “how much computing power would be necessary to implement neuromorphic AI?”, combining them into a “picture of the future according to you”. Another goal of the project is to provide an alternative to the futurist methodologies of storytelling and scenario building, which dominate the field even though they often cause futurists to overestimate the probability of precise, vivid stories at the expense of a wider space of neglected possibilities.
9. Code?
If the past decade is any evidence of the future, in a few decades you’ll have an impressive pile of papers, a vast media library of talks, ties to everyone who matters, limitless funding,
and no Singularity.
It looks like you’re dangerously closing in on the event horizon of that bureaucratic black hole where even the meetings have meetings. The next decade calls for a change in priorities: less talk, less publicity, less meetings, more code. Or just more code. Lots more. Unless you make changes you’ll end up where you’re going.
2000-2009: theory, getting funding, firehosing the memes
2010-2019: theory, code
2019 – what’s going to be different? I’d like to see the SIAI come up with actionable plans for the next decade that take concrete, measurable steps on the path to Singularity. Not just plans of plans of doing so.
You need to get coding or in a few years people may start pulling their support until something concrete materializes, since you can get the same sweet sounding hot air in any science fiction meeting. And after a decade, the air is getting stale.
Correct me if this view has nothing to do with reality. That’s just how it looks like to a not-so-casual observer.
Code or it didn’t happen.
I have to agree with Cranky Singularitarian’s comment. I am a little disappointed that in the course of one year this is all SIAI has to show for it. I was really hoping to see some technical papers, code and math. I am happy to see that after all these years SIAI has finally realized the importance of being accountable for how they spend their time and money. On the other hand I think this accounting shows some significant problems with SIAI and how it spends its time and money or the efficiency of how things are being done.
Just my ten cents…
Before writing a single line of code you need a precise theory that tells you what code you should write and as far as I can tell, mathematically rigorous FAI theory that would accomplish this does not exist yet – why should they write code that shall be thrown away latter, or in the worst case, produce an unfriendly AI?!
It is really good to see this report. As mentioned by Skeptic, it shows that SIAI holds itself accountable for making progress.
It also shows that SIAI had more publicly visible accomplishments in 2009 than in any previous year. Congratulations!
I agree with the first two posters that SIAI should be cranking out more technical articles and math. I appreciate that it is hard to do this on a schedule, but visible results are useful for accountability. And in fact, some technical conference talks and publications did emerge this year — wonderful! Let’s get some more!
As Rok said, we should not be expecting AGI code, since SIAI has decided to focus on safety. Probably this point should be made more clearly in public communications. But there is some code that might be useful, e.g. Uncertain Futures, or software experiments related to safely, and along with the publications it would be good to see more of those.
Keep up the good work; I’ll be contributing to the Challenge.
@Skeptic: “I was really hoping to see some technical papers, code and math”
– well I kept badgering Eliezer to release any highly dangerous AGI ideas we had straight into the public domain where anyone in any country with any agenda could use them, but he kept going on about this darn safety thing. He’s a real stickler for that…
Great comments. I, too, would like to see code. Part of the challenge, however, is that writing code before you understand what you are going to write would be analogous to climbing Mt. Everest on a whim. IMO, much of the work towards AGI and Friendly AI will be theory-level work and math, while the actual coding part will be intellectually simpler (though still very difficult) but more expensive. Writing code just for the sake of writing code for keeping up appearances at this stage of the game would probably be nothing more than a PR game where each dollar spent would probably be better spent on theory and academic papers.
Keep in mind that people have been telling SIAI to “write code!” since the beginning, and if that is all we had ever done, then there would be no Singularity Summit, no Less Wrong, no Visiting Fellows program, etc. But all these things have benefited us hugely in terms of support, and we still don’t have enough supporters to fund a serious AGI effort. You can’t fund an AGI project on $250,000/year, even when researchers are taking minimal salaries because of their dedication (which is currently the case).
Still, we invite everyone to suggest ideas for writing code that pertain directly to our core investigative goals of self-modifying goal systems and decision-theoretic features that can handle ambiguity.
All that said, I would like to see more math and papers. If I had a copy of myself that studied math for the last ten years instead of science, philosophy, and technology, then I might be able to contribute to that, but I don’t, so it falls to others.
Cranky, unfortunately we lack the support base to move forward confidently on an AGI project. We lack the people and we lack the money. Starting a full-scale AGI programming effort in 2010 would leave us broke and without additional forward theoretical progress. Building an AGI with a shoddy theory is like not building an AGI at all. It would be like building a massive doorway without the benefit of the concept of the arch. We would also run out of money before we even got going.
Still, perhaps funding some small project that makes toy code would be better than nothing. Conversely, it might be pointed to as evidence that our “AI project” is useless, even if the project were only funded by a small grant and worked on by one or two staff or Visiting Fellows. I think a better strategy is something like we’re doing here — creating a list of open projects, putting out a call for any others, and funding them individually. We lack the funds and support to launch a monolithic AI project.
As “optimist”, aka Roko, says, there is also a degree of secretiveness with some of the core ideas. Whether or not this is warranted is difficult to evaluate because I am as much in the dark about some of the ideas as anyone else is. But, I see very rich discussions on decision theory and sometimes mathematics that give me a lot of food for thought and more than satisfy me that progress is being made. A crucial feature is that the more you understand the material currently being discussed the more you see why it is more important than toy code. We try to get people interested in the nuts and bolts of decision theory because once you understand it you know why it is necessary.
As Eliezer recently commented on Less Wrong:
The whole idea of starting to produce large amounts of code in an AGI project (toy models are different) without having a complete theory where you feel satisfied with your understanding seems bad to me. It’s pro-AGI, not pro-FAI. If you still have a theory with huge holes in it, or merely the outline of a theory, you don’t want to build a sophisticated artifact whose proper functioning depends on that theory. That would be like trying to build a plane without the laws of flight, or building a steam engine without the basic laws of motion and thermodynamics, or building an atom bomb without atomic theory, or building a skyscraper without information about physics and material strength, etc. Seed AI isn’t something that we can build again and again and start again if we mess up. The penalty for not getting it right the first time should be assumed to be the extinction of the human race. Assuming anything else is not conservative enough.
In any case, if you look at item #4, we did produce five papers this year, which is an improvement on zero from last year. All because we focused on building the organization’s structure rather than just writing papers. I used to think that organizational structure would pop up out of nowhere if we did good things like writing code or papers, then I realized that organizational structure is something that needs to be invested in specifically. If we continue simultaneously building our intellectual capital base and producing papers, that seems like an ideal strategy.
You should be proud of your accomplishments over the past year.
If I can add my suggestions, you should try to be as open a research forum as possible, consider different ideas, and subject all ideas to intense scrutiny.
Sometimes a group can fall under the influence of a charismatic leader, get stuck in an attractor, and fail. What are your alternatives to CEV? It seems to me that CEV is “FAI-complete”. You need an AI that wants to understand and follow the volition of a human mind (an FAI), rather than maximize smiley faces, in order to even start talking about the general will of humanity (CEV). Once you have such an AI, it’s straightforward to make multiple copies, or interfaces, for different humans and rules on how to weigh inconsistent interests.
Optimist’s comments above were instructive. Eliezer hasn’t released his super-secret code because it the wrong hands, of dumber researchers, it would destroy humanity. Lesser researchers, with PhDs and tenure, haven’t come within lightyears of what Eliezer has developed in his abundant free time. Optimist has made my point better than I could.
I’m sorry for my negative post. You should be proud of your accomplishments this year. With an open approach to the problems of FAI, the SIAI can make great progress for the future.
“Optimist’s comments above were instructive. Eliezer hasn’t released his super-secret code because it the wrong hands, of dumber researchers, it would destroy humanity. Lesser researchers, with PhDs and tenure, haven’t come within lightyears of what Eliezer has developed in his abundant free time. Optimist has made my point better than I could.”
Hmmm…. has Eliezer told you he has code which he refuses to release or is it possible he doesn’t have any code? Also I believe the accomplishment listed was Timeless Decision theory which is not equal to FAI. So I ask what is the harm in releasing math for a decision theory which is a long way from FAI?
As for the comment about orders of researchers do you personally have a PhD or personally do research?
Gus, there is no such thing as “super-secret code”. There are only ideas which are discussed in small groups. A great many totally public ideas are frequently discussed on Less Wrong.
In criticism of SIAI, it is important to distinguish between two types. The first type is criticism that springs from skepticism over the whole project. The second is legitimate criticism from those who share our premise that Friendly AI is essential. Criticism of the first type may appear in the guise of the second — criticism over details coming from those who don’t care about what we’re doing to begin with. This would be analogous to technical criticism of the construction of a bridge from someone who doesn’t really care or want the bridge to get built anyway.
If you understand the importance of the Singularity, then you view even the smallest efforts in the right direction as worthwhile. The professionality, comprehensiveness, technicality, and communication of the work are all important to improve upon in quality, but the fundamental thing you care about is that someone is doing anything, period.
Michael:
I want to apologize for the tone of my last post. Immediately after posting it, I regretted it. I really have a lot of respect for you personally and for your blog.
Regarding your classification of criticism. I am NOT the first type of critic. I agree with the importance of FAI and am grateful for the work that you do. I am the second type of critic in that I agree with some FAI ideas and disagree with others.
I view the efforts towards a positive Singularity as extremely worthwile. I also want as many smart scientists (people much smarter than me and with more to contribute) to be motivated to help. So the way SIAI communicates its message is important to me.
As a transhumanist who is an outsider to your community, here are my perceptions: There is a great deal of interesting talk, but at the end of the day, SIAI brooks no dissent from the ideas of EY. Richard Loosemore disagreed regarding biases and was kicked off the mailing list. Robin Hansen disagreed on the hard ascent so Less Wrong schismed from Overcoming Bias. EY’s CEV is seen as the high level solution to the FAI problem. What are the alternatives to CEV and who is pursuing them?
I am nobody important and my ideas don’t matter. But if academic researchers view SIAI as closed minded, then they may not become involved with SIAI, and this may unnecessarily delay a solution to the FAI problem.
> Robin Hansen [sic] disagreed on the hard ascent so Less Wrong schismed from Overcoming Bias
Hanson disagrees with EY on the probability of hard takeoff, but not on whether it is important to attempt to do something about; last time I spoke to Robin about this I think he said something like “I am glad someone is doing something about hard takeoff AI singularities” citing a probability of 1% for hard takeoff.
Hi Gus,
Both myself and Michael Vassar have questioned the practicality of CEV and we are President and Media Director of SIAI respectively, so dissent is obviously allowed. Even EY “dissents” from his own ideas by pointing out that the option of allowing someone to check the CEV output (“Last Judge”) is a hack. If I had a good idea about Friendly AI that contradicted what EY has said, I would say it without any hesitation. It’s just that I honestly think that his ideas are really good.
The Richard Loosemore thing I wasn’t following closely, for I forget. But keep in mind that there are many other people on the mailing list that regularly criticize Eliezer’s ideas and are encouraged to stay.
As far as I know, it was always the plan to create a rationalist community that anyone could participate in with Less Wrong. Also, everyone already knew beforehand that Hanson was skeptical about hard takeoff. So the spinning off of Less Wrong has nothing to do with Eliezer not tolerating criticism.
Original, non-Eliezer-endorsed ideas about FAI are widely welcomed. Behind the scenes, I feel free to criticize EY if I think he makes some sort of mistake. The encouragement of mutually constructive criticism seems to me to be a basic precondition of any aspiring rationalist community, and our chatty little group is no exception.
Michael:
I may have been a bit harsh. I’m new to your community and don’t yet know the culture and history, though I find your work fascinating and important. It’s an exciting time to be alive!
BTW, I was reading GQ Magazine earlier today and saw an article on the NY Singularity Summit. It was right after an article about Rihanna! If that’s not mainstream …..
Congratulations on a great year!
“You can’t fund an AGI project on $250,000/year, even when researchers are taking minimal salaries because of their dedication (which is currently the case).”
How is it that SIAI’s budget is still so small after almost ten years in operation? That would make sense maybe if this was year 2 or 3 but nearly 10 something must be wrong.
“How is it that SIAI’s budget is still so small after almost ten years in operation? That would make sense maybe if this was year 2 or 3 but nearly 10 something must be wrong.”
Budgets of non-profit organizations are usually very small, in proportion to their social, political and human capital, because funding yourself by getting people to give away their money is *hard*. The Planetary Society only has a budget of $3,000,000, even though they have one hundred thousand dues-paying members.
Ben Goertzel is SIAI’s director of research, and his commercial enterprises certainly produce code, or else they wouldn’t have any products.
But these requests for code from SIAI raise the question, what would such code look like? What would it do?
SIAI’s mission is a Friendly Singularity. One might imagine this arising because someone managed to write a Friendly seed AI and run it. Recall what these concepts refer to: a seed AI is a self-enhancing AI, something which might start out less intelligent than human (but with specialized abilities relevant to software design) but ends up more intelligent than human. A Friendly seed AI is a seed AI which, in its superhuman form, is (minimal definition) not a threat to the human race and (more expansive definition) acts benevolently. And an *un-Friendly* seed AI is an unstoppable superhuman threat to the human race and possibly the end of the world.
The people who are asking for code may already be aware of this, but it cannot be stated too many times:
If you actually run a seed AI, you may have no second chance, once it surpasses human intelligence. Therefore, this is not an enterprise where you run an experiment and hope cool stuff will happen. This is more like igniting the first nuclear chain reaction, in order that benefits will thereby flow out into the world, but only after you’ve studied the physics enough to know that you won’t instead be blowing up the earth. Given our current ignorance, it would be a *highly risky* enterprise to run a seed AI. But the worldwide enthusiasm for computers and programming means that year by year we get closer to the seed-AI threshold anyway, as programs are made more and more capable, and are linked up in increasingly complex ways. Therefore, there is some pressure to try to get it right as soon as possible.
So SIAI is not really in the business simply of, say, producing models of “general intelligence” or code which implements them, though it has a vital interest in understanding such progress as it is achieved. Its job is specifically to ensure that the first AI which crosses the self-enhancement threshold (the first AI with the “motive” and capacity to enhance itself to superhuman intelligence) is a “Friendly” one. Maybe, as AI develops, and if SIAI expands, it will find time to concern itself with Friendliness in a non-Singularity context – one where self-enhancement and superhuman AI is not the issue. We may live for a while in a world where there are less-than-human AI-like agents making what we would now call moral decisions or decisions of ethical import. In such a world we will want to know what sort of AI is ethical, for the same reason we may care about morality in another human being. We don’t care about morality in others only if they are about to take over the world. The same applies to AIs.
Either way, the core mission of SIAI is Friendliness; how to make an AI ethical; how to ensure that it remains ethical even when it has sufficient autonomy to modify itself. So, returning to our original questions – what sort of code might be relevant to such a mission – it occurs to me that such questions might also be motivated by the expectation that SIAI is relevant only if it “gets there first”. It must not only have a theoretical account of what would make for a Friendly AI, it must actually produce the first one – and to do this, it must lead the way, not just in Friendliness theory, but in the general progress of AI. If those are your assumptions, I suppose there is some reason to act impatient and say “where’s the evidence of material progress?”
So a few comments regarding this outlook. First of all, given that the world is competitively advancing towards AI, it is strategically risky to try to solve the Friendliness problem by doing everything yourself, if there is another way to proceed. And there is another way to proceed, which is exemplified by the Singularity Summits: namely, talk to the other people involved in AI. Try to build some collective awareness of the risk, of the need for something like a theory of Friendliness. If all or most of the contenders in the race are aware of the danger of unFriendly AI, and consider Friendliness an essential property in a human-like artificial agent – that is an improvement on what we have now, and our odds of survival are improved. SIAI really can make a contribution even if it doesn’t get there first and build the first superhuman AI itself.
So instead we should be looking for progress in Friendliness theory and practice. Now I submit that it’s hard to even think of what code or a demo would look like here. We are talking about a large-scale property of a software system of a complexity that does not exist yet, which it is nonetheless vital to get right in advance. So what we really need is more like progress in theoretical computer science: new algorithms, new high-level concepts, new conceptual frameworks. Something big and abstract which can guide the creation of those complex future systems, so that when those systems achieve autonomy, they don’t turn around and bite us.
And to some extent this is what you see coming out of SIAI: theory, and the intention to develop theory. So I think they’re on track. Of course they need to go faster and better, but I don’t think they’re off-track, or headed in the wrong direction entirely. They’re headed the right way, it’s just quite difficult and there are many many distractions.
“Budgets of non-profit organizations are usually very small, in proportion to their social, political and human capital, because funding yourself by getting people to give away their money is *hard*. The Planetary Society only has a budget of $3,000,000, even though they have one hundred thousand dues-paying members.”
Excellent example 3 million would be more like it after 10 years of operation. Or take a look at RAND which 270 million in total support and revenues. Just because you’re a non-profit doesn’t mean that you necessarily have to have a small budget. Or take a look at Pew Charitable Trusts which has millions in donations per year.
Skeptic, I’m amazed. Many non-profits have small budgets. Like many other domains, non-profits are winner-take-all markets where a few receive all the funds because many of their donors donate for social signaling instead of a genuine desire to help. For instance, Toby Ord has found out that some African charities have a quality-adjusted life year per dollar impact 10,000 times greater than others, yet these charities receive relatively low support.
Your notion that a typical non-profit should grow to a budget of $250,000 within a few years and millions within ten years shows that you obviously have never actually worked for a non-profit or seen how difficult it is to scale. I’m pretty offended here because I’ve put much of my adult life into this non-profit and very happy by the gains that we’ve achieved.
Your reply to Tom is totally banal. Yes, a few brand-name non-profits have huge budgets, but the vast majority are unknowns with small budgets. You seem to be suggesting that the typical non-profit budget is $3 million after 10 years of operation, which simply isn’t true.
It’s possible to understand the missions of many non-profits after hearing a couple sentences from them. For us, it requires our supporters to read hundreds of pages of material, at least. Given that fact, our growth rate is actually not bad. Also, the media coverage we have received given our size would be considered excellent.
Some non-profits pass a “social signaling threshold” where multiple corporations start donating 1% of their earnings to them every year, mostly to look socially responsible. I doubt SIAI will ever reach that point, and neither will 99.9% of other non-profits. We depend entirely on shrewd individuals breaking from the pack and donating in spite of minimal social signaling benefits. They actually have to care about helping the world and believe that AI could be as much of a risk as we frequently argue it is.
You completely misunderstood me but that’s okay the issue is not important enough to me to attempt to remedy your shortcomings in understanding.
“I’m pretty offended here because I’ve put much of my adult life into this non-profit and very happy by the gains that we’ve achieved.”
Ok… and… doesn’t that go without saying I would imagine that you would no longer work with SIAI if you were dissatisfied with its results. (by the way coming from the PR person your opinion on the relative value of SIAI is a bit biased I would imagine)
On the point of banality of responses, here is a quote from you:
“It’s just that I honestly think that his ideas are really good.” (regarding EYs work on FAI)
So what? You are not an academic, you don’t know the technical side, so who cares that you may think the solution is a good one. It would be like a physicist thinking their unifying theory of physics is good, because they asked someone, with no knowledge of the technical aspects of physics, for their opinion.
“It’s possible to understand the missions of many non-profits after hearing a couple sentences from them. For us, it requires our supporters to read hundreds of pages of material, at least. Given that fact, our growth rate is actually not bad. Also, the media coverage we have received given our size would be considered excellent.”
Well I actually have read most of the content on the website and I am not attacking SIAIs mission, in fact I am totally indifferent to SIAI’s success or failure having bigger fish of my own to fry, as it were.
As for your “professional” opinions on SIAI’s current place and all, please spare me, since, I have seen what you have done and you are in no position to talk down to anyone. If you wish to continue doing so I believe that a demonstration will be necessary such as an impromptu math and neuroscience test. Perhaps we can just have you try your hand at the MIT graduate final for some math courses or maybe some neuroscience courses. (when you fail, and you will, you can try the talking down to people you don’t know in the proper context of your state of personal knowledge or lack of)
Since I am not attacking SIAI I don’t quite understand what is so offensive. I am not interested in getting into a credentials contest with you since I am not interested in embarrassing you (also since your comment is founded on misunderstanding it would not be fair). But since my comments are not appreciated I will cease.
“having bigger fish of my own to fry, as it were.”
To know some specifics about the larger specimens would no doubt be valuable for those considering what to do with their otherwise-doomed-to-be-small-fish life.
Michael is a good conduit and filter of info. No need for “whose brain is configured more optimally for [hard science]” contest. To have an elementary understanding of a field of science is nearly useless; you can’t do anything interesting that hasn’t been done before – most certainly better – with that. You need 10-to-life years of increasingly narrow niche studies to have a chance of contributing to a field, and even then only a handful of experts will fully appreciate what you know. Generally, you can’t expect people to be blinded by the brilliance of your mind, no matter how obviously superior it is in all respects to theirs. You just have to keep it between you and the science.
“…in fact I am totally indifferent to SIAI’s success or failure…”
This statement doesn’t make sense unless you’re a masochist.
“…having bigger fish of my own to fry, as it were.”
Something bigger than the fate of the multiverse? Please elaborate.
I doubt, that the Singularity will come out from the SIAI. Pretty much irrelevant what do they do.