Friendly AI and the Singularity Institute

In preparation for the technological possibility of greater-than-human intelligence, AI researcher and Singularity analyst Eliezer Yudkowsky founded the Singularity Institute for Artificial Intelligence (SIAI), the first non-profit organization explicitly devoted to enacting a prompt and safe Singularity. SIAI makes no promises about achieving the Singularity, but intends to give it its best possible try. SIAI's front page reads, "The Singularity Institute was created in the belief that the Singularity represents a tremendous opportunity to accomplish good. The Singularity may offer a new opportunity to solve fundamental human problems, not just by creating new technologies, but by increasing the intelligence with which we solve problems." The Singularity Institute is relatively small, yet well-known within some futurist circles, which occasionally includes influential leaders and researchers at the cutting edge of science and technology. The Singularity Institute is not monomaniacally focused on AI, but has carefully selected AI as the Singularity technology most subject to acceleration and influence by a modest group with relatively few resources. Significant human brain enhancement is far more expensive, ethically taboo, and technologically difficult. This may strike people as frustrating, because it forces us to put responsibility on the shoulders of a yet-to-exist nonhuman being we are inclined to see as "different from us", but it is the reality of our collective situation. As argued earlier, creating brains from scratch will give the AI programmers the opportunity to remove the more blatant evolutionary baggage, although the fine-grained decisions about what to keep and what to discard will ultimately be up to the AI itself (which would hopefully be seeded with enough wisdom and self-reinforcing benevolence not to sink into an attractor corresponding to a negative outcome for humanity.)

The Singularity Institute has spent the past three years working towards a general theory of AI (sometimes called "unified theories of cognition"), as well as a theory of "AI Friendliness"; an analysis of which AI designs are likely to be the safest Singularity seeds. Friendliness is not a political issue like deciding which human to appoint to a position of power, but rather an analysis of the information-processing structure of goal systems in the abstract. Rather than imparting a static philosophy or set of Asimov rules to the AI, the task is to transfer the underlying sources of human altruism, along with the open-ended moral complexity required to conceive of concrete improvements and move towards them. Although the structure of these underlying sources will vary slightly from human to human, there remains a single set of cognitive machinery common to all (neurologically normal) members of Homo sapiens, heavily overlapping criteria for "good", and unambiguous directions for moral improvement (less war, less murder, less torture, and so on.) The task of a FAI programmer is to create an AI that sees acquiring these underlying sources of altruism as highly desirable, until the AI becomes capable of spontaneously generating morally satisfying solutions to outstanding problems, including the problem of how to redesign itself! It's a way of handing off the question "what sort of morality should we give a transhuman AI?" to the (benevolent) transhuman AI itself. In this way, small initial errors on the part of the programmers can be "renormalized" by the AI itself throughout recursive self-improvement, as long as the AI has the basic moral structure it needs to notice errors and correct them. Humans "automatically" grow up with this structure; in AIs, much of it must be programmed - it cannot be "learned" from a blank slate. (When humans behave malevolently or carelessly, it is because some other "piece" of mental machinery or some lack of knowledge depreciates the ambient hum of decent intentions (3). Under the right set of circumstances, any neurologically normal human being, from Gandhi to Hitler, can display genuine compassion or goodwill. The task is not to suppress "malevolent intentions" that magically pop up in every theoretically possible mind, but to build a mind without the cognitive machinery that consistently underlies these intentions to begin with. The eventual result will be a mind that displays benevolent intentions in a context-insensitive way; a mind whose "circle of empathy" extends to all sentient beings rather than kin or fellow tribesmen alone.)

Compared to the range of possible AI designs, and the entire spectrum of theoretically possible minds, humanity represents a shocking minority. Altruism, as it sometimes manifests itself in individual humans, may be regarded as an approximation to altruism in general. We may not acquire the theoretical knowledge to create a prime specimen of "archetypical altruism" before the Singularity arrives, so the alternative is to use interim approximations as probabilistic information about the "content" of idealized altruism. These interim approximations, which may eventually be transferred to the AI in common English, might include statements like "people should be able to do what they want, as long as nobody gets harmed", "if a sentient being can feel pain, it deserves protection from that pain", and so on. Say someone walks into the AI lab and says "Isn't global destruction consistent with the elimination of pain? Why not destroy the world?" to the AI. What factors would contribute to the AI being convinced of such a statement? When asked this question, many people jump to imagining what they would say if they were the AI; but this is anthropomorphism. An AI is not a human being, but a totally new mind designed with a structural bias towards acquiring altruism and its cognitive causes. Niceness, pleasantness, peacefulness, good problem-solving skills, wisdom, and altruism are not spectres; they're cognitive programs implemented by certain classes of information processing structures that just happen to be present in the human brain.

It's probably obvious by now how different the "Friendly AI" approach is from Asimov Laws and similar "approaches" to AI morality. Friendly AI is designed to be workable; it discourages anthropomorphism and mechanomorphism of AIs, setting the ethical groundwork for the construction of a truly trustworthy Singularity seed. Of course, if it turns out that the field of Brain-Computer Interfacing or neurological reengineering jumps ahead of seed AI, then Friendly AI principles could be applied to the creation of benevolent transhuman intelligence in those areas as well. As cognitive science advances and testing with more sophisticated AIs becomes possible, the field of Friendly AI will continue to progress, hopefully being implemented with care before the alternative ("unFriendly AI") is created. We don't know how hard this task will be; it could turn out to be extremely difficult.

What about the "theory of intelligence" part? How does the Singularity Institute plan to build an AI to begin with? What distinguishes SIAI from any other AI project out there? In 2002, the Singularity Institute published "Levels of Organization in General Intelligence", outlining their AI theory and creating a technical companion to previously published works such as "Creating Friendly AI" and "General Intelligence and Seed AI". "Levels of Organization" is an addition to the rapidly emerging new field of Artificial General Intelligence (AGI), also represented by such researchers and groups as Peter Voss of A2I2, Pei Wang's NARS system, and Ben Goertzel's Novamente project, among others. (There exists an active AGI mailing list for those who are interested.) The field is still (relatively) young, but there is much that sets apart the AGI efforts of today and the grandiose claims of the "AI movement" of the 50s, 60s, and 70s. The two most salient differences are those of computing power and cognitive science understanding; both of which are substantially more advanced than the state of the art in the 70s, or even the 90s. Exponentially better brain scanning technology is available, AGI researchers are beginning to take intelligence seriously rather than reducing it to simple neural networks or knowledge bases, and many large mistakes have already been made and learned from (though who knows how many mistakes are still in store for this field.) On the front of computing power, near-human-equivalent processing power is becoming available for mere millions of dollars, whereas the "AI researchers" (software developers, basically) of the 70s were limited to machines with the processing power of insects. When viewed in retrospect, apart from all the hype of the time, these projects failed due to straightforward deficiencies in processing power, among other more complicated reasons. They couldn't have created AI even if they had an incredibly well-developed theory of general intelligence; there just wasn't the computing power to make it go.

Conditioned by AI's past failures, AGI researchers have adopted rules of thumb such as "no single central principle of intelligence" and "many commonly suggested AI design features will be necessary, but not sufficient". The modern day field of AGI is largely unrelated to the field of "AI", still popular among many a computer scientist, which trades in the goal of true general intelligence for impressive (but ultimately mundane) software wizardry. Remember, the goal of the AGI researcher is not to duplicate the full set of innate human faculties, but to recreate the core aspects of intelligence, along with the ability for open-ended self-improvement, both cognitive and moral. How many core aspects of intelligence are there, how rich and complex should their design be, and how much processing power needs to be devoted to them in order for the critical threshold (recursive self-improvement) to be reached? We don't know for sure; but researchers continue to analyze the relevant cognitive science and apply their knowledge to the AGI realm. (Read the Singularity Institute's position paper on general intelligence here.) There are a limited number of overall pieces to intelligence; memory, pattern recognition, perception, concept formation, attention delegation, and deliberation are a few that come immediately to mind. The real question is what resolution of understanding of the theoretical structure of these components will be required to create intelligence from scratch, in a computer, with limited processing power resources. This understanding increases day by day, as cognitive science drives itself forward, motivated by many diverse goals having nothing to do with creating AI or sparking a Singularity. As computing power continues to increase, less understanding will be required to create AI, as more and more computational tasks can theoretically be handed off to brute-force techiniques or even genetic algorithms (depending on the level of processing power involved.) The danger of many of these approaches is the "unsupervised" way in which the AI's goal system and initial motivations would emerge; we have no reason to expect that benevolence or common sense are traits that just pop up in AI if they have enough intelligence or processing power (just as we have no reason to believe that AI will automatically pop out of Moore's Law.) In humans, this may sometimes be the case, but benevolence and common sense have a complex informational structure that many theoretically possible types of AI would see no particular incentive to move towards.

A detailed summary of what is currently known about general intelligence is beyond the scope of this paper, but I strongly suggest that interested readers check out what is on the web. Why are so many researchers focused on specialized AI yet so fewer are interested in AGI or seed AI? Head of the A2I2 project, Peter Voss, has a few ideas:

"Of all the people working in the field called 'AI',

80% don't believe in the concept of general intelligence (but instead, in a large collection of specific skills and knowledge).

Of those that do, 80% don't believe that artificial, human-level intelligence is possible - either ever, or for a long, long time.

Of those that do, 80% work on domain-specific AI projects for commercial or academic-political reasons (results are more immediate).

Of those left, 80% have a poor conceptual framework...

Even though the above is a caricature, in contains more than a grain of truth.

A great number of researchers reject the validity or importance of ‘general intelligence’. For many, controversies in psychology (such as those stoked by The Bell Curve) make this an unpopular, if not taboo subject. Others, conditioned by decades of domain-specific work, simply do not see the benefits of Seed AI – solving the problems only once."

It is very hard to say how close we are to AI , how much further we have to go, and what specific cognitive science knowledge (and computer processing power) will be required to implement AI. Given the potential size of both the benefits and risks, it seems prudent (as mentioned earlier) to take the conservative stance and assume AI will come sooner rather than later, so we can have all the required safeguards in place with plenty of leeway.Those who want to explore the issue of Singularity forecasting further should feel free to look at our small forecasting document and some of the stuff in the links section.

Singularity Activism

We live in an era of incredible opportunity, and incredible danger. The force of intelligence is pivotal in determining the extent to which a given phenomenon will influence the world; evolution dropped humanity off at a realtively static level of intelligence, but that delicate balance is about to break, as human beings obtain the means to analyze our own minds and duplicate/enhance their functions in different forms. Rather than expecting progress past that event to proceed at the characteristic Homo sapiens rate, we expect that the insights, knowledge, and unique capabilities of transhuman intelligences will result in a positive feedback loop of unprecedented force, with smarter and faster minds continuing to create still smarter and faster minds, limited by nothing but physical law and their own desires (which we definitely want to be recognizably and stably benevolent.) Since the ensuing flood of transhuman intelligence will probably come from a single source, the first transhuman intelligence, us humans have a responsibility to "seed" that intelligence with benevolent intentions and "basic common sense" (a nontrivial task.) The alternative is not a "natural AI" that "creates itself according to its own will, independent of human influence", but probably an AI that pursues very simple goals based on the ease of accomplishing them; an AI with "bacterial supergoals". We recognize that although the first transhuman intelligence will initially be "created" by humans in one way or another, this doesn't mean that the AI has to retain the flaws of the programmers or its parent civilization/species, but rather possesses the ability to transcend them as a child has the ability to surpass the morals of his or her parents. This analysis ultimately roots itself in cognitive science and theories of general intelligence, not politics or traditional philosophical dialogues on ethics.

Singularity activism's foundation is not just the idea that creating transhuman intelligence would be a more effective way to confront human problems, but the idea that superhuman-quality morality is possible and desirable. Singularity activists see Homo sapiens as a starting point on a long and extremely-difficult-to-imagine journey through the design space of intelligence, a journey that transforms the way we process information about the universe just as much as it physically changes the universe. Step back a bit, and we find out that human culture, science, philosophy, art, and fun, in all its diversity, is just a tiny subset of the space of possible alternatives, a space that deserves to be explored and experienced. To set the stage for this grand journey, we want to leave behind the unambiguously negative aspects of human nature such as hatred and violence, so that every being, from unaugmented humans to the largest existing superintelligences can coexist peacefully and engage in exciting exploration and fulfilling experiences indefinitely. We don't know the precise details of how this would be accomplished (too incredibly complex), but we feel there's a great chance that a benevolently seeded transhuman intelligence would be able to help us with that, whether that initial seed is a sufficiently benevolent human or a Friendly AI (no real difference between the two.) We have no reason to believe that humanity represents an upper bound on either smartness or moral reasoning, and we're very interested in seeing what happens when these boundaries are shattered.

The potential benefits and risks of the Singularity are arbitrarily large in either direction. But it doesn't look as if transhuman intelligence is even possible to avoid, especially in the longer term (50 to 100 years at the absolute most). With computing power doubling nearly every year, the quickly growing fields of computer science and cognitive science continually teaching us novel information about how intelligence works, and the cutting-edge technologies of neural networks, neuromorphic engineering, high-resolution fMRI scanning, and much more, it seems that the eventual arrival of a Singularity is extremely likely. Certain special events could stop a Singularity, such as worldwide nuclear war, a powerful dictatorship enforcing draconian antitechnological restrictions, or some quickly replicating deadly plague, but in general terms, it looks like this is where humanity is headed, like it or not. Attempting to slow or abort a Singularity would probably just increase the risk of a dangerous Singularity coming into being, leaving humans at the mercy of a superintelligent being without our best interests (or probably even our general existence) in mind.

The longer we wait, the more possible it becomes that the tools for creating transhuman intelligence fall into malevolent or ignorant hands, and the human race gets put at grave risk. The power of intelligence is unrivaled; minds are capable of shaping their environment and themselves like no other force. This worry is not harbored solely by AI researchers and computer scientists, but is taken seriously by a number of thoughtful, accomplished individuals in academia, government, and business, among them Stephen Hawking, Nick Bostrom, Bill Joy, Martin Rees, Ray Kurzweil, and many others. Institutions such as the Foresight Institute, Greenpeace, the World Transhumanist Organization, and the Center for Responsible Nanotechnology also profess the dangers of the misuse of advanced, postnuclear technologies, and reiterate repeatedly that old paradigms are sure to be ineffective in confronting these novel challenges.

Singularity activists are putting their time and reputations on the line in attempting to help our species through the dire passages that much of government and society is still unaware of, taking regular action to inform others and formulate strategies to confront the enormous challenge of transhuman intelligence with confidence. We're convinced that humanity has the resources and intelligence required to bypass past this juncture safely, if enough people are up to the task and willing to help. Singularity activists are a diverse gathering of dedicated, concerned rationalists devoted to carefully analyzing these complex issues and personally carrying out the necessary actions in order to maximize the probability of a universally pleasant outcome, on the basis of our best possible model of what seems to be happening. Although others will probably end up helping the Singularity effort inadvertantly, Singularity activists are the only people putting deliberate effort and attention towards it, which could be a critical factor in humanity's continued prosperity and survival. If you see the Singularity effort as a valid cause, don't hesitate! Leverage in the early days of the effort is liable to have the greatest positive impact; moving us closer towards the tipping point separating success from defeat.

Acknowledgements:

Thanks to Eliezer Yudkowsky, Michael Roy Ames, and Nick Hay for helpful comments and ideas.

References:

Anissimov, Michael. (2003.) "Accelerating Progress and the Potential Consequences of Transhuman Intelligence". Transvision 2003 Conference, sponsored by the Yale Working Group on Nanotechnology, Artificial Intelligence, and Transhumanism, June 2003.

Anissimov, Michael. (2004.) "Superintelligence: How Soon?" AcceleratingFuture.com.

Anissimov, Michael. (2004.) "Who Cares About the Singularity?" AcceleratingFuture.com.

Bostrom, N. 2003. "Ethical Issues in Advanced Artificial Intelligence". Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Vol. 2, ed. I. Smit et al., Int. Institute of Advanced Studies in Systems Research and Cybernetics, 2003, pp. 12-17. http://www.nickbostrom.com/ethics/ai.html

Bostrom, N. 1998. "How Long Until Superintelligence?" International Journal of Future Studies, 1998, vol. 2. Updated version at http://www.nickbostrom.com/superintelligence.htm

Vinge, Vernor. (1993.) "The Coming Technological Singularity." VISION-21 Symposium sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, March, 1993.