How fast are the best contemporary supercomputers? All it takes is a quick glance at top500.org to figure out.

The computational power of many of these machines are rapidly approaching human equivalency, if not already surpassing it. (Depending on which estimate you use.) A common estimate for the human brain's processing power is 100 Tflops/sec (10^15 floating operations per second), but neuroscientific and evolutionary evidence suggests this may be an overestimate (Bostrom 1997). Values as low as 100 Gflops/sec (10^13 floating operations per second) have been proposed (Moravec 1998). The vast majority of computing experts, including the people at Intel, predict the continuous acceleration of available computing power through 2015 at least, at which point we would need to switch to nanocomputing or quantum computing to maintain continuous progress. The point at which human-equivalent computing power becomes available is highly significant because it puts the possibility of Artificial General Intelligence (AGI) within reach. The creation of AGI would signify the arrival of a new intelligent species, the greatest milestone in humanity's history, and either our extinction or salvation - depending on its motivations (Bostrom 2003).

The really interesting thing about engineering AGI, however, is that computer scientists won't necessarily need human-equivalent computing power to implement successful Artificial Intelligence. Implementing AGI with 1/100th, or even 1/1000th computing power may be possible. Biological evolution, being a nonforesightful process constrained by the inherent mechanics of biology, incremental adaptation, constant need for an immediate fitness advantage, weakness with simultaneous dependencies, and so on, falls far short of the efficiency and foresight that human experts can muster. Programmers, rather than implementing massive clusters of virtual neurons, will simply create programs that roughly duplicate the functionality of entire modules, linking these modules together in ways fundamentally inaccessible to biological evolutionary processes. An AGI running on the gigahertz processors of today's PCs would already possess a thinking speed far more rapid than the characteristic 200Hz speed of our neurons, and metacomputing (distributing the problem across a network, a la Folding@Home) will allow human-equivalent processing power to be reached and surpassed far before 100 Tflops/sec supercomputers (or PCs) are actually on the market. Whether or not this is relevant to AGI depends on what extent the computation making up an Artificial Intelligence can be distributed, but the central point still holds; human-equivalent computing power is around the corner, if it isn't already here.

A virtual mind capable of true innovation and self-construction could improve upon its own design, resulting in a feedback loop of a type totally absent from nature or human civilization, sometimes called strong self-improvement as opposed to the "weak self-improvement" underlying our human minds and cultures (Yudkowsky 2002). Humans possess hardware-bounded brains and rigidly set biological limits for learning and memory, and are constrained to only understand and process information that bears a correspondence to adaptive challenges in the environment where our brains evolved (Cosmides & Tooby 1997). Our brains are designed well enough to outsmart our competitors and reproduce, but not better - evolution could never afford the energy or complexity cost. Many of the design constraints we bear as evolved, biological organisms are so ingrained and universal that only recently scientists have begun noticing and analyzing them (Forster 1999).

A true AGI, by virtue of its design (intelligence-built rather than evolution-built) and native substrate (flexible computing), would be capable of conducting fine-grained improvements on sensory modalities and cognitive mechanisms, inventing entirely new types of cognition which could have taken evolution billions of years to create independently. This self-improvement loop could take off with the presence of only a single true AGI. Designing an AGI community from scratch would be unnecessary, because an AGI of human-surpassing smartness could create copies of itself, or install specialized cognitive features to subsume and exceed the collective problem-solving capacity of human groups, committees, even entire cultures or civilizations. This means the first AGI we create may be the last AGI (or the last piece of technology, for that matter) humans need to build manually. All other AGIs will build each other or themselves. Simply put, all of humanity's future well-being may be contingent on programmers getting the top-level motivations of the first AGI right. An Artificial Intelligence without any values could be a profound threat if it acquired the desire to enhance its own intelligence indefinitely, irregardless of the consequences. Asimov Laws, and other oversimplified solutions, just won't cut it.

Fostering Benevolence

Despite the frightening and overwhelming potential of self-improving, self-replicating AGI, we must consider what concrete steps we can take to improve the chances of a pleasant outcome. The first step in understanding the space of possibilities is to relinquish what AGI researcher Eliezer Yudkowsky calls the "Adversarial Attitude" - the idea that any physically possible mind is fundamentally selfish or against us, and that the only way to achieve a desirable outcome for humanity is to constrain the Artificial Intelligence from taking any possible action giving it the power to put humans at risk. But constraint and control are ultimately failing strategies. It is prudent to assume AGIs will eventually become independent agents, capable of rewriting their own source code and extending into the world with flexible robotics. The construction of the first Artificial Intelligence must be approached as an act of creation, not an act of coercion. We just might be able to create AGIs that "constrain" themselves from taking actions that harm humans - not because they are being "controlled", but because they genuinely want to act that way, just like any human altruist.

The ultimate issue is about the delicate details of design, not forcing AI researchers to behave in a certain way or obey certain regulations. AI is not a typical technology, and getting a benevolent AI design correct, as fast as possible, seems like a better plan than trying to regulate bad designs out of existence. Benevolent AI could help us with the risks of poor AI design, give us advice for improving our methods, and much more. AIs are a different species than humanity; they are designed, not evolved. Creating AIs that are genuinely benevolent is more a matter of transferring over moral complexity, the human-universal complexity underlying our judgements between good and bad, than arguing down or intimidating a potential rival [Yudkowsky01]. This discipline is often called the field of Friendly AI.

The first AI needs to be a good person. As the first non-human mind embarks on a self-improvement trajectory that could quickly lead to full-scale superintelligence, it will need to make intelligent, benevolent, and altruistic choices at every step of the way (especially choices concerning how it should modify itself). This becomes especially important at the level of superintelligence, where the slightest level of indifference towards sentient life could easily result in millions or billions of deaths. The first step is to throw "Asimov Laws" in the trash. Command lines phrased in human English are susceptible to flawed interpretation, ambiguous meaning, and excess anthropomorphism. Asimov Laws also target externally observable behaviors rather than the internal cognitive operations giving rise to these behaviors - tactics suited to constraining the behavior of other humans rather than building altruistic minds from scratch. We want an AI that understands the spirit - not just the letter - of morality, in the same way that we humans do, and carries that moral model to supermoral heights, making continuous improvements on benevolence as well as intelligence. An AI undergoing rapid, recursive self-improvement (a "hard takeoff") needs a level of self-control and moral judgement that humanity has not yet reached, and must make decisions that distribute the benefits of superintelligence (including the opportunity to become superintelligences themselves) to all sentient beings.

One essential design feature of any benevolent AI is a non-observer centered goal system. Evolution crafts goal systems centered around the observer because brains, so far, have always been constrained to remain within their respective organism (which happens to be a locus of selection pressures). If brains were independent from the loci of selective pressures, then they may have developed in radically different ways, but this currently isn't the case. The point is that self-centered goal systems emerge naturally in biological evolution, but by no means emerge inevitably in all types of minds. To bypass humanity's characteristic tendencies towards selfish (knowingly or not) goal-pursuing, Yudkowsky suggests the Friendly AI model; a framework for benevolent AI that eschews observer-biased goalmaking, instead centering the AI's goal system around the desire to do what's right (volitionism - respecting "what people want" - is his best guess).

The "selfish instinct" has not been suppressed or perverted - it just isn't there to begin with. Which brings us to our next feature - lack of instinct. Every decision or action should flow from the desirability of the AI's supergoal (benevolence) rather than limited sets of spontaneous, narrow-domain-adapted behaviors, as in all presently existing animals. Instincts evolved because they were easy for evolution to engineer, not because they are optimal. An AI needs to be selfless, ready to shut itself off or engage in full-scale self-revision in order to keep pace with the highest, most benevolent, non-invasive morality it can formulate, a morality that coincides with humanity's greatest ideals, including our ideals about open-ended moral improvement.

What Kinds of Goal Systems Are There?

People nowadays, and throughout history, have been wary of words like "morality" and "good". Does true "good" exist? What is the correct "morality"? Before trying to answer these questions, let's put them into the context of the human design of nonhuman intelligence, or AI. With AI, we have the opportuntiy to craft a goal system from scratch - every humanly engineerable goal system is potentially within reach. What kind of goal systems could we engineer? We could engineer goal systems with the supergoal of immediately self-destructing. We could engineer goal systems that value nothing but an arbitrary object or mathematical equation, and acts only to increase the plentitude of that object [Bostrom03]. We could engineer goal systems based on pleasure feedback. We could engineer goal system that are selfish, as most animals designed by evolution are, goal systems that are psuedoselfish, as humans are, or goal systems that are selfless, as nothing has yet been. The goal systems we've seen before - human goal systems - make up an incredibly tiny cluster of the vast space of engineerable goal systems.

One of the fundamental discoveries of evolutionary psychology, and cognitive science in general, is that human consciousness and introspection only perceives the tip of the iceberg of our full cognitive complexity. When we conduct an action like picking up a cup, an immense amount of neural circuitry is busily at work, calculating the spatial position of the cup, the ideal form our hand should be in while grasping the cup, and thousands of other variables that never enter into conscious attention. Our notion of "what's good" works the same way. Human intuitions about good are overwhelmingly complex. It is necessary that those intuitions are duplicated in the first transhuman intelligence, where directly, or through unambguous pointers to the roots of these intutions. "Unambiguous pointers" means that the first transhuman may not initially possess all the complexity inherent in human altruism, but does possess the knowledge and desire to seek them out. The main danger is that the creators of the first transhuman intelligence will be in a rush, lacking the necessary knowledge to transfer over the essential priors of human altruism, and end up creating a transhuman intelligence with goals too simple to permit configurations of matter as complex as sentient beings enjoying themselves. Sometimes such excessively simple goal-sets are called "bacterial supergoals".

Human goals are complex. "Being nice to your fellow human" requires a detailed modeling of what "fellow human" is, what that fellow human's wishes are, and how to carry them out. For humans; this sort of operation is natural - although we certainly fall short of the theoretical optimum of kindness. Despite our ability to do so, we are very rarely entirely altruistic. The first transhuman intelligence, due to its capacity for recursive self-improvement, will require a standard of altruism far in excess of the human norm. Superintelligence must go hand in hand with supermorality. And this standard does indeed seem possible. Some "Friendliness" (AI morality) researchers have called it "the human moral frame of reference". We want AIs adept at empathizing with humans, and AIs that invoke human empathy and care. We must not be misled by the fact that kindness and respect can seem relatively simple within ourselves (when we choose to do it) - that apparent simplicity is deceiving, and it will take a lot of work to duplicate the essential prerequisites of empathy and altruism in a synthetic mind. But it does seem like it can be done.

Unfortunately, the word "AI" can throw a negative connotation on the whole effort of Friendly AI. It's worth keeping in mind that in consideration of wide range of possible post-Singularity societies, the word "AI" becomes useless quickly. The word "AI" should be reserved to describe synthetic minds primarily bearing the human design signature, rather than the design signature of evolution or recursive self-improvement. The latter should be referred to as "transhumans", "nonhumans", or "superintelligences". By no means should we expect this qualitatively new types of intelligence to necessarily possess "mechanical", "selfish", or "monomaniacal" tendencies - unless these tendencies emerge automatically as a result of the initial set of programmed goals. "Programmed goals" need not be inflexible; even in the presence of top-down techniques - in fact, comparatively top-down design techniques will be required if we want a goal system exhibiting precise "flexibility" in ways we see as making sense. Humanity's characteristic goal system flexibility is only flexible in relatively narrow, evolution-determined grooves - the emergent result of the sophisticated, interconnected operation of suites of (cognitive) complex functional adaptations. We shouldn't be ashamed that our "wild tendencies" don't actually incorporate "true chaos" in their workings; like all complex machines, they must possess a massive amount of underlying order for smooth functioning.

The required design features for human-friendly Artificial Intelligence are many, but it does indeed seem that certain progress has been made. The Singularity Institute of Artificial Intelligence (SIAI) has published a book-length text outlining the requirements behind a robustly benevolent AI morality, "Creating Friendly AI" (CFAI). One example of a feature described in CFAI is probabilistic goals and supergoals - goals symbolized as probabilistic guesses, not absolute dogmas, and the continuous analysis of morally relevant perceptual input, revising the probabilities of goals and beliefs accordingly. CFAI's proposals view future AIs as potential moral agents, and observe that conditioning an AI purely through positive or negative feedback fails to foster the sort of fine-grained decision-making capability that comes through critically analyzing every action and aspect of a goal, while presenting the danger of nonrecoverable errors in the code, such as inescapable pleasure loops. All in all, there is a mountain of work to do in this controversial, acutely urgent new field, and I hope the finest minds on the planet contribute to it in the coming years.

References:

Bostrom, N. 2003. "Ethical Issues in Advanced Artificial Intelligence". Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, Vol. 2, ed. I. Smit et al., Int. Institute of Advanced Studies in Systems Research and Cybernetics, 2003, pp. 12-17. http://www.nickbostrom.com/ethics/ai.html

Bostrom, N. 1998. "How Long Until Superintelligence?" International Journal of Future Studies, 1998, vol. 2. Updated version at http://www.nickbostrom.com/superintelligence.html

Cosmides, L. & Tooby, J. 1997. "Evolutionary Psychology Primer". At http://www.psych.ucsb.edu/research/cep/primer.html

Forster, M. 1999. "How Do Simple Rules ‘Fit to Reality’ in a Complex World?" Minds and Machines 9: 543-564. At http://philosophy.wisc.edu/forster/papers/Fast&Frugal.pdf

Yudkowsky, E. 2001. "Creating Friendly AI". Publication of the Singularity Institute. At http://singinst.org/CFAI/

Yudkowsky, E. 2002. "Levels of Organization in General Intelligence". Publication of the Singularity Institute. Ben Goertzel and Cassio Pennachin, eds. Real AI: New Approaches to Artificial General Intelligence (in press). At http://www.singinst.org/LOGI/