Who are Technological Singularity Activists?

©2004 by Michael Anissimov

1 2 3

Who are Technological Singularity Activists? Singularity Activists are scientific researchers, theorists, and laypeople working on a long-term project to create an altruistic, smarter-than-human (transhuman) intelligence. Such an hypothetical achievement has been called a "Technological Singularity" or simply "the Singularity" by futurists, since mathematician Vernor Vinge coined the term in 1991. The term "Singularity" is capitalized to distinguish it from singularities in math and physics. In general, the term "singularity" refers to a point in any system where conventional laws stop applying, and new theories must be formulated in order to explain observed events. As the laws of physics behave unexpectedly around gravitational singularities (black holes), many human expectations and assumptions would cease to apply when a transhuman intelligence is created and begins to affect the world. This applies especially if the transhuman intelligence produces offspring, or improves its own underlying hardware and capabilities. (Inevitable, given enough time.)

Such a transhuman intelligence would not necessarily adhere to human limitations. For example, a transhuman intelligence could potentially have a brain composed of hardware implementing cognitive processes at millions or billions of times the human rate. Because of the technologies it could develop and the humanitarian opportunities it could offer, benevolent transhuman intelligence has been taken up as a specific goal by Singularity Activists. Serious efforts to implement the Technological Singularity first began in 2001, with the founding of the Singularity Institute for Artificial Intelligence. The original proposed methods include Artificial Intelligence (AI) or the neurological enhancement of human brains ("IA", for Intelligence Amplification). There are a number of discussed approaches to both methods. These include brain-computer interfacing, nanomedicine, neurosurgery, uploading, and seed AI.

We know that the human brain is a complex organ, a particular type of biological machine. We currently know a great deal about how a brain is physically constructed, which parts of the brain-machine perform which functions. However we know much less about the details of its operation, how the parts work together to make a living, conscious person. If we imagine the physical structure of a brain to be like a computer, then we know almost everything about the brain-hardware but very little about the brain-software. Once we discover enough about the software of the brain, we will be able to create a computer software program to do the same thing. This software program would be an intelligent, thinking, conscious person in the same way as human beings are. Just like other software programs, we could copy it and upgrade it, and make it much smarter than the people who created it.

Singularity Activists argue that creating such a smarter-than-human intelligence would be an effective way to confront current and future risks such as nuclear war, destructive nanotechnology, environmental degredation, and human-unfriendly AIs. Smarter-than-human intelligence could take the lead in helping us accomplish humanitarian goals such as minimizing poverty and suffering, and assist in channeling the immense power of technological progress to benefit everyone. By exploiting the leverage point of intelligence as a unique force, we seek to have a greater effect than an exclusive focus on developing better technology with our curent level of intelligence. Humans should not be the final word when it comes to advanced intelligence. Faster, better, more efficient, wiser, more rational and flexible forms of intelligence should be possible. The ability to engineer an intelligence greater than ourselves may one day be seen as a special human capacity, one that distinguishes us from animals.

Most Singularity Activists have chosen Artificial Intelligence as a preferred Singularity path (rather than human Intelligence Amplification) because AI seems much easier technologically than IA, meaning we will probably have to confront its' creation first, whether we like it or not. Rather than reacting anthropocentrically and rejecting the AI-first scenario as unacceptable, Singularity Activists plan for such a scenario by developing theories for benevolent goal systems, a field sometimes called "Friendly AI". There are reasons to believe humans will eventually be able to program AIs equal or superior to humans in the domains of trustworthiness, benevolence, and compassion. Greater intelligence does not necessarily entail selfish behavior. The essential algorithms underlying altruistic behavior are complex, but with the appropriate knowledge, it should be possible to duplicate them in an Artificial Intelligence system. This would open up the possibility of creating truly non-selfish and philanthropic beings, of the sort that would never emerge under the harsh conditions of natural selection. As such beings begin to engage in self modification, they would make choices based on their initial motivations and goals. A benevolent AI wouldn't want to turn itself into a selfish AI, because such revisions would contradict its benevolent values.

Humanity's greatest successes come from our ability to solve problems. But our problem-solving ability is not infinite, we have limitations. All of humanity's difficulties, past and present, can be attributed to limitations in our problem-solving ability. Another way of saying this is that human intelligence has limits. Singularity Activists are attempting to improve global quality of life by increasing the intelligence we have available to solve problems. More intelligence gives us better solutions to problems, whether they be social, humanitarian, technological, practical, or emotional. It would be nice if we could create a substantially better society using the intelligence of humans alone, but we've been trying that for the past 50,000 years, and progress is slower than it should be. Creating a benevolent Singularity, increasing our available problem-solving intelligence, is both very feasible and very desirable. It is high time we found better solutions to humanity's problems, and creating smarter-than-human intelligence is a very powerful way we could do that.

Scientific Materialism and Rationality

Almost universal among Singularity analysts and activists are a few basic assumptions about the nature of the world. The first assumption is scientific materialism. Scientific materialism, broadly defined, is the idea that the observable physical universe follows natural laws. If we want to become more effective and successfully improve our lives and the lives of others, we can do so by studying the causes and effects which give rise to the complex forms making up our universe. Our primary methods will be science and reason. The assumptions and knowledge conferred to humanity by scientific materialism are responsible for the computer you're using, the car you drive to work in, the manufacture and transportation of most of your food, clothing, and furniture, the architectural principles used to construct your home, and much more.

The next assumption is that having true and accurate beliefs is desirable. That believing false or inaccurate things, whether comforting or not, hinders our ability to understand reality. We can call this philosophy "committment to truth-seeking" or "normative rationality". This philosophy argues that any goal we want to accomplish is best pursued by improving the quality and accuracy of our thoughts, rather than engaging in wishful thinking, overestimation, unjustifiable optimism, blind confidence, and so on. Devotion to truth-seeking is not at odds with love, sensitivity, empathy, or emotion, but can work in parallel with them to achieve the maximum in personal fulfillment, effectiveness, and sanity. Devotion to the truth is also a force that helps us arrange our life goals into a coherent framework, and create realistic plans for achieving these goals. Obviously, people who discuss the truth need not overweight confidence in their own ideas; we can and should calibrate our beliefs to compensate for overconfidence.

Existential Risk

Scientific materialism and a desire for accurate beliefs. Keeping these relatively demanding constraints in mind, we begin by asking "what are some strategies for preserving ourselves (humanity), while maximizing our capacity for self-direction, freedom, and the pursuit of happiness?" This is a Big Picture question that humans have been asking ourselves throughout history, both explicitly and implicitly. Recently, eminent thinkers such as Martin Rees and Nick Bostrom have been alerting us to the looming possibility of existential risks, near-future technological disasters that threaten to eliminate humanity or severely curtail its potential. However, the majority of humans are concerned with local questions, about issues that tangibly and immediately effect them. There is nothing wrong with this - for ages, society has been steadily improving as a result of humans working to build up their own lives and the lives of those immediately around them. But as we continue to develop more advanced technology, we need to be considering global, long-term strategies and options. These problems do not have the political valence to attract much attention, which is unfortunate because the potential punishment for unpreparedness is so great. The investment of even a small percentage of our planetary intellectual resources would be well worth the price to head off disaster.

People have been thinking about humanity's destiny since the dawn of time, but it wasn't until the nuclear bomb that the possibility of complete annihilation concretely presented itself. A variety of solutions were attempted - a stronger United Nations to preserve world peace, the construction of bomb shelters and autonomous underground habitats, even the idea of a Doomsday Machine (brought up in the classic film Dr. Strangelove), an apocalyptic network of nuclear bombs designed to automatically detonate in the event of an attack, actually implemented with the acronym MAD - Mutually Assured Destruction. Over the past few decades, many scientists and activists have pushed for complete nuclear disarmament, represented today by organizations such as the International Network of Engineers and Scientists for Global Responsibility. Yet humanity's safety is still at risk - the technological prerequisites for the manufacture of nuclear weapons are becoming ever cheaper and more available. More dangerous technologies are on the near horizon.

As our level of technology continues to advance rapidly, our empathy and intelligence remain limited to the human level, resulting in a most dangerous type of mismatch.

Nanotechnology

In recent years, the prospect of nanotechnology has created popular concern about technological disaster. Nanotechnology is the ability to manufacture molecule-scale devices and program them to perform useful tasks, in aggregate, on the macro-scale.

A nanometer is a billionth of a meter, and nanomachines would be built out of components on that scale; individual atoms and molecules, each in their place. Nanomachines could conceivably manufacture a variety of products at extremely high levels of efficiency and performance. Mass manufacture of high quality products would be possible due to the efficient nature of nanomachines and other advantages of molecular-scale construction. Although the danger of runaway self-replicating nanomachinery - the "grey goo" scenario - has been overstated, deliberate or politically-motivated misuse of nanotechnology is almost certain to be a major threat. All other technologies have been used for both good and bad purposes, and nanotechnology will be no exception. Preliminary projections of the capabilities of nanomachinery from known physical principles suggest that their impact could be greater than several Industrial Revolutions packed into a single decade - beyond anything humanity has yet seen. With sufficient knowledge and software, there is nothing preventing us from programming ensembles of nanomachines to build mansions for everyone on Earth, transform dirt into food, completely clean up the environment, extend human lifespan indefinitely, kickstart a new space age, and other achievements we cannot yet imagine. Nanotechnology's massive potential has been described in publications from Popular Mechanics to the New York Times.

Leaders in preparing for the arrival of nanotechnology are the Foresight Institute and the Center for Responsible Nanotechnology (CRN), who emphasize the potential benefits of nanotechnology while formulating policy to minimize the danger. CRN has proposed a safety paradigm centered around "nanofactories", tabletop units with inbuilt constraints to prevent their misuse, as opposed to the commercialization of general-purpose or free-floating nanomachinery. Nanofactories would be a prudent approach to nanotech policy because they could (theoretically) be countable, limited, visible, standardized, and sellable. Some of the easily imaginable alternatives, such as free-floating nanomachinery, could cause serious problems if used in a soviety not prepared to handle that level of technology. As nanotechnology is steadily being developed, very few people are confronting these historically unfamiliar challenges and the dangers they inevitably represent. Nanotechnology will open up a huge design space for new products, and we lack the intelligence to forsee the consequences of these innovations.

On their website, CRN describes a hypothetical nanofactory; "It seems like magic. A small appliance, about the size of a washing machine, that is able to manufacture almost anything. It is called a nanofactory. Fed with simple chemical stocks, this amazing machine breaks down molecules, and then reassembles them into any product you ask for. Packed with nanotechnology and robotics, weighing 200 pounds and standing half as tall as a person, it can produce two tons per day of products. Control is simple: a touch screen selects the type and number of products to produce. It costs very little to operate, just the price of materials fed into it. In one hour, $20 worth of chemicals can be converted into 100 pairs of shoes, or 50 shovels, or 200 cell phones, or even a duplicate nanofactory!" Amazing, isn't it?

The problem, of course, with being able to build nearly any product or structure from raw materials for negligible price is the huge potential for misuse. Nanofactories must be completely unable to manufacture any sort of dangerous or illegal product, including toxic chemicals, weapons, currency, copyrighted items, drugs, tools that could be used to tamper with the nanofactory itself, tools for creating new nanofactories without constraints, or tools that could be used to construct free-floating or general purpose nanomachinery. If the rules were violated, even slightly, and illegal or dangerous products were manufactured (such as nano-weapons or additional nanofactories), law enforcement and other security systems would need to possess the capability to quickly neutralize the threat (with minimal damage). This applies on the international plane as well as the local plane. If nanofactories are 99.999% resistant to tampering, and a hundred million models are distributed within a year, and one out of every hundred customers attempt to tamper with their nanofactory within that year, then the probabilities indicate 100 successful instances where nanofactory constraints are circumvented within that year alone. This seems like a small number - but what about when nanofactories become widely distributed, and more advanced? Constraint-free nanofactories could self-replicate exponentially and travel through black markets. With nanotechnology, a single mistake could turn out really really bad. Is human management and regulation enough, or might smarter-than-human intelligence be able to offer better, fairer, quicker, more elegant solutions?

One of the most interesting aspects of CRN's analysis is their projected timeline for the development of nanotechnology; "less than twenty years off—maybe less than fifteen". They state "we have not seen anything to make us believe that a five-year $10 billion fabricator project, starting today, would be infeasible, though we don't yet know enough to estimate its chance of success. Five years from now, we expect that a five-year project will be obviously feasible, and its cost may be well under $5 billion." Others working on the enabling technologies for nanotech agree. Most recently, CRN's Director of Research Chris Phoenix published "Design of a Primitive Nanofactory", outlining the steps between a low-level nanotechnological assembler and a fully functioning desktop nanofactory. One aspect of nanotech likely to vastly accelerate deployment is the powerful self-applicability of intermediary nanotech products to their own advancement - for example, if someone had low-level nanotechnology permitting the manufacture of bulk nanocomputers, those nanocomputers could then be used in rapid prototyping for designing the next round of nanotech devices. CRN states that nanotech self-acceleration is so significant that even a few months delay could be sufficient to leave a non-nanotech nation helpless against a nanotechnological rival.

Nanotechnologists acknowledge that the social, technological, and political challenges accompanying nanotechnology are likely to be great. The Center for Responsible Nanotechnology, along with the larger Foresight Institute, have performed the deepest analyses of the possible dangers of nanotechnology thus far. Potential dangers include disruption of the economy, perpetuation of unnecessary poverty due to unfair costs, terrorism, backfiring strategies, societal turbulence, environmental damage, and arms races, and many others. Mark Gubrud, of the Center for Superconductivity Research, presented a talk and paper "Nanotechnology and International Security" at the Foresight Institute's fifth conference on molecular nanotechnology, analyzing the potential for arms races after the development of nanotechnology, also pointing to Artificial Intelligence (AI) as a relevant factor. AI will be the next major danger we'll be discussing - but like nanotechnology, the potential benefits are just as massive as the dangers, and avoiding the development of these technologies seems implausible (we will eventually be forced to confront them).

Artificial Intelligence

The potential peril and promise of Artificial Intelligence has been consistently underestimated, partially because it has a history of being over hyped, and people have become desensitized. AI is a philosophically sticky topic - many incorrectly underestimate the potential power or intelligence of AIs because they feel personally uncomfortable with the idea of a non-human being possessing human-level or superhuman intellectual or physical capabilities. Avoiding this irrational human-centric, or "anthropocentric" type of thinking is the first step on the road to confronting the challenges of AI responsibly and safely. Admitting that an AI could become many times smarter than us humans is not betraying humanity; it's simply being realistic.

It has been said that AI has not yet been developed mostly because the required computing power is unavailable. Theories that adequately describe intelligence have also not been developed, and the ones that do exist are still incomplete and unproven. The larger estimates of a human brain's computational capacity are 10^17 operations per second (ops/sec), with more conservative estimates at around 10^14 ops/sec. Some even assert that running the bare essentials of general intelligence might require 10^12 ops/sec or less, which is within the range of today's computers. Singularity analyst Nick Bostrom calculates, "The human brain contains about 10^11 neurons. Each neuron has about 5*10^3 synapses, and signals are transmitted along these synapses at an average frequency of about 10^2 Hz. Each signal contains, say, 5 bits. This equals 10^17 ops (...) The true value cannot be much higher than this, but it might be much lower. There seems to be great redundancy in the brain; synchronous firing of large pools of neurons is often required if the signal is not to drown in the general noise." It's also worth keeping in mind that human brains were built by evolution, an often elegant but inefficient designer lacking foresight, which is restricted to using the materials and methods it has available at the time. Evolution has the advantage of being able to run massive series of tests (generations) over geological time periods - yet it is not able to plan what it is building, and is subject to a long list of unavoidable constraints. (Some of which are described further below, and in the document "Relative Advantages of Computer Programs, Minds-in-General, and the Human Brain".)

In mere thousands of years, humanity's general-purpose intelligence has allowed us to colonize a wide range of terrestrial environments, travel in the upper atmosphere and outer space, and refine and use hundreds of thousands of chemicals that biological evolution is too blind to utilize. Our intelligence has the potential to wipe out all Earthly life, or in contrast, to spread the descendants of Earth throughout the entire universe. Intelligence has the ability to be more moral than evolution (for example, in curing polio), or less moral (in deliberately wiping out millions of people). It's the intelligence that gives us these abilities, not the power of our hands or technology in particular. Our hands and our technology are what directly implement our will in the world, but these generally follow the commands of our intelligence. Considering that we may eventually be sharing this planet with a totally new form of intelligence, it seems worthwhile to examine some of the likely differences between humans and early AIs and the processes that have or will create them. Humans were created by evolution and natural selection, AIs will be created by intelligent design.

Let's begin by examining the process that built humans, biological evolution. Unfolding proteins, the only possible building blocks of evolution, are required to follow a long checklist of rules, constraints, and biological limitations throughout the process of brainbuilding. The selection process itself is tightly constrained by a variety of factors. In evolution's kingdom, each new mutation must create an immediate benefit for the holder, or the mutation will not persist into future generations. Not a tiny benefit, either, but a recurring benefit that persists long enough for the new mutation to spread throughout a population and become species-typical. Organisms must evolve with their breeding circles; they cannot evolve independently. If an isolated individual happens to stumble upon a significant fitness peak, it will do nothing unless that mutation is probable enough to recur frequently throughout the entire breeding circle (which often extends to the entire species). Evolution can only make use of the fundamental building blocks that originally initiated the whole process - DNA and proteins. It can't handle simultaneous dependencies, that is, evolution will more easily stumble upon a single minor beneficial mutation than two mutations which confer a major benefit when taken together yet create a minor inconvenience when they occur in isolation. Human programmers possess none of these limitations, and neither will the AI minds they create.

From a cognitive architect's standpoint, DNA, proteins, and neurons are materials that are mediocre at best for building an intelligent mind. Specialized nervous system cells (animal neurons) are large, cumbersome, and slow compared to the ideal construction pieces for a brain, which might employ optical or quantum computation rather than electrochemical signals, and be organized and built in completely different ways. Human neurons are simply tweaked versions of the nervous system cells that have been around for hundreds of millions of years; evolution never had a chance to step back and say "is this really the best we can do?" Evolving a new organism is like rebuilding an entire plane in mid-flight - it must be done extremely carefully, conservatively, and slowly. Organisms do not need to be the best possible in order to survive and reproduce; they simply need to be slightly better than their competitors. If a simple-and-dirty evolutionary strategy works, it can crush the competition for millions upon millions of years, prohibiting the creation of more complex or sophisticated designs. That is the nature of the process that built the 3-lb lump of specialized animal cells known as the "human brain".

But AI minds will not come from evolution. They will come from intelligent designers. Humanity will stand with respect to AI in the same way that evolution stands with respect to humanity. The result will be that AI minds and human minds will be organized and built in completely different ways. "Human-like minds" are incredibly complex and precise objects, multipurpose machines with a great number of interlocking problem-solvers, attention-delegators, priming mechanisms, and so on. Evolution did all the work, but due to our limited introspective abilities, we are only able to visualize a small portion of the total complexity. The precise set of cognitive machinery that makes us human was crafted by evolution over a period of millions of years; but AI will be crafted deliberately and holistically over mere years or decades. Human programmers will use different design techniques than evolution did, on different timescales, with different materials, code, purposes, motivations, demands, and pressures.

AI would be a qualitatively new kind of entity; it needn't possess or display humanlike thinking or behavior unless we program it precisely that way. We shouldn't expect the first AI to behave or think in ways similar to the spirits of myth, the aliens of conspiracy theories, the AIs of science fiction, or anything else we have previously confronted. We shouldn't expect AIs to display love or hate or competitiveness or rationality or anything else unless we program it. We shouldn't expect AIs to lean away or towards any of these complex traits unless we program them that way. The selfishness that characterizes evolved organisms might be totally absent from AIs, or amplified to enormous proportions - all depending on the initial conditions and how they develop as the AI acquires the intelligence and ability to revise its own source code. When trying to predict the behavior of future AIs, we must not anchor our models in familiar fictional examples and adjust from those, but create totally new models from scratch based on more abstract principles culled from what we know about how brains work. For example, an AI with a level of cognitive complexity granting it approximately "human-similar" general intelligence, but with the unique capacities to manipulate its own source code, think using components millions of times faster than human neurons, add new computing hardware, and so on, would be anything but "human-similar"; it would be thoroughly transhuman. Conclusions like these follow very quickly from what we know about how brains work.

The processing speed of computers has been doubling regularly every 12-18 months for the past few decades, and there is every indication it will continue to do so for another decade or two. Intel's roadmap calls for 20GHz chips in 2007, a slight acceleration relative to the doubling periods of the past decade. Parallelized computing techniques are allowing us to make more out of the processing power we have, and millions of people are being introduced to computers and the Internet every year, which greatly encourages lower prices and more research and development. Since the speed of thought in a brain is dictated by the speed of its cognitive components, this abundant computing power will make it possible to run a vastly accelerated intelligent AI once we possess an adequate program. Such an AI could undergo thousands of subjective years of intelligent thought in mere minutes. Computing power to spare would also allow the overclocking of cognitive components; an ability foreign to nature, the magnification of performance through means other than writing specialized code. How would you score on an IQ test if your visual cortex were given extra computing power for manipulating mental imagery, or if your working memory had enough power to easily store 500 distinct pieces of information relevant to a problem rather than just 5? Once you pull the algorithms of thought out of the human brain and onto a flexible, high-performance computing substrate, there is a lot more you can do with them.

These deep differences between humans and AIs have correspondingly deep implications. One is that more intelligence in an AI need not correlate to more humanness. A very intelligent AI could be quite unhumanlike or humanlike, and still only in very specific ways. Another implication is that a seemingly small upgrade to a subhuman AI could quickly result in an AI of superhuman intelligence - since we don't know everything about how intelligence works, we wouldn't be able to predict which upgrades would give rise to which levels of intelligence, except in very vague terms. Small modifications could have big effects - we just don't know. Still another implication is that very intelligent AIs need not be biologically inspired except in the most general sense. Since digital processing encompasses all the possibilities of analog but not vice versa, a superintelligent AI wouldn't need to be implemented on a neural network. Another implication is that future AIs should not be viewed as pieces of technology or mere adjuncts to humanity, but as potentially independent agents. The last and most important point here is that since an AI will initially have no tendencies, thoughts, dreams, or inclinations outside of those derived from the brainware we give it, the programmers will have a great moral responsibility to ensure that the first AIs are good people with altruistic values and a sufficient understanding of moral philosophy that they can transcend errors in their initial programming. Since AIs will be the ones making the choices about how to reprogram themselves as they develop, benevolent AIs will reprogram themselves to be more benevolent, while selfish AIs will reprogram themselves to be more selfish.

Technological Difficulty of Real AI

How far off is real AI? A hundred years? A thousand? That's what popular opinion would suggest, but many experts are pointing to the ballpark of ten to thirty years, and they certainly aren't making these claims without sufficient evidence to back them up. In his 1997 paper, "How long until superintelligence?", Oxford philosopher Nick Bostrom (who has also published in the field of computational neuroscience) made some of the most bold claims yet, presenting the case that true Artificial Intelligence is likely to emerge within the first third of this coming century, quickly precipitating the arrival of still smarter intelligence and genuine "superintelligence"; "an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills". This is not idle talk - Nick Bostrom is not a part of the Artificial Intelligence community, and has no particular incentive to hype its arrival. Bostrom's opinions are echoed by dozens of figures in business, academia, and futurism, such as Ray Kurzweil, who has argued that molecular-level scanning of the human brain will allow us to fully duplicate its capabilities by 2030. There is robust evidence that the complexity of intelligent behavior doesn't come from a near-infinitely complex software (as many assume), but through the combinatorial interplay of a basic set of components. This view of the human organism as a Swiss Army Knife of adaptations was described by evolutionary psychologist Steven Pinker in The Blank Slate.

Possible AI construction methods range across a continuum from perfectly emulated human brains initially embedded within human-familiar virtual bodies and environments, to more unusual designs only remotely inspired by biology. The precise, molecular-level emulation of the human brain would serve as an upper bound for the difficulty of achievement of AI - but we can't say much about a lower bound. The limitations of evolution and biology relative to human ingenuity and computing devices suggest that the lower bound could be quite low indeed. The potential power and intelligence of early AIs makes it prudent to be conservative about their possible arrival date - the same thing holds true of nanotechnology - realizing that the issue is not a contest regarding how enthusiastic one should be over the eventual arrival of a given futurist technology, but a concrete strategic issue, relevant to humanity's survival over the coming decades. Consider this: if the US government is willing to spend billions to lower the likelihood of a terrorist attack, then how much would it be worth to lower the likelihood of a risk such as dangerous AI? At this point it seems useful to remark that arguing for the near-term likelihood of nanotechnology and AI has absolutely nothing to do with loving technology, escaping the mundane routines of daily life, or an obsession with science fiction. These forecasts spontaneously emerge from a systematic review of the literature on knowledge acquisition, evolution, computing, programming, computational neuroscience, and other technical areas. Nick Bostrom and Ray Kurzweil are simply two people who have reviewed a lot of that literature.

Data related to incidents of microcephalia, hydrocephalia, and hemispherectomies suggest that it is possible to have a normal IQ with a very small or unusual brain (some cases involved individuals with brains weighing as little as those of H. erectus!) Our brain evolved to be massively redundant, which also strongly suggests that the human brain is much more complex than it needs to be to produce intelligence. Pre-intelligent AIs could assist in the construction of fully intelligent AIs, just as a pre-intelligent AI could participate in its own construction, a paradigm known as "seed AI". Once a subhuman AI passes a certain critical threshold, its design intelligence will surpass that of the programmers, who will no longer be necessary to further the AI's level of intelligence. All these factors combine to suggest that 1) AI will be a complex achievement, but not as complex as most people naively guess, and 2) once the AI reaches a certain level, development could accelerate rapidly.

Recursive Self-Improvement

At some critical point, the assistance of the programmers would no longer be required for further increases in intelligence. Depending on its desire and ability to enhance its own intelligence, an AI would take steps to understand the exernal world insofar as that understanding contributes to its increased intelligence and the accomplishment of its goals (which hopefully include empathy and benevolence). And we can be pretty sure it would contribute a lot. The goal of increased intelligence, and the desire to build better models of real-world objects is likely to be prominent, because a better ability to model the world is a subgoal of nearly all goals. No AI is going to stay on its original computing substrate forever; whatever its goals are, they will eventually spill over from the "virtual world" to the real one, if only for the reason that the real world must underlie any virtual world. A connection to the Internet is not likely to be necessary. Given an AI with a vastly accelerated thinking speed, the ability to improve its own source code, and the ability to create dedicated brainware for accomplishing specific tasks, this spillage could occur quite rapidly, probably affecting all of humanity. It would require some sort of robotics. The process could begin when an AI gains access to nanotechnology, or it could begin earlier. Since AIs would run on transistors millions of times faster than the human brain, they would have many subjective years to devise plans to influence the real world per human second.

Using its unique cognitive advantages, an AI might rearrange its circuits to create radio broadcasting devices, projecting its information patterns out into the world. Or, perhaps more plausibly (but who knows?), it might employ a desktop scanning-tunnelling microscope to develop self-replicating nanomachinery, or self-assembling materials to construct physical appendages of arbitrary size, speed, composition, and dexterity. I, personally, might use utility fog. An AI of transhuman intelligence could compel humans (deviously or honestly, depends on its morals, and perhaps many humans at once, through the Internet) to assist it in acquiring specific tools or resources necessary to expand beyond its origin computer. It might tap into technologies beyond our wildest dreams, entirely bypassing currently imaginable methods in the same way that human culture bypassed the immensely slow process of biological evolution. Remember; how an AI gets from its computing platform to influencing the real world depends not on what you can imagine, but what the AI can imagine. An obstacle that holds steady in the face of a million subjective years of peasant-thought might fall after a mere thousand subjective years of Einstein-level thought, or a mere hundred years of AI-level thought. Just because we have a stereotype of AIs piloting clunky robot bodies does not mean that actual AIs will share our stereotype.