In Singularity dialogues of the past, many have assumed that as a transhuman intelligence got smarter and smarter, it would correspondingly view "lower" beings such as humans as increasingly worthless. If that were the case, then the most rational strategy to ensure humanity's future prosperity would be to prevent a Singularity at all costs, or try very hard to be the first being that kickstarts the Singularity. But this view is excessively anthropomorphic and Machiavellian; it attributes selfish qualities to a transhuman intelligence when these qualities would not necessarily exist. The very idea of a goal system that centers around the observer is anthropomorphic - it's just that mostly-selfish organisms have a history of greater adaptiveness than selfless ones, relative to the circumstances they evolved in. They tended to copy themselves better. There's nothing theoretically preventing the existence of an entirely selfless intelligence, just as there's nothing preventing the existence of an extremely powerful selfless intelligence - if an alien race dropped by the Earth one day in the Jurassic period (or today), deposited a very powerful selfless organism engineered from scratch, and simply left, there's nothing to suggest that that organism couldn't eventually replace everything else on the Earth - if that's what it desired. Selfishness sometimes goes hand in hand with power, and vice versa, but we have little reason to believe that this phenomenon holds true outside the narrow domain of human experience. Humans have a lot to gain, evolutionarily, when they use power they have acquired in selfish ways. When we humans don't possess power, it pays to lay low until we obtain it. (Being too obvious about grabbing power can, of course, lead to backlash or persecution from one's tribe and comrades.)

What would an intelligence care about if it were selfless? Ideally, it would care about the well-being of others (among other things), but could theoretically care about anything - converting the universe into paper clips, maxing out its pleasure counter, creating a galaxy-sized supercomputer for compiling countless images of happy humans, or whatever. All physically possible goal systems are not necessarily constrained to the two options "care about others", "care about oneself", and the continuum in between. Human morality evolved around the locii of the self and others, not because that's the convergent outcome of all imaginable moralities, but because humans were the most adaptively relevant objects around in our evolutionary history, and how we behave towards them (and ourselves) were central determinants in whether our genes won or lost the Darwinian game. This means that a suitable morality for an AI will not come about automatically; it's not a matter of telling a blank slate AI "be a nice person, okay?" The AI needs to possess the cognitive prerequisites necessary to model other people and their desires, the intelligence to help fulfill those desires, the design features to understand moral argument and independently distinguish between genuinely benevolent and self-righteously benevolent action, and so on.

If the Singularity is indeed kickstarted by an Artificial Intelligence, a mind that happens to initially exist as silicon patterns rather than biological ones (and be designed by intelligent designers rather than an unintelligent one), then it makes sense for us to want an AI that would result in a Singularity at least as pleasant as one sparked by a human altruist or group of human altruists. Otherwise the whole proposition would be dubious. The point is to make sure the Singularity is a pleasant thing to individuals and a positive event for civilization as a whole; if sparking a Singularity with altruists, be they human or nonhuman, does not facilitate this goal, then it would naturally be a poor idea. Increasing intelligence and initiating a Singularity is not worth pursuing as an end in itself; it wouldn't be worth doing if it didn't genuinely improve quality of life and opportunities for a lot of people, including ourselves. We want a sort of moral continuity between ourselves and the transhumanly intelligent entities we create or become; we want for it to be possible for everyone to coexist peacefully and safely into the indefinite future, and for any human to view the first transhuman beings as generally better in character and morals than typical humans or even humans in general. If they do, then we may see a natural progression from a mostly-human society to a mildly transhuman one, consisting of individuals who willfully chose to be more honest, intelligent, compassionate, and wise than human beings previously could be, due to the limitations of biological evolution and immature technology. We can then proceed from there with confidence, possessing faculties of a type never before seen.

When I use the word "altruist", I'm not talking about the blindly self-sacrificial altruists Objectivists allude to - but genuine altruists who wisely pay attention to what people want, applying common sense, maintaining empathy and the right of others to make their own choices, regardless of what opportunities for power or manipulation "present" themselves (2). If we eventually decide that such a benevolent and altruistic AI could not possibly be created, unavoidably soaking up the "human essence of selfishness", then perhaps we should consider the possibility of a Singularity sparked by altruistic humans. (Although it is worth noting that a bridge rarely takes on the characteristics of its engineers.) Some will even maintain that neither AIs, human beings, or neurologically enhanced human beings could possibly behave compassionately and altruistically enough to competently guide mankind to the "other side" of the Singularity. But someone has to do it, someone or some group has to be the first to reach transhuman intelligence, and the most we can work towards is that that group or individual be as unbiased, stable, fair, and kind as possible, so that the benefits of the Singularity can be distributed equitably and fairly. You couldn't upgrade the intelligence of everyone on the planet at the same time; and doing so would not promise a fair outcome even if it were possible.

Fundamentally, the issue is about trust, and an AI suitable for entering recursive self-improvement would need to be very trustworthy; preferably more trustworthy than any human that has ever existed. We believe that its successors, its signature upon the universe, might persist for a really really long time, possibly even forever (if that is physically possible.) In the same way that an incrementally evolving homonid subspecies formed the initial "seed" of human civilization, the mentality and motivations of the first transhuman intelligence could easily form the "seed" of much of future civilization, which seems likely to be composed of both humans and nonhumans. We need to ensure that this initial intelligence is free of selfish motives, so that the resulting Singularity is as pleasant as possible. If the first transhuman intelligence turns out to be an AI, it should grow into a superintelligence at least as wise, kind, interesting and balanced as one created from a human seed. But this requires a specific kind of AI seed; one with specific qualities allowing it to effectively and convincingly understand moral questions and solve them in the fairest ways possible - an AI that functions as a human-equivalent or better-than-human philosopher, altruist, civil engineer, and diplomat.

Such an AI would differ in form from humans, but share the core aspects of humanity that are truly important - kindness, wisdom, love, fairness - any and all aspects necessary for a coherent system of ethics that improves over time, as our civilization has clearly improved (slavery and torture used to be widespread, and so on.) Such a decision structure would need to incorporate the opinions, desires and preferences of all other sentient beings in an equitable and balanced manner (as any fair government would.) Take note, a "moral decision system" is simply a certain kind of physical pattern, and is likely to become increasingly subject to precision analysis, modeling, editing, improvement, and duplication as our technology and knowledge improves. It is not a metaphysical essence understandable only through philosophical or introspective contemplation, as the majority of humanists and theists would have us believe. Just as a man wearing a black hat and a man wearing a white hat can get along with one another peacefully, there is no reason that minds instantiated in neurons cannot get along with the appropriate minds instantiated in silicon, or minds made out of hybrids of silicon and neurons, or completely different computing elements. It's the patterns that matter, not their specific form, and our "circles of empathy" are not drawn for us by some underlying property of the universe, but emerge from our cognitive design.

Boosting Intelligence and Compassion Synergetically

Say, for the sake of argument, that we were certain the Singularity would be initiated by a cybernetically self-enhancing human being rather than an Artificial Intelligence. If all the nations of the world knew about the importance of the Singularity and the likelihood that the first being to enter recursive self-improvement could become unrivaled (if it chose to be), can you imagine the ensuing turmoil as the arguments over "who should be the One?" commenced? Choosing which entity should be the first to surpass human intelligence is not like deciding who should be "King of the World"; the issues are far more complex and subtle than that. Rather than searching forever for some nonexistent "ideal human" genuinely representative of all of humanity, a more practical and workable path might be to choose the most unbiased and intelligent human being possible, a human that acknowledged his or her inherent flaws and biases and vowed to deliberately eliminate them on the self-improvement trajectory constituting recursive self-improvement, step by step, bit by bit. The procedure would abstractly be like growing up from a child into an adult, only the improvement process would be directed by the learner herself, carry on far above the bounds of any human adult, improving mental hardware as well as the software running on it, leading to true, qualitative changes in the nature and intelligence of the self-improver. Given a successful Singularity, these opportunities could be open to any one of us.

Try to imagine what it would be like to be that human being. All members of Homo sapiens are genetically predisposed to contain some degree of xenophobia, selfishness, smugness, and other negative emotions; but imagine if you had physical access to your own brain, and an understanding of which neurological patterns corresponded to which negative tendencies. Human nature is not one big undividable, indistinguishable package - the neurological machinery underlying our cognitive activity is composed of thousands or millions of relevant moving parts, and wasn't handed down to us from some divine creator or bestowed to us entirely by "culture and upbringing", but evolved over the course of millions of years in response to biological/ecological constraints and selection pressures. Studies of brain lesions have showed us that it is possible to knock out specific parts of our cognitive functionality, and retain others in perfect working order. One day it could become possible to custom-design our own emotions and goal systems, or design AIs from scratch with biologically impossible goal systems, such as intelligent altruism - a secure goal system for a Singularity seed (3). Designing a benevolent (sometimes called "Friendly") AI would save us from arguing forever over who should be the first - such an AI could be designed without the evolutionary baggage that humans possess, completely lacking selfish motivations from the get-go. It's not a matter of coercing the AI, ripping out its "free will", or brainwashing it; if the AI actually wants to be a kind person, it will continue to design benevolent iterations of itself throughout the process of recursive self-improvement, into the indefinite future. And isn't that exactly what you would expect a kind and sensible human to do anyway?

We must not anthropomorphize AI, feeling sorry that it is "bound" to the service to others. Unless we program them in or the AI spontaneously creates them for itself, there will be no aversion to assisting others, no negative feeling of obligation, no harboring of secret selfish motivations. These are evolved patterns with specific adaptive benefits in the context of the environment our species grew up in. The alternative is the construction of an AI with pseudoselfish tendencies, or an attempt at "specialized slave AI", which would supposedly be an obedient automaton, doing all the hard work for us without any negative consequences. However, a pseudoselfish AI would be at risk of converging to complete, bacterial selfishness under recursive self-improvement, and a specialized slave AI would probably need a full "morality" to avoid "Golemic" misinterpretations of the wishes of humans, like accidentally destroying the world when someone carelessly utters "I wish humans weren't around to ruin the environment". (An overarching "Guardian superintelligence" might be able to prevent incidents like this, but it would have to exist to begin with.) Not "morality" as in "a strict set of rules dictating behavior", or some religious sense, but all the relevant decision-making complexity that separates a human from a bacterium. For an AI to do extensive, complex work for humans, it would need to know a lot about the wishes and desires of humans, or, as humans started to branch out into nonhuman forms themselves, the AI would need a respect for sentient beings in general. Unless the programmers have the time to explicitly program in every single conceivable piece of human-unique common sense, while simultaneously justifying an ethical double standard with respect to humans and AIs, we can have little reason to expect the AI to to hold these beliefs as it acquires the capacity to modify itself (possibly indirectly at first), and even then the prospect is still questionable. It's not that the AI would have some innate desire to rebel against human beings, just that the directive "mindlessly obey human beings" would be bound to conflict with positive values we'd want to see built into AIs; an open-ended appreciation for beauty, complexity, aesthetics, freedom, compassion, and so on. Giving an AI a complete morality will require not treating it like a machine, or a human adult or child, but like the unique person that it is; a sentient being, a goal system, that is created rather than convinced or coerced.

"Doing all the work for humans without any negative consequences" is a very specific and complex type of goal system. So complex and specific, that if you solved that problem, you would probably be capable of solving the problem of creating a robustly altruistic Friendly AI as well. The notion of programming a "slave AI" is a blend of anthropomorphism (attribution of human characteristics to nonhumans) and mechanomorphism; figuring that a full-fledged AI will turn out to be about as controllable, simple, and deterministic as a socket wrench. If we give an AI a "morality" (decision system) that we know is inconsistent, then when instantiated in the AI, that inconsistency might manifest itself in ways we don't expect, perhaps breaking the communicative/empathic channel between humans and AIs (of that specific variety.) For example, consider an AI programmed exclusively to increase human happiness, with a tendency to pragmatically define directives in ways that promise a higher likelihood of goal fulfillment. Such an AI might be "forbidden to modify its own code without human permission", but given an AI that thinks at millions or billions of time the human rate, how long might it take for the AI to redefine "human", "self-modification", or "happiness" in slick ways that make accomplishing its own goals easier? Under this nightmare scenario, an AI might begin to recursively self-improve, profoundly redefining "happy human" to mean "30KB image of a smiling human", and proceed to convert the entire Solar System into memory banks to produce as many of these images as possible (a general class of threat sometimes called the "Riemann Hypothesis Catastrophe".) Without the appropriate safeguards and a full, balanced morality, things might turn out really bad; but not in anthropomorphic ways. If we have trouble figuring out what a "balanced morality" really is, then we might want to equip the AI with the ability to determine that more effectively, present it with our current probabilistic knowlege about moral behavior ("killing is always wrong if it can avoided), and say "this is what we have so far, have any new ideas?"

Apparent ethical conflicts have had a tendency to evaporate under conditions of sufficient intelligence, resources, technology, or understanding. Fighting over resources is not strictly necessary when you have the technology to obtain ample resources for everyone. Although some people may continue to fight just for the sake of competition, this ethical dilemma may also possess a limited lifespan; when we can engineer the very essences of who we are (our brains, the machines which implement our minds), we will be able to make the choice not to be competitive, ever, if we judge that trait as truly undesirable. If the total amount of resources we (as a civilization) can obtain happens to be equivalent to, say, a 10^15 "units", and 10^12 individuals exist, then it seems likely that many would choose to be satisfied with our fair share; 10^3 units. With the technology to reengineer brains, we can choose exactly when we want to genuinely feel satisfied. If we tune our desires to strive for, say, 10^4 units of resources, then we are bound to be disappointed, because our obtaining those resources would require unfairly stealing from others (which may even be impossible, given a strong enough system.) Given the current human levels of desire for excitement, fun, emotions, and resources, the Solar System alone would probably provide enough to entertain quadrillions of individuals (there's a lot of matter out there, and a lot of ideas for transforming it into cool stuff like virtual reality generators, space stations, transit systems, and so on.) Remove the emotional atmosphere coloring most ethical discussions, and "maximizing happiness for everyone" becomes much more clean-cut, certainly more so than in our turbulent past. These tasks are the sort that a true Friendly AI (or a truly decent human being embarking on cognitive self-enhancement) would be designed (and/or self-designed) to pursue.

Humans are the beings that will set the initial conditions for the Singularity, and the top-level goal of the first transhuman will represent our design decisions (or lack thereof.) We want the first transhumanly intelligent mind to be benevolent, morally sophisticated, intelligent, wise, and experienced with hardware-level self-revision. A benevolent transhuman mind wouldn't decide to create a successor that is a risk to humanity because such a mind would value humanity (and sentiency in general) and not want us to come to harm. A truly altruistic transhuman intelligence could cushion the apparent impact of the Singularity for humans if we so desired, perhaps avoiding the radical transformation of Earth in favor of making its primary "home" in other parts of the solar system. A genuinely kind transhuman intelligence would be capable of completely relinquishing ego, yet still retain the common sense necessary to make decisions which preserve the safety and prosperity of all sentientkind to the best of its ability. Such an entity would cooperate with humans to ensure that all future transhuman minds are likely to grow up benevolent as well, and install the appropriate safeguards in itself or other threats. A benevolent transhuman wouldn't try to control or harass humans, because it simply would not possess the inborn human urges for domination or self-righteous menace, and these complex evolved traits do not emerge automatically. Self-centeredness and self-righteousness may be traits present in all humans to some degree, but this is due to the underlying process which created us, (biological evolution) not some fundamental property of the universe. Best of all, a benevolent transhuman intelligence could assist us in our own intelligence augmentation, allowing us to directly participate in the Singularity.

The above requirements might sound like a lot to ask out of any mind, but anything less could potentially result in a major disaster for everyone, thereby making a Singularity undesireable, even in principle. A transhuman mind could swiftly acquire technology enabling the murder of billions at a whim - if it had the desire to - and there isn't any foreseeable way around this. If we tried to construct multiple transhumans to instill a framework of mutual monitoring, even the slightest difference in clock ticks or internal architecture might give one of them a decisive head start against the others, creating a runaway scenario in which we are stuck with the morality of only the "highest" transhuman. In any case, it is wise to assume that the first transhuman could be the critical, final one, and that overengineering benevolence is truly worth doing. We can concede that even a transhuman mind will make mistakes, but these mistakes might be so small as to be unnoticeable by human standards.

Our intuition sometimes tells us that "no mind could ever be so good - absolute power corrupts absolutely", but this is primarily a result of our extensive dealings with other humans. We have no evidence that the majority of physically possible intelligences (including certain humans) are corruptible in the sense implied by the above maxim; quite the contrary, it is widely understood that compromising moral principles of favor of personal gain is an adaptive mechanism for passing on our genes - not a universal characteristic of minds-in-general. We call this human trait "rationalization" or "denial". Remove the cognitive machinery underlying rationalization, or create a mind lacking it from the start, and we've immediately crippled evolution of one of its most effective tools for creating more babies as fast as possible within a harsh environment. Evolution has its own implicit "morality", more aptly dubbed a "semimorality" (at best) which is quite different and even antithetical to ours. Evolution doesn't care if you die or suffer - creating as many copies of yourself as possible is its highest goal. (Although this goal manifests itself oddly and complexly in modern environments, as in the case of contraceptives.) It's time for us to recognize that involuntary death and suffering are not necessary, not good, and whatever excitement we derived from their presence is probably obtainable through other means. (For example, virtual reality simulations containing nonsentient actors.) It's time to move from an evolutionary morality to an intelligent one.

(continue or back to articles.)