Dale Carrico Classics
Just in case there are new readers, I want to refer them to the writings of Dale Carrico, probably the best transhumanist critic thus far. He's a lecturer at Berkeley. (Maybe The New Atlantis should try hiring him, though I sort of doubt they'd get along.) I especially enjoy this post responding to my "Transhumanism Has Already Won" post:
The Robot Cultists Have Won?
When did that happen?
In something of a surprise move, Singularitarian Transhumanist Robot Cultist Michael Anissimov has declared victory. Apparently, the superlative futurologists have "won." The Robot Cult, it would seem, has prevailed over the ends of the earth.
Usually, when palpable losers declare victory in this manner, the declaration is followed by an exit, either graceful or grumbling, from the stage. But I suspect we will not be so lucky when it comes to Anissimov and his fellow victorious would-be techno-transcendentalizers.
Neither can we expect them "to take their toys and go home," as is usual in such scenes. After all, none of their toys -- none of their shiny robot bodies, none of their sentient devices, none of their immortality pills, none of their immersive holodecks, none of their desktop nanofactories, none of their utility fogs, none of their comic book body or brain enhancement packages, none of their kindly or vengeful superintelligent postbiological Robot Gods -- none of them exist now for them to go home with any more than they ever did, they exist only as they always have done, as wish-fulfillment fancies in their own minds.
You can read the whole thing at Dale's blog.
Matter, Antimatter Origin Theories — Baryogenesis
I remember reading somewhere that one possibility in the early universe is that a tremendous amount of matter and antimatter both formed, most of it annihilated itself, and the small amount that remained became our present matter-dominated universe. From a few casual Google searches I have not been able to find this reference. It was probably some popular physics book written in the 1990s. Possibility one in the summary below would appear to correspond to this scenario, however.
The question is that of baryogenesis, which is not well understood. Here's the background from Wikipedia:
The Dirac equation, formulated by Paul Dirac around 1928 as part of the development of relativistic quantum mechanics, predicts the existence of antiparticles along with the expected solutions for the corresponding particles. Since that time, it has been verified experimentally that every known kind of particle has a corresponding antiparticle. The CPT Theorem guarantees that a particle and its antiparticle have exactly the same mass and lifetime, and exactly opposite charge. Given this symmetry, it is puzzling that the universe does not have equal amounts of matter and antimatter. Indeed, there is no experimental evidence that there are any significant concentrations of antimatter in the observable universe.
There are two main interpretations for this disparity: either the universe began with a small preference for matter (total baryonic number of the universe different from zero), or the universe was originally perfectly symmetric, but somehow a set of phenomena contributed to a small imbalance in favour of matter over time. The second point of view is preferred, although there is no clear experimental evidence indicating either of them to be the correct one. The preference is based on the following point of view: if the universe encompasses everything (time, space, and matter), nothing exists outside of it and therefore nothing existed before it, leading to a total baryonic number of 0. From a more scientific point of view, there are reasons to expect that any initial asymmetry would be wiped out to zero during the early history of the universe. One challenge then is to explain how the total baryonic number is not conserved.
I've been told that a lot of stuff exists outside of our local universe, but I don't want to make this more complicated than it already is.
Experimental Support for Monkey Self-Agency
For a contemporary press release relevant to my recent debate with Alex Knapp, "Rhesus monkeys have a form of self awareness not previously attributed to them":
In the first study of its kind in an animal species that has not passed a critical test of self-recognition, cognitive psychologist Justin J. Couchman of the University at Buffalo has demonstrated that rhesus monkeys have a sense of self-agency -- the ability to understand that they are the cause of certain actions -- and possess a form of self awareness previously not attributed to them.
The study, which will be published July 6 in Biology Letters, a journal of the Royal Society, may illuminate apparent self-awareness deficits in humans with autism, schizophrenia, Alzheimer's disease and developmental disabilities.
Rhesus monkeys are one of the best-known species of Old World monkeys, and have been used extensively in medical and biological research aimed at creating vaccines for rabies, smallpox and polio and drugs to manage HIV/AIDS; analyzing stem cells and sequencing the genome. Humans have sent them into space, cloned them and planted jellyfish genes in them.Couchman, a PhD candidate at UB, is an instructor at UB and at the State University of New York College at Fredonia. He points out that previous research has shown that rhesus monkeys, like apes and dolphins, have metacognition, or the ability to monitor their own mental states. Nevertheless, the monkeys consistently fail the mirror self-recognition test, which assesses whether animals can recognize themselves in a mirror, and this is an important measure self-awareness.
"We know that in humans, the sense of self-agency is closely related to self-awareness," Couchman says, "and that it results from monitoring the relationship between pieces of intentional, sensorimotor and perceptual information.
"Based on previous findings in comparative metacognition research, we thought that even though they fail the mirror test, rhesus monkeys might have some other form of self-awareness. In this study we looked at whether the monkeys have a sense of self agency, that is, the understanding that some actions are the consequence of their own intentions."
Ransom for “Fable of the Dragon-Tyrant” Flash Animation
My friend Kent Kemmish, at Halcyon Molecular, has offered to put up $50 for someone who does the best animated flash version of Nick Bostrom's classic essay "The Fable of the Dragon-Tyrant" (German version). Let's say that the challenge stands for one month, until August 7th.
My other friend Kevin Fischer is also putting in $50 for a total of $100.
Would anyone else be interested in adding to that purse? We probably need to boost it by a few times to make this happen.
Updates:
Kent Kemmish initially put in $50.
Kevin Fischer put in $50.
Luke Parrish put in $50.
Steve put in $50.
Christopher Hamersley put in $50.
Lincoln Cannon put in $50.
Didier Coeurnelle put in $200.
The purse is now at $500.
Kickstarter won't work because it has to be created by the person who does the project, and they are encumbered by having to promise deliverables. Both of these do not apply in this situation.
The Benefits of a Successful Singularity
What is the point of a beneficial Singularity? A challenging question because there are so many potential benefits. Some of the benefits I enjoy more might not be the same as the benefits you would enjoy. People can disagree.
What kind of Singularity happens depends on what kind of singleton we end up with, but we can wistful and optimistic, right? The Singularity I'm working towards would have the following components:
1) Invention of molecular nanotechnology or superior manufacturing technology, enabling the production of near-unlimited food, housing, clean water, and other products.
2) Enforcement of local "volitional bubbles" that reduce the rate of non-consensual violent crime to zero. I'd be curious to see how altruistic superintelligence or the CEV output would handle cases where people join "fight clubs" where the risk of death is part of the bylaws.
3) Unless the current overall system is objectively optimal even to an altruistic superintelligence, presumably this would be rearranged for the better as well, though exactly how and in light of what drives and freedoms is hard to say. Probably this won't be a straightforward extension of the politics of the 21st century, like how human politics isn't a straightforward extension of conodont politics.
4) Possible amplification and diversification of every single object, skill, or practice. So instead of a few general different types of asteroids they might become 10^12 different kinds of asteroids. At first I thought of saying "amplification and diversification of everything of which intelligence is capable", but why not amplify and diversify everything in the entire cosmos? This would include art, music, aesthetics, "dance", communication, "philosophy", world building, etc.
5) Presumably, if the Church-Turing thesis is true and phenomenally conscious uploads are possible, then the mass conversion of matter into "computronium", though maybe not all matter. The simple reason why is that this would allow more space, more joy, more possible experiences, more security, etc. If phenomenally conscious uploads are not possible, then similar actions in the same space might include making space colonies, or hollowing out the underground and pumping water, air, and sunlight down to create vast new living spaces.
6) The possibility of guided transformation of willing humans to superintelligence, through pathways determined to be of less "risk", i.e., wireheading. How risk is defined will be partially subjective and partially objective, like most things.
7) Thoughtful preservation of the outlines of existing human societies and cultures (minus violence presumably, which is a central part of many cultures) by those who wish to do so. This would be in contrast to the default today, which is the disintegration of most cultures and integration into Anglosphere or east Asian hegemons.
8) The possibility of eliminating suffering and exploring "gradients of bliss" as everyday reality, along the lines of the Hedonistic Imperative. We might find that it is pleasant to increase our happiness set-points somewhat, or possibly even experiment with lowering them temporarily. We might find that it is pleasant to be in a state of revelatory or orgiastic bliss non-stop, or maybe not. A Singularity would at least give us the option of exploring those possibilities.
9) The potential of dispelling the mystery of human interactions by using "x-ray glasses" (advanced analytical AI linked directly to our brains, or part of our brains) to see their complex internal structure in a deterministic fashion, if only briefly, using fine-grained simulations. From the perspective of a higher intelligence, to what extent is human nature truly computationally "chaotic" in the sense of chaos theory, and to what extent is it entirely deterministic, like a Newtonian universe? We may be surprised by the answer.
10) Pursuit of higher aesthetics and moralities beyond the human realm. Hopefully whatever singleton we are stuck with still allows a wide berth for personal experimentation, perhaps with supervision by AI experts who have already "been there". Surely there must exist interesting value systems and aesthetic points of view which we would be quite excited to experience which we have neither yet seen nor even thought of yet.
11) Exploration of the entire space of thoughts not only directly adjacent to the human realm, but also far beyond it. It sounds pretty simple to just state a sentence like that, but in practice such fringe ideas can instill powerful feelings of confusion, conflicting intuitions, and even awe when they turn out to be correct or useful. The purpose is not just exploring thoughtspace for its own sake but for the complex manifold of emotional and intellectual interactions that emerge from connecting disparate concepts and exploring the "outer solar system" of thought.
12) Most seriously and urgently, ending the orgy of killing and torture upon the human and conscious animal species by man, other animals, disease, environmental factors, and "other".
The Final Weapon
It's not really "fair", but history generally consists of people getting better and better weapons, and whoever has the best weapons and the best armies makes the rules. The number of historical examples of this phenomenon are practically unlimited. The reason America is respected and feared today is because of our military capabilities, particularly nuclear weapons. Complain if you want, this is reality.
I am excited by the possibility that the 200,000 year arms race will finally come to an end by a singleton. It had to end sometime. Personally, it will be a relief, if we survive. While many people can happily enjoy their lives on a daily basis, just focusing on their tiny sphere, myself and others are cursed with concerns about the overall trajectory of humanity and human conflict. My relationship with Murphy's law is so close that I would hardly be surprised to hear the detonation of nuclear weapons in the distance, practically anytime or anywhere.
Nuclear weapons, of course, are toys in comparison to the products of MNT, or worse yet, true superintelligence. MNT could enable the creation of "fingers of god", large satellites that condense sunlight to deliver terrific beams to the surface. A square mile of space-based solar panels, properly utilized, would provide enough energy to set entire districts alight. Anti-satellite missiles could be quickly detected by their thermal signature and neutralized long before reaching the target.
In comparison to the weapons of the future, nuclear weaponry is quite mundane, which makes living in 2011 less distressing than living in the future. However, there's plenty to look forward to. The world's most sophisticated militaries are aggressively pursuing AI technology for robotic warfighters. The problem with mindless AI soldiers is that it decouples the delivery of force from the ostensible wisdom and benevolence of human actors. This is the problem with nukes as well -- for all of Reagan's faults, at least he was terrified of nuclear war, and would have done anything he could to avoid one. One can only hope Obama would have the same attitude. Bush Sr. probably would have as well, but who knows? Eventually there will be a President that won't hesitate to press the button. I have sympathy for Robin Hanson's position that he votes for Presidents entirely based on their propensity to initiate wars or not. Everything else is truly secondary.
It might be overly masculine of me, but I see the Singularity as the final chapter in this ongoing and stressful arms race called human history. A recursively self-improving AI is the way of securing the highest power available in this time and place, and directing it towards constructive ends. Once we press the big red button, that's that. It won't matter so much who did it, as the effects that flow from that cause. It's easy to imagine AI or human goals drifting under self-improvement to the point where the singleton doesn't really care whether we're around or not, and starts sucking up all the air or free energy for its own purposes. After all, our planet is tiny. Truly bite-size from the perspective of a self-improving superintelligence, it could be consumed in a matter of days.
Transhumanism itself is secondary to this event. In retrospect, no one will care so much about cyborgs. They will care about recursively self-improving superintelligence undergoing a hard takeoff. Cyborgs, implants, gene therapy, life extension, seasteading, the Internet -- these are toys. Superintelligence is what really matters. I'm truly happy that more people are beginning to understand that.
Replying to Alex Knapp, July 2nd
Does Knapp know anything about the way existing AI works? It’s not based around trying to copy humans, but often around improving this abstract mathematical quality called inference.
I think you missed my point. My point is not that AI has to emulate how the brain works, but rather that before you can design a generalized artificial intelligence, you have to have at least a rough idea of what you mean by that. Right now, the mechanics of general intelligence in humans are, actually, mostly unknown.
What’s become an interesting area of study in the past two decades are two fascinating strands of neuroscience. The first is that animal brains and intelligence are much better and more complicated than we thought even in the 80s.
The second is that humans, on a macro level, think very differently from animals, even the smartest problem solving animals. We haven’t begun to scratch the surface.
Based on the cognitive science reading I've done up to this point, this is false. Every year, scientists discover cognitive abilities in animals that were previously thought to be uniquely human, such as episodic memory or the ability to deliberately trigger traps. Chimps have a "near-human understanding of fire" and complex planning abilities. Articles such as this one in Discover, "Are Humans Really Any Different from Other Animals?", and this one in New Scientist, "We're not unique, just at one end of the spectrum" are typical from scientists who compare human and chimp cognition. It's practically become a trope for the (often religious) person to say humans and animals are completely different, and the primatologist or cognitive scientist to say, "not nearly as much as you think..."
One primate biologist says this:
"If we really want to talk about the big differences between humans and chimps — they're covered in hair and we're not," Taglialatela told LiveScience. "Their brains are about one-third the size of humans'. But the major differences come down to ones of degree, not of kind."
There's a really good paper somewhere out there on cognitive capacities in humans and chimps and how human cognitive abilities seem to be exaggerations of chimp abilities rather than different in kind, but I can't find it.
Arguments that chimps and humans are fundamentally different tend to be found more often on Christian apologetics sites than in scientific papers or articles. The overall impression I get is that scientists think chimp cognition and human cognition are different in degree, not in kind. There are humans out there so dumb that chimps are probably more clever than them in many important dimensions. Certainly if Homo heidelbergensis and Neanderthals were walking around, we would have even more evidence that the difference between humans and chimps is one of degree, not kind.
Another point is that even if humans were radically different in thinking than animals, why would that automatically mean AI is more difficult? We already have AI that utterly defeats humans in narrow domains traditionally seen as representative of complex thought, no magical insights necessary.
Yet another possibility is one of AI that very effectively gathers resources and builds copies of itself, yet does not do art or music. An AI that lacks many dimensions of human thought could still be a major concern with the right competencies.
But before scientists knew anything about birds, we basically knew: (a) they can fly, (b) it has something to do with wings and (c) possibly the feathers, too. At that stage, you couldn’t begin to design a plane. It’s the same way with human intelligence. Very simplistically, we know that (a) humans have generalized intelligence, (b) it has something to do with the brain and (c) possibly the endocrine system as well.
I should think that many tens of thousands of cognitive scientists would object to the suggestion that we only know a "few basic things" about intelligence. However, it's quite subjective and under some interpretations I would agree with you.
The above paragraph is a vast oversimplification, obviously, but the point is to analogize. Right now, we’re at the “wings and feathers†stage of understanding the science of intelligence. So I find it unlikely that a solution can be engineered until we understand more of what intelligence is.
The impression that one has here probably correlates with how much cognitive science you read. If you read a lot, then it's hard not to think of all that we do know about intelligence. Plenty is unknown, but we don't know how much more needs to be known to build AI. It could be a little, it could be a lot -- we have to keep experimenting and trying to build general AI.
Now, once we understand intelligence, and if (and I think this is a big if), it can be reproduced in silicon, then the resulting AGI probably doesn’t necessarily have to look like the brain, anymore than a plane looks like a bird. But the fundamental principles still have to be addressed. And we’re just not there yet.
Yet formalisms of intelligence, like Solmonoff induction, are not particularly algorithmically complicated, just computationally expensive. Gigerenzer and colleagues have shown that many aspects of human decision making rely on "fast and frugal heuristics" that are so simple they can be described in pithy phrases like Take the Best and Take the First. Robyn Dawes has shown how improper linear models regularly outperform "expert" predictors, including medical doctors. Rather than possessing a surplus of cognitive tools for addressing problems and challenges, humans seem to just possess a surplus of overconfidence and arrogance. It is easy to invent problems that humans cannot solve without computer help. Humans are notoriously bad at paying attention to base rates, for instance, even though base rates tend to be the most epistemologically important variable in any reasoning problem. After you read about many dozens of experiments in heuristics and biases research where people embarrass themselves in spectacular fashion, you start to roll your eyes a bit more when people gloat about the primacy of human reasoning.
I correspond with lots of neuroscientists. Virtually all of them tell me that the big questions remain unanswered and will for quite some time.
I correspond with neuroscientists who believe that the brain is complex but that exponentially better tools are helping quickly elucidate many of the important questions. Regardless, AI might be a matter of computer science, not cognitive science. Have you considered that possibility?
AIXI is a thought experiment, not an AI model. It’s not even designed to operate in a world with the constraints of our physical laws.
Sure it is. AIXI is "a Bayesian optimality notion for general reinforcement learning agents", a yardstick that finite systems can compare against. It may be that the only reason our brains work at all is because they are approximations of AIXI.
My point is to recognize that the way machine intelligence operates, and will for the conceivable future, is in a manner that is complementary to human intelligence. And I’m fine with that. I’m excited by AI research. I just find it unlikely, given the restraints of physical laws as we understand them today, that an AGI can be expected in the near term, if ever.
"If ever"? You must be joking. That's like saying, "I just find it unlikely, given the restraints of physical laws as we understand them today, that a theory of the vital force that animates animate objects can be expected in the near term, if ever", or "I just find it unlikely, given the restraints of physical laws as we understand them today, that a theory of aerodynamics that can produce heavier-than-air flying machines can be expected in the near term, if ever". Why would science figure out how everything else works, but not the mind? You're setting the mind apart from everything else in nature in a semi-mystical way, in my view.
I am, however, excited at the prospect of using computers to free humans from grunt work drudgery that computers are better at, so humans can focus on the kinds of thinking that they’re good at.
To be pithy, I would argue that humans suck at all kinds of thinking, and any systems that help us approach Bayesian optimality are extremely valuable because humans are so often wrong and overconfident in many problem domains. Our overconfidence in our own reasoning even when it explicitly violates the axioms of probability theory routinely reaches comic levels. In human thinking, 1 + 1 really can equal 3. Probabilities don't add up to 100%. Events with base rates of ~0.00001%, like fatal airplane crashes, are treated as if their probabilities were thousands of times the actual value. Even the stupidest AIs have a tremendous amount to teach us.
The problem with humans is that we are programmed to violate Bayesian optimality routinely with half-assed heuristics that we inherited because they are "good enough" to keep us alive long enough to reproduce and avoid getting murdered by conspecifics. With AI, you can build a brain that is naturally Bayesian -- it wouldn't have to furrow its brow and try real hard to obey simple probability theory axioms.
Singularity Summit 2011
The press release for SS11 is posted. Featuring Ken Jennings, Christof Koch, Tyler Cowen, Ray Kurzweil, and many others. The venue will be the same as 2009 -- the 92nd St. Y in New York City.
The theme we are pegging this year's conference to is the Watson victory.
Registration is here, register early because price will go up:
https://www.singularitysummit.com/registration/
October 15-16, 2011. Press passes are available for actual press, ask me about it.
If the Singularity movement is an outgrowth of the Russian Orthodox Church, as Stross argues, then this would be our Pascha.
The Illusion of Control in a Intelligence Amplification Singularity
From what I understand, we're currently at a point in history where the importance of getting the Singularity right pretty much outweighs all other concerns, particularly because a negative Singularity is one of the existential threats which could wipe out all of humanity rather than "just" billions. The Singularity is the most extreme power discontinuity in history. A probable "winner takes all" effect means that after a hard takeoff (quick bootstrapping to superintelligence), humanity could be at the mercy of an unpleasant dictator or human-indifferent optimization process for eternity.
The question of "human or robot" is one that comes up frequently in transhumanist discussions, with most of the SingInst crowd advocating a robot, and a great many others advocating, implicitly or explicitly, a human being. Human beings sparking the Singularity come in 1) IA bootstrap and 2) whole brain emulation flavors.
Naturally, humans tend to gravitate towards humans sparking the Singularity. The reasons why are obvious. A big one is that people tend to fantasize that they personally, or perhaps their close friends, will be the people to "transcend", reach superintelligence, and usher in the Singularity.
Another reason why is that augmented humans feature so strongly in stories, and in the transhumanist philosophy itself. Superman is not a new archetype, he reflects older characters like Hercules. In case you didn't know, many men want to be Superman. True story.
Problems
The idea of a human-sparked Singularity, however, brings about a number of problems. Foremost is the concern that the "Maximillian" and his or her friends or relatives would exert unfair control over the Singularity process and its outcome, perhaps benefiting themselves at the expense of others. The Maximillian and his family might radically improve their intelligence while neglecting the improvement of their morality.
One might assume that greater intelligence, as engineered through WBE (whole brain emulation) or BCI (brain-computer interfacing), necessarily leads to better morality, but this is not the case. Anecdotal experience with humans shows us that humans which gain more information do not necessarily become more benevolent. In some cases, like with Stalin, more information only increases the effect of paranoia and the need for control.
Because human morality derives from a complex network of competing drives, inclinations, decisions, and impulses that are semi-arbitrary, any human with the ability to self-modify could likely go off in a number of possible directions. A gourmand, for instance, might emphasize the sensation of taste, creating a world of delicious treats to eat, while neglecting other interesting pursuits, such as rock climbing or drawing. An Objectivist might program themselves to be truly selfish from the ground up, rather than just "selfish" in the nominal human sense. A negative utilitarian, following his conclusions from the premises, might discover that the surest way of eliminating all negative utility for future generations is simply to wipe out consciousness for good.
Some of these moral directions might be OK, some not so much. The point is that there is no predetermined "moral trajectory" that destiny will take us down. Instead, we will be forced to live in a world that the singleton chooses. For all of humanity to be subject to the caprice of a single individual or small group is unacceptable. Instead, we need a "living treaty" that takes into account the needs of all humans, and future posthumans, something that shows vast wisdom, benevolence, equilibrium, and harmony -- not a human dictator.
Squeaky Clean and Full of Possibilities -- Artificial Intelligence
Artificial Intelligence is the perfect choice for such a living treaty because it is a blank slate. There is no "it" -- AI as its own category. AI is not a thing, but a massive space of diverse possibilities. For those who consider the human mind to be a pattern of information, the pattern of the human mind is one of those possibilities. So, you could create an AI exactly like a human. That would be a WBE, of course.
But why settle for a human? Humans would have an innate temptation to abuse the power of the Singularity for their own benefit. It's not really our fault -- we've evolved for hundreds of thousands of years in an environment where war and conflict were routine. Our minds are programmed for war. Everyone alive today is the descendant of a long line of people who successfully lived to breeding age, had children, and brought up surviving children who had their own children. It sounds simple today, but on the dangerous savannas of prehistoric Africa, this was no small feat. The downside is that most of us are programmed for conflict.
Beyond our particular evolutionary history, all the organisms crafted by evolution -- call them Darwinian organisms -- are fundamentally selfish. This makes sense, of course. If we weren't selfish, we wouldn't have been able to survive and reproduce. The thing with Darwinian organisms is that they take it too far. Only more recently, in the last 70 or so million years, with the evolution of intelligent and occasionally-altruistic organisms like primates and other sophisticated mammals, did true "kindness" make its debut on the world scene. Before that, it was nature, bloody in tooth and claw, for over seven hundred million years.
The challenge with today's so-called altruistic humans is that they have to constantly fight their selfish inclinations. They have to exert mental effort just to stay in the same place. Humans are made by evolution to display a mix of altruistic and selfish tendencies, not exclusively one or the other. There are exceptions, like sociopaths, but the exceptions tend to more frequently be towards the exclusively selfish than the exclusively altruistic.
With AI, we can create an organism that lacks selfishness from the get-go. We can give it whatever motivations we want, so we can give it exclusively benevolent motivations. That way, if we fail, it will be because we couldn't characterize stable benevolence right, not because we handed the world over to a human dictator. The challenge of characterizing benevolence in algorithmic terms is more tractable than trusting a human through the extremely lengthy takeoff process of recursive self-improvement. The first possibility requires that we trust in science, the second, human nature. I'll take science.
Trust
I'm not saying that characterizing benevolence in a machine will be easy. I'm just saying it's easier than trusting humans. The human mind and brain are very fragile things -- what if they were to be broken on the way up? The entire human race, the biosphere, and every living thing on Earth might have to answer to the insanity of one overpowered being. This is unfair, and it can be avoided in advance by skipping WBE and pursuing a more pure AI approach. If an AI exterminates humanity, it won't be because the AI is insanely selfish in the sense of a Darwinian organism like a human. It will be because we gave the AI the wrong instructions, and didn't properly transfer all our concerns to it.
One benefit to AI that can't be attained with humans is that an AI can be programmed with special skills, thoughts, and desires to fulfill the benevolent intentions of well-meaning and sincere programmers. That sort of aspiration voiced in Creating Friendly AI (2001) -- which is echoed by the individual people in SIAI -- is what originally drew me to the Singularity Institute and the Singularity movement in general. Using AI as a tool to increase the probability of its own benevolence -- "bug checking" with the assistance of the AI's abilities and eventual wisdom. Within the vast space of possibilities of AI, surely there exists one that we can genuinely trust! After all, every possible mind is contained within that space.
The key word is trust. Because a Singularity is likely to lead to a singleton that remains for the rest of history, we need to do the best job possible ensuring that the outcome benefits everyone and that no one is disenfranchised. Humans have a poor track record for benevolence. Machines, however, once understood, can be launched in an intended direction. It is only through a mystical view of the human brain and mind that qualities such as "benevolence" are seen as intractable in computer science terms.
We can make the task easier by programming a machine to study human beings to better acquire the spirit of "benevolence", or whatever it is we'd actually want an AI to do. Certainly, an AI that we trust would have to be an AI that cares about us, that listens to us. An AI that can prove itself on a wide variety of toy problems, and makes a persuasive case that it can handle recursive self-improvement without letting go of its beneficence. We'd want an AI that would even explicitly tell us if it thought that a human-sparked Singularity would be preferable from a safety perspective. Carefully constructed, AIs would have no motivation to lie to us. Lying is a complex social behavior, though it could emerge quickly from the logic of game theory. Experiments will let us find out.
That's another great thing -- with AIs, you can experiment! It's not possible to arbitrarily edit the human brain without destroying it, and it's certainly not possible to pause, rewind, automatically analyze, sandbox, or do any other tinkering that's really useful for singleton testing with a human being. A human being is a black box. You hear what it says, but it's practically impossible to tell whether the human is telling the truth or not. Even if the human is telling the truth, humans are so fickle and unpredictable that they may change their minds or lie to themselves without knowing it. People do so all the time. It doesn't really matter too much as long as that person is responsible for their own mistakes, but when you take these qualities and couple them to the overwhelming power of superintelligence, an insurmountable problem is created. A problem which can be avoided with proper planning.
Afterword
I hope I've made a convincing case for why you should consider artificial intelligence as the best technology for launching an Intelligence Explosion. If you'd like to respond, please do so in the comments, and think carefully before commenting! Disagreements are welcome, but intelligent disagreements only. Intelligent agreements only as well. Saying "yea!" or "boo!" without more subtle points is not really interesting or helpful, so if your comments are that simplistic, keep it to yourself. Thank you for reading Accelerating Future.