Reasons to Focus on Cognitive Technologies
Posted by Jeriaska on July 31st, 2007
Eliezer Yudkowsky is one of the world’s foremost researchers on Friendly AI and recursive self-improvement. He created the Friendly AI approach to AGI, which emphasizes the importance of the structure of an ethical optimization process and its supergoal, in contrast to the common trend of seeking the right fixed enumeration of ethical rules a moral agent should follow.
In 2001, he published the first technical analysis of motivationally stable goal systems, with his book-length Creating Friendly AI: The Analysis and Design of Benevolent Goal Architectures. In 2002, he wrote “Levels of Organization in General Intelligence,” a paper on the evolutionary psychology of human general intelligence, published in the edited volume Artificial General Intelligence (Springer, 2006). He has two papers forthcoming in the edited volume Global Catastrophic Risks (Oxford, 2007), “Cognitive Biases Potentially Affecting Judgment of Global Risks” and “Artificial Intelligence as a Positive and Negative Factor in Global Risk.”
In his Transvision 2007 talk on the power of intelligence, called “Mind Is All That Matters: Reasons to Focus on Cognitive Technologies,” he reminded us that the human brain more than any other development in history has changed the face of the earth, and logically one can expect that improved cognitive technologies will likely have the same dramatic impact on human conception and control over the physical universe.
The following transcript of Eliezer Yudkowsky’s July 25, 2007 Transvision presentation “Mind is All That Matters” has not been approved by the author. An audio mp3 is also available.
Reasons to Focus on Cognitive Technologies
And now for something completely different…
I’m Eliezer Yudkowsky, a cofounder and current research fellow of the Singularity Institute for Artificial Intelligence, a 501c3 non-profit located in Silicon Valley, and I’d like to talk to you today about the most powerful force in the known universe. In our skulls we each carry three pounds of slimy wet gray stuff, corrugated like crumpled paper. If you didn’t know anything about human anatomy, if you didn’t know what a brain was and you saw it lying on the street, you’d probably say yuck and try not to get any on your shoes.
Aristotle thought the brain was an organ that cooled blood. It doesn’t look impressive. It doesn’t look big or dangerous or beautiful. It does look like it might be complicated. But nowhere near as complicated as when you look inside. This for example is a diagram of the visual system. This is the data flow between the major modules in the software that let you look around and see things. A skyscraper, a sword, a crown, a gun, a nuclear weapon, a computer—all these are byproducts of the wet gray thing. They popped out of the brain like a jack in the jack in the box.
Almost everything you see in the room around you, everything in your line of sight–the chairs, your clothes, the ceiling–are effects caused by human intelligence, the exceptions being the air molecules and you. Point to something in your environment and ask, ‘Why is it that way?’ Often enough the answer is ‘Human intelligence.’ Human beings leave behind a pattern like smoke puffs from an engine. We breathe out complexity and sweat order. There is a very powerful trick inside that gray lump. A very powerful trick embedded in those complicated neural circuits, though I should really call it a supertrick composed of a lot of subtricks. But, when you put it all together, well–a space shuttle is an impressive trick, a nuclear weapon is an impressive trick, an automated factory is an impressive trick. But none are as impressive as the master trick. The brain trick: the trick that does all these other tricks at the same time.
In everyday life, we take our human intelligence for granted because everyone has it. I mean, suppose everyone had the powers of Spider Man. Suppose we could all shoot webs from our hands and climb buildings. Then, no one would notice. No one would think it was important. We all have a superpower that’s much more impressive than web shooting. But since we all have the same superpower, we tend to forget how powerful it is. People say things like intelligence is no match for a gun, as if guns had grown on trees. They say, ‘How will this artificial intelligence affect the world if it doesn’t have any money?’ As if mice used money, or as if the human species had found dollar bills fluttering down from the sky on the savanna.
In everyday life, when you say the word ‘intelligence,’ people think of book smarts: calculus, chess, memorizing lists of facts, applying strict rules to well understood situations. We imagine the professor with an IQ of 160 and the billionaire CEO with an IQ of 120. It takes more than book smarts to succeed in the human world. Persuasiveness, enthusiasm, empathy, strategic thinking, rationality…. But notice that every factor I just listed was cognitive. Empathy happens in the brain, not the kidneys. There are not many famous novelists or famous military generals or famous politicians who are monkeys. Intelligence is the foundation of human power, the strength that fuels our other arts. If you don’t have human intelligence, you’re not even in the game.
People think about intelligence as if it were a scale that ran from village idiot to Einstein. We forget the things we have in common and remember only our differences. IQ tests are things that you can only administer to humans. A mouse would just eat the IQ test. If you can take an IQ test for humans in the first place, you’ve already established yourself as one of the smartest beings on the face of the earth. This scale from village idiot to Einstein measures the centimeter height differences within a species of giants. When you hear the word ‘intelligence,’ don’t think of Einstein, think of humans. When I talk about intelligence I’m talking about the scale that runs from rocks to humans. On that scale, the whole human species is packed into a small dot. Human beings in general have three times as much brain as a primate our size should have. That’s true whether you’re a village idiot or Einstein. There’s variance, but not that much variance.
Let us turn now to matters of transhumanism. To paraphrase Robert Heinlein, a transhuman is a transhuman mind. Anything else is a distraction. If you could give people the ability to control an extra pair of arms, that’s really cool, you have benefited humanity, but you have not created transhumans. If you invent artificial red blood cells that let people hold their breath for four hours, you can save many people who now die of heart attacks. That is a true and worthy application of intelligence, but it does not create a transhuman. Humanity did not rise to prominence upon earth by holding its breath longer than other species. If you can give someone the ability to shoot webs from their hands, all you’ve done is create a mere superhero.
In today’s society, most people get their idea of what’s important from advertising. Madison Avenue would have you believe that important technologies are new, improved, shiny, with extra features, and expensive. They want to sell you products by calling them futuristic, so people get the idea that the future is about new products. Hollywood movies tell you you’re in the future by showing you soaring skyscrapers and glittering gadgets. In movies, the future is like Oz: a distant land full of curious people and curious customs. We all know what the future looks like, even though we’ve never been there, the same way we know what Oz looks like… because the movies show it to us.
And the futurists who make their living as entertainers would tell fantastic stories to an audience because they are part of the same Hollywood – Madison Avenue culture. They make comfortable prophecies that in the future you will be able to buy ever more cool devices, maybe with a side order of improved health care. And that culture treats intelligence enhancement the same way–like an upgrade you can buy for your iPhone. In movies, if anyone has a direct brain interface to a computer, that’s just part of the background scenery that tells you you’re IN THE FUTURE… like people wearing strange clothes. You’ll be able to buy brain implants just like you can buy BOTOX injections, and use your brain implant to play fancy, expensive video games.
But intelligence is not just another hobby. Intelligence is the root of all our technology. All our impressive gadgets grow out of that. When you mess with intelligence, you shake the technology tree by its roots. If drugs or neurosurgery or gene therapy can make you smarter, that is not just another gee-whiz smart new technology. It is messing with the master trick–the brain trick–the first cause of which all technology is an effect. Artificial intelligence is not like interstellar travel, a cancer cure, or nanomanufacturing. Progress in brain-computer interfacing does not belong on the same graph that shows faster aircraft, smaller mp3 players, or better medical care. One of these technologies in not like the other. One of these technologies doesn’t belong. If you want to understand the future, look to the cognitive technologies. Look to the technologies that impact upon the mind.
Now, a common reaction to this notion is that we cannot ever have real cognitive technologies because science will never understand intelligence. The master trick is unlike anything else known to science. Therefore, it is magic. So, at this point I would like to recite an inspirational quote from Lord Kelvin. “The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motion of moving particles in the demonstrated daily miracle of our free will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms. Modern biologists were coming once more to the acceptance of something and that was a vital principle.”
Sounds familiar, doesn’t it? Can you think of any people who talk like that about any current mystery known to science? Intelligence is not the first confusing phenomenon that science has ever encountered. The secret of life seemed a lot more mysterious at the time than intelligence seems to us now. People have no sense of historical perspective. They learn about stars and chemistry and biology in school, and it seems that these matters have always been the property of science–that they never have been mysterious. When science challenges some new puzzle, it is a great shock to the children of that generation–because they have never seen science successfully explain something that feels mysterious.
Over the last 40 years there has been steady progress, believe it or not, in our basic understanding of the mind, which I don’t have nearly enough time to go into. But if you look at, for example, the vehicle that won the DARPA Grand Challenge, that was the modern maturity of a mathematical revolution that started 20 years earlier. Something which, if you study it, makes you say, Wow, this really is progress in our understanding of intelligence itself. We know something that we didn’t know before about how the mind exploits the structure of the environment.
AI is slowly creeping up along the absolute scale of intelligence. But humans use the realative human scale, so all humas see is that modern AIs are dumber than a village idiot. If you accept that the brain does not run on élan vital and that the master trick is not magic, then at some point we are going to see some major cognitive technology. Something like a broadband brain-computer interface, or broadband computer-assisted telepathy that creates 64 node clustered humans, or genuine artificial intelligence. We’re going to see cognitive technologies that break the upper bounds on intelligence that has held since the rise of the human species–something that cracks the ceiling of the known universal mind. And that is a sea change with the past–with a world of patterns created only by human intelligence.
To understand the true strangeness of the future we need to think about powerful intelligences that do not work the same way as modern human minds. The anthropologist Donald Brown once compiled a list of more than 200 human universals–characteristics that appear in every known human culture from Palo Alto to the hunter gatherers in the Amazon Rainforest: tool-making, weapons, grammar, tickling, meal times, mediating conflicts, dancing, singing, personal names, promises, and mourning the dead. Anthropologists don’t even think to report these characteristics, because they are everywhere. You won’t read an excited article about a newly discovered tribe: they eat food, they breathe air, they feel joy and sorrow, their parents love their children. We forget how alike we are under the skin, living in a world that only reminds us of our differences.
But human universals are not truly universal. A rock feels no pain. An amoeba does not love its children. Mice don’t make tools. Chimpanzees don’t hand down traditional stories. So, why is there human nature? The answer may be found in a principle of evolutionary biology which states: A complex adaptation–that’s an adaptation made up of many interdependent genes–must be universal within a sexually reproducing species. If gene B depends on gene A to produce its effect, then gene A has to become nearly universal in the gene pool before there is a substantial selection pressure in favor of gene B. A fur coat is not an evolutionary advantage unless the environment reliably throws cold weather at you. Well, other genes are also part of the environment. If gene B depends on gene A, then gene B is not a significant advantage unless gene A is reliably part of the genetic environment.
So if you have a complex adaptation with six parts, and each of the six genes were independently at 6% frequency in the population the chances of assembling the whole working package would be a million to one. And that wouldn’t happen often enough to give any of the genes a fitness advantage. The way that complex machinery gets started is that a single gene is an advantage all by itself and it increases to universality. Then gene B, which is dependent on gene A, arises and also increases to universality. And then mutant A’, which is dependent on gene B comes along, and increases to universality. And now you have irreducible complexity: A’ that depends on B, and B that depends on A’.
Now, in comic books, you find mutants who all in one jump, as the results of one mutation, hae the ability to throw lightning bolts. When you consider the biochemistry needed to produce electricity and the biochemical adaptations needed to keep yourself from being hurt by electricity, and the brain circuitry needed to control electricity finally enough to throw lightning bolts, this is not going to happen as the result of one mutation. That’s not how evolution works. At any given point in time, every member of the species has some version of the complex adaptation that’s evolving. When you apply this principle to the human mind, it gives rise to a rule that evolutionary psychologists have named “The psychic unity of humankind.” Yes, this is the official term.
Complex adaptations must be universal in sexually reproducing human species, including cognitive machinery in Homo sapiens. In every known culture, humans experience joy, sadness, anger, disgust, fears, surprise. In every known culture humans indicate these emotions using the same facial expressions. That is part of the wiring. The psychic unity of humankind is both explained and required by the mechanics of evolutionary biology. And when something is universal, we take it for granted. We assume it without thinking. In the movie “The Matrix” there is an artificial intelligence named Smith. Agent Smith. At first Agent Smith is cool, dispassionate, emotionless as he interrogates Neo. Under sufficient emotional stress, however, Agent Smith’s cool breaks down. He vents his disgust with humanity and, low and behold, his face shows the human expression for disgust. This is the great failure of imagination–anthropomorphism.
Back in the era of pulp science fiction, magazine covers occasionally showed a bug-eyed monster, colloquially known as a B.E.M., carrying off an attractive human female in a torn dress. For some odd reason, it’s never an attractive man in a torn shirt. Would a non-humanoid alien with a completely different evolutionary history sexually desire a human female? It seems rather unlikely. People don’t make mistakes like that by deliberately reasoning, ‘All minds have to be wired the same way. Therefore, a bug-eyed monster will find human females attractive.’ Probably the artist did not even think to ask whether an alien perceives human females as attractive. Instead, a human female in a torn dress is sexy, inherently so. As an intrinsic property. You’re not thinking about the evolutionary history and cognitive mechanisms of the bug-eyed monster, you’re thinking about the torn dress. If the dress were not torn, the woman would be less sexy, the bug-eyed monster does not venture into it.
This is a case of what E.T. Jaynes called the Mind Projection Fallacy, which happens when you project mental properties onto the outside world. For example, if I am ignorant about a phenomenon, then this is a fact about my state of mind, not a fact about the phenomenon itself. Every mysterious phenomenon is mysterious to some particular person. There are no phenomena that are inherently mysterious of themselves. And this, of course, includes the brain. The reason why it can’t be magic is because there is no magic, because magic remains something we cannot explain. This is a fact about us. Not a fact about whatever it is we are calling magical. Similarly, is a woman in a torn dress sexy, or is she sexy to human males?
Your ancestors evolved in an environment where every mind they met worked the same way they did. So your brain has evolved to model other minds through empathy, thinking, ‘What would I do in this other mind’s shoes?’ This is going to give you wrong answers if the other mind does not work like you do. We can understand that not everyone believes the same things we believe, because in our ancestral environment people did believe different things. But as you can see from this drawing here, we did not evolve to want things with different emotions. So the empathy trick–using your own brain to model the other brain–gives the wrong answer. And the empathy trick is instinctive, unconscious, faster than deliberate thought. We don’t think, ‘And now I will use the empathy trick to predict the thoughts of the Swamp Thing.’ We think, ‘That girl is attractive, so the B.E.M. will be attracted to her.’
So, if the empathy trick doesn’t work, how can you predict other minds? Once we learn to create true artificial intelligence, what will AIs be like? This is of course the great big $64 trillion trick question. In Hollywood movies, all the AIs are same type, a single tribe. Asking what AIs will do is a trick question because it implies that all AIs form a natural class. Humans do form a natural class, because we do share the same brain architecture. But when you say “artificial intelligence” you are referring to a vastly larger space of possibilities than when you say ‘human.’ When we talk about AIs, we are really talking about minds in general.
Imagine a map of mind design space. In one corner, a tiny little circle contains all humans. This is inside transhuman mind space, which includes all human possibilities as a subset, and is inside posthuman mind space, which is everything a transhuman might grow up into. And then there’s all the rest of mind design space–the space of minds in general, including AIs so strange, they aren’t even recognizably posthuman. Resist the temptation to generalize over all of mind design space. Let’s say that a mind in this space is embodied in a billion bits. Then, there are 2^billion minds possible made up of a billion bits. And every universal generalization you try to make that covers all those minds has 2^billion chances to be false. You can’t talk about ‘AIs,’ because you cannot talk about all possible minds at once. First you have to point to somewhere specific in mind space so you can say what kind of mind you’re talking about.
The only reason you could find yourself knowing what a generic mind will do is if you use the empathy trick–imagine what you would do in that mind’s place–and get back an anthropomorphic, generally wrong answer. There’s a lot more going on in the category ‘AI’ than there is in the category ‘human.’ And this lesson applies beyond pure AIs. Is suspect that even if you start with a human base and modify it, even if you’re just working with brain hacks and brain-computer interfaces, you’ll still get results that are a heck of a lot more surprising than a lot of people seem to expect. If you look at earth right now, there’s a lot of weird people packed into that tiny little human dot. Step outside that human dot and things will get stranger than they’ve ever been in human history. Unless, of course, that’s not what the transhumans want.
One often hears in futurism a line of reasoning that goes something like this: someone says, ‘When technology advances far enough, we’ll be able to make minds far surpassing human intelligence.’ Now, it’s clear that if you’re baking a cheesecake, how large a cheesecake you can bake depends on your intelligence. A superintelligence could build enormous cheesecakes: cheesecakes the size of cities. And Moore’s Law keeps dropping the price of computing power. By golly, the future will be full of giant cheesecakes! I call this the Giant Cheesecake Fallacy. It happens whenever someone leaps directly from capability to actuality, without considering the necessary intermediate of motive. But then, most futurists are part of the entertainment industry, telling wonderful stories about the future, and the first rule of storytelling is to be specific. A story, to be entertaining, cannot be vague. So, if you don’t know, you make something up. Saying AIs will cure your cancer, or AIs will take over the world, sounds much more interesting than saying, I don’t know, or it depends on the initial conditions of the AI.
Giant Cheesecake Fallacies are another case of trying to reason about the future by asking what AIs will want, forgetting that the space for possible minds is much wider than a tiny human dot. In principle, most people are willing to agree that other minds can be different, until you present them with some concrete difference, and then they’ll say, Oh no, that’s impossible. Sooner or later, people unconsciously use the empathy trick and they believe the answer they get back from their own brain is the only answer. For example, I sometimes say, sort of to annoy people really, that you should never really trust a politician unless you have seen its source code. In our current world, there is no way to be sure that your elected representatives are working on your behalf and often they aren’t. So, maybe we should replace the president with an open source artificial intelligence so we can all view the source code and make sure the AI really does care more about civil rights than holding onto power.
A typical response I get is that if you hand over power to an AI, the AI will become a tyrant. Absolute power corrupts absolutely. But, human beings are specifically evolved to be corrupted by power. When it comes to politics, human beings are defective by design. We are adapted to deceive ourselves–that we are taking over for the good of the tribe, and then use the political power to grab the best food and mates. There may even be a specific circuit in the brain somewhere that makes human beings want to abuse power. And it might be that all we need to get the right government is to install circuit breakers on our politicians. It probably isn’t that simple in real life, but we haven’t tried it, so, who knows. Maybe it is that simple in real life.
Evolution adapted humans to respond to power by abusing that power for personal interest. If I was building an AI from scratch, it would be just as easy to shape the AI, so that instead of responding to political power by being corrupted, it responded by using the power as little as possible. In fact, somewhere in the unimaginable vastness of mind design space, there is a mind that responds to political power by singing a tune and doing a little dance. And if you think that sounds odd, think how odd it would look to an alien species that didn’t have a sense of humor if I smooshed a cream pie into my face and the audience started laughing. He did what? And what are they doing?
All previous political systems have suffered from the crippled handicap of being made entirely out of humans. If you have new raw materials, you can build new things. If you don’t realize how wide mind design space is, you’ll miss all sorts of interesting possibilities. 300 years ago, democracy was a wild-eyed, crazy idea. The whole notion of having written laws is just a couple of thousand years old. The human species is just a few dozen millennia old. Things have not always been the way they are now. And they’re not going to stay the way they are now. The scary truth is that there are no limits. With enough knowledge, you could build any kind of mind you could imagine. It is as free as the art of computer programming itself. There are as many degrees of freedom in building minds as there are in making arbitrary computer programs or arbitrary configurations of matter, because infinity equals infinity, equals infinity.
Now, admittedly, right now we don’t know how to build any minds at all. Neither augmented humans or pure AIs. This ignorance is a temporary condition. It won’t be long before the absolute freedom starts. And then you are going to be faced with a choice. There are infinite possibilities, but which ones should we make real? For every mental transaction you can imagine, there is the bizarro mind that does it exactly the opposite way, and stranger minds that do things which aren’t even recognizably the opposite. We are faced with absolute freedom, and rather than go into existentialist shock mode, I suggest that we deal with it and move on. I mean, all you need to figure out is what you want out of life. Shouldn’t be too hard, right?
There is a final point on the importance of cognitive technologies, which is that once upon a time, the way the world used to work was that intelligence created technology. Brains made space shuttles. Now, suppose you had someone with a brain-computer interface that makes them substantially more intelligent. They should be able to create more technology. What kind of technologies might they create? Flying cars? Cancer cures? One good bet is that they’d use their enhanced intelligence to create the next generation of brain computer interfaces. Cognitive technology closes the loop between intelligence and technology, creating a positive feedback cycle. The smarter you are, the more cognitive technology you can invent to make yourself even smarter. It’s a tipping point. The purest form of this positive feedback cycle would be an AI rewriting its own source code–what the mathematician I.J. Good called an Intelligence Explosion. The AI redesigns itself to make itself smarter, which increases the AI’s ability to redesign itself–lather, rinse, repeat, FOOM!
I.J. Good’s Intelligence Explosion should not be confused with Vernor Vinge’s singularity. Vernor Vinge’s singularity is the breakdown in our model of the future that occurs when we try to extrapolate our model of the future past the point where the model predicts the existence of entities smarter than us. I.J. Good’s Intelligence Explosion should not be confused with Ray Kurzweil’s accelerating change. The idea of an Intelligence Explosion does not logically require nor logically imply that 1970-2000 was a time of greater change than 1940-1970. It’s agnostic on that thesis. The first AI to improve itself could conceivably be created during an age of accelerating technological progress, or an age of slow but steady progress, or even a global stagnation of most technologies. And once we do go past the tipping point, it is not necessarily true that progress will be exponential. Exponential is a specific mathematical curve. It is not correct to say exponential when you mean accelerating or positive second derivative.
But, nonetheless, if you’re wondering why the graph I showed earlier only showed AI creeping up to the rough vicinity of human intelligence and didn’t show what happened after that–it’s because an AI that is as smart as its programmers is smart enough to take over the job of improving itself. Not every positive feedback cycle in the universe follows a smooth exponential curve, but they do tend to accelerate.
For those of you in the audience who have never heard of me before and have no idea what the Singularity Institute for Artificial Intelligence does, to make a very long story very shor, we’re trying to take humanity through the intelligence explosion safely, by figuring out how to design a Friendly AI. We want to reach into mind design space very precisely with very exact targeting and pull out an AI which is knowably nice, good, helpful, and furthermore, stays that way while it improves itself. Somewhere in mind design space is an AI such that we will not regret creating it, but to find it will take new math. So, the Singularity Institute is ideally created to sustain people who want to do research on a subject that is not going to have instant returns in three years, as Professor Marvin Minsky has extremely correctly bemoaned. New math takes a long time to develop. For much more on the subject, see the Singularity Institute website and/ or my personal website at yudkowsky.net.
To sum up, intelligence supports everything we do as humans, from math to persuasion, from chess to music. It’s the root of all our technologies, from mp3 players to fire. The rise to human intelligence has significantly transformed the world. We’ve carved our faces on the mountains and set our footprints on the moon. Whatever touches on intelligence reaches down to the roots and picks up the tree. A transhuman is a transhuman mind. Anything else is a side issue. Cognitive technologies upon up a vast space for possibilities unlike anything in human experience. It closes the loop and creates a positive feedback cycle. It may even let us create a self-improving AI, a fast road to superintelligence. And if so, we need to work out the math in order to make that AI friendly. And, above all, intelligence is the master trick.
You could try to solve poverty and AIDS and environmental problems and various other diseases by applying your own intelligence, which is a worthy endeavor. Or, you could turn the human gift upon itself, fathom the power, make it burn brighter, or take on new form outside of humans and shape that form. And in this way, reach into mind design space, pull out a bigger hammer, and hit the entire set of problems at the same time. To grasp the true strangeness of the future, ignore the cool devices with blinking lights, forget about mere superheroes, and focus on technologies that impact upon the mind. This has been Eliezer Yudkowsky for the Singularity Institute for Artificial Intelligence.
Related article: 7.26.07 The Coming Merger of Human and Machine: Ray Kurzweil outlines some of the foreseeable implications of accelerating technological progress.