If the Future Contains Minds That Are Smarter Than Us…
Posted by Jeriaska on October 18th, 2007
On October 10, a Facebook group composed of Silicon Valley residents met for an informal dinner conversation with Singularity Institute Co-Founder and Research Fellow Eliezer Yudkowsky. The topic was the future of intelligence: Before we can have any understanding of what to expect if the future contains minds far more intelligent than we are, we might consider what it is that makes a mind intelligent in the first place.
The following transcript of Eliezer Yudkowsky’s informal dinner conversation piece recorded October 10, 2007 has not been approved by the author.
If the Future Contains Minds That Are Smarter Than Us…
If the future contains minds that are smarter than us, that is a break with the past of a kind that isn’t implied just by, say, agriculture. Another important observation is that up until this point, our intelligence has been the source of our technology, and to some extent we’ve improved our intelligence by making discoveries about how to think. Science, for example, is a discovery about what patterns of thinking work best for exploring the universe and building technology.
If you look for example at the difference in brain size between a human being and a chimpanzee, we’ve got around three times too much brain and six times too much prefrontal cortex for a primate our size. The implication being that you can go from chimp intelligence to human intelligence using a sixfold increase in processing power, which is about the amount of territory that Moore’s Law covers every five years. The moral there is that if you’ve actually got intelligence making technology that improves intelligence, you are closing the loop and creating a positive feedback cycle, which is not something that’s really existed before. If it has existed, it’s been on a very high level of “This is how you write a science paper” as opposed to “Okay, we’re going to redesign neurons to run a million times faster.”
These all present some interesting issues and form part of the family of things you’re interested in if you care about where intelligent life goes on earth from here, and colonizing the galaxy. The basic answer of why I should be interested in this is that if you are an expected utility maximizer, there is more expected utility in building superminds that can colonize the galaxy than there is in, say, inventing a new and improved laundry detergent. That’s the basic pitch of the Singularity Institute. This is what the human species ought to be paying attention to instead of watching the Michael Jackson trial on television. And with that I will open it up for questions, comments, outraged objections, and so on.
Do you think that the primary difference between chimp intelligence and human intelligence is simply the six-fold increase in raw material? Is it scale or complexity?
My guess is that it’s mostly complexity. And the reason I talked about the six-fold increase was to emphasize that you could in fact support that complexity using only a six-fold increase in hardware. Of course that is going to reflect some kind of balance between the evolutionary ease of supporting more brain tissue versus the evolutionary ease of supporting more genetic complexity. If evolution could just go out and buy one hundred times as much hardware the way Facebook can buy one hundred times as much hardware if it has a business case for it, then our brains probably would have expanded a lot more and had less software complexity. If brain tissue was more expensive we probably could be running human intelligence on a chimp brain, it just would have taken longer to evolve. That’s my guess.
If it turns out that it’s not chemistry but quantum mechanics that drives human consciousness, how does that affect the proposed timescales for the Singularity?
I think it is worthwhile to distinguish between properties that we usually associate with personhood, including consciousness, versus what it takes to make a very powerful optimization process that can steer the future and create its own technology. I think it is extremely unlikely that we are doing anything quantum significant. As long as it’s not actually magic, we can do it using technology. If it is actually magic, we’ll call in some sorcerers.
I think that a lot of what is being said right now about consciousness bears a remarkable resemblance to what was being said about life in the age of vitalism. There is a wonderful quote by Lord Kelvin where he goes: “In its ability to create generation after generation of trees from a single seed, in our ability to direct the motions of our muscles by conscious will, life is infinitely beyond the reach of any scientific investigation hitherto entered upon.” Basically, the thing is, if you are ignorant about a phenomenon, this is a fact about your own state of mind, not about the phenomenon. This is the fundamental and fully general reason why throughout human history there have been many mysterious questions but no mysterious answers. This I believe will turn out to hold for intelligence.
What are your thoughts on the timing of self-improving artificial intelligence?
Consider how hard it is to say, “I think someone is going to build a heavier-than-air flying machine eventually” versus “The Wright Brothers are going to do it in 1903.” Doing timing is a lot harder than predicting that it’s going to happen sometime. A lot of questions are asked about what will an AI do, which ignore the point that there is more than one possible AI. When we talk about AI’s, we’re really talking about mind design space in general. So if you imagine this enormous space of mind designs the size of this whole restaurant, then humans are all packed into this one tiny little dot in the space of possible minds, because as a sexually reproducing species we need to share all the genes that specify our complex machinery or it won’t get built. In other words, if there were ten different genes for eyes, which is really an underestimate, and we all only had those genes at 90% probability, complete functioning eyes would be very rare. So all of the machinery of the human body, including the human brain, is going to be universal in the species, except for like one or two minor adjustments that are under selection at any given time.
As humans who spend all day interacting with other humans, we tend to automatically underestimate the range of possibilities. In fact, we probably have built-in hardware for estimating what another mind will do, which assumes that the other mind is behaving like a human mind. We are using empathy, putting ourselves in the other mind’s shoes.
Have we really hit upon the structure of logic and it’s just a matter of scaling that up now?
General is not the same as generic. This knife has no preconceptions because it has no brain. So if you take a mind and you start eliminating its preconceptions what you end up with is not a fully general intelligence but a rock. So the modern understanding of cognitive science and particularly evolutionary psychology is that we’re not general intelligences on top of a lizard brain, but that we’re evolved, directed, biased, emotional, social and high-level deliberate intelligences on top of emotional lizard brain. In other words, it’s emotions all the way up. Our ability to learn how to drive a car, program a computer, and eat with a knife and fork has not changed our tendency to use all of these things in order to gain wealth and social status, protect ourselves and our loved ones, versus deliberately rendering yourself a pauper and walking off a cliff. It is still being directed toward particular ends. The Soviets thought that you could raise someone to be a perfect Communist worker, and it didn’t work. There were still built-in instincts. We’re not generic intelligences. We have the ability to understand a very wide, very general variety of phenomena and adapt them to our own purpose but we’re not blank slates. We do this in a very complex, biased and even emotional fashion.
To get back to the original question of why be interested in the Singularity, we could end up in a very different world or not end up being anywhere at all depending on exactly what kind of AI is built and starts improving itself. That’s where the expected utility comes in. These are big events that could conceivably go any number of different ways and that actually depend on how we go about it. Did we understand enough cognitive science and computer science that when we were building the AI we could make strong statements about where it was steering the future and what it was using its intelligence for? Did it maintain these properties in rewriting its own source code? This is my own, declared mid-long-term research interest going by the name of reflective decision theory.
From my own perspective, whether this occurs in 2040 or 2100, it still matters to me what humanity does with itself. Even if you told me “You’re definitely not going to live to see it,” it would still matter to me. But I think there’s a pretty darn decent chance of living to see it. Because, hey, it’s like the whole galaxy, man. It’s our whole future light cone. I see this as being the meaning of having been born a Homo sapiens, one of the first sentient species ever to exist. Hopefully the vast majority of sentient beings that will exist will be born on a different galaxy to intelligent designer parents. I figure that having found myself in the very surprising position of being a Homo sapiens, I’m going to make the most of that, and not pass up my chance to influence the destiny of the galaxy.
What has to be in place to enable this to occur?
It depends on who you ask. Kurzweil sees it in terms of a brain-computer interface/ brain scanning/ brain simulation deal. I see it in terms of ten or a hundred guys and a brain in a box in a basement. I see it as pretty much a pure computer science issue. That’s my guess, that we are separated from this strictly by understanding we don’t have. It appears to me that there is typically about a 20 year time lag between when somebody has the brilliant AI idea and its starting to have commercial applications. Like the robot that won the DARPA Grand Challenge, that wasn’t based on any brilliant new insight into how the mind works. It was based on a brilliant old insight into how the mind works, the Bayesian revolution that got started in 1987.
Now, the thing is, there has actually been serious and even steady progress in artificial intelligence and our basic understanding of the mind. It goes largely unreported because the interesting stuff does not make for a nice newspaper article and by the time it’s being used in those neat robots that the media wants to pay attention to, the ideas behind it are 20 years old. They are no longer breaking news.
We have one instance of how to make minds already. Do you think the fastest path is through computer science or understanding how the information found in the genome yields a brain that contains a mind?
Consider the Turing test. The Turing test is a thought experiment that you use as a philosopher if you have got absolutely no idea what intelligence is. It’s sort of like, well, I believe in principle that we can build a heavier-than-air flying machine because birds are not magic. So if I can build something that is indistinguishable from a bird, I will say that it flies. This logic is valid. If you can build something that is indistinguishable from a bird, it will fly. It’s not very likely that you will be able to build something indistinguishable from a bird, or indeed that flies at all, until you can conceive of flight as a more abstract function apart from birds. The first attempted aircraft had beaks, and flapping wings, and feathers. But you didn’t actually get flying machines until you understood flight apart from birds. Even now we probably can’t build something that is so close to a bird as to fool a professional ornithologist, but we have got supersonic aircraft and even space shuttles. I think that’s how it’s going to go with artificial intelligence. The business about needing to know how the human brain works is an attempt to come to terms with the very large gap of their own understanding of how intelligence works, rather than a savvy prediction of how it’s going to play out based on technological history so far. It’s that basic gap in understanding that makes it seem like magic. But the confusion is in our minds, not in reality itself.
Over time, our understanding of cognitive science has not gotten any less sophisticated. It’s made huge leaps. It is still making huge leaps. But I think humans tend to see the scale of intelligence as ranging from village idiots up to Einstein, and anything that’s dumber than a village idiot drops off the end of the scale. It just looks really stupid. But the real scale of intelligence we’re interested in is the scale that starts with the rock at zero and then goes to an earthworm, then goes to a spider, then a lizard, a mouse, a chimp, and all humans. All humans are packed into one tiny little narrow area. AI is slowly creeping up. It’s past earthworm, it’s probably even past spider, well on its way to lizard, but to a human being it’s all just dumber than a village idiot.
You don’t see artificial intelligence being on a local maximum path where it could hit a wall before reaching human-level intelligence?
Well, the past is a lot easier to predict than the future. It’s been on its way. I see no reason for it to hit a sudden barrier. I’m pretty sure that Bayesian networks are interesting, but to actually defend the statement I would probably have to get out a blackboard. Going through the math you can say, “Oh yeah, humans are doing this. This is why all those artificial intelligence systems that we were trying to make before failed and humans could do it off the top of their head.” It’s that kind of insight. Some of it is being abstracted from a human model and some of it is genuinely pure computer science. We never really understand any aspect of cognition until we can see how humans are doing it wrong. That’s the rule I would offer. As soon as we really understand something, not “Humans are doing this magical thing and it’s probably because of emergence!” As soon as we can actually get it boiled down to equations or programs we can see humans are doing this, sort of, but they’re also getting it wrong in a lot of places. That’s why I don’t think that the first artificial intelligences made are going to be humanlike, because I can already see that you would have to be on crack to design something to work like a human does.
Can we really be sure that machines will be capable of intelligence when it’s hard enough to define what intelligence is in the first place?
If you have insufficient evidence to settle something, that doesn’t mean you can make up your own opinion, it means there is insufficient data for the correct answer. If my AI rewrites its own source code a million times in a row, then builds its own nanotechnology and flies off to Alpha Centauri and converts the star there into a Dyson sphere, I bet it was intelligent.
Consider a car. For the purpose of concreteness, let’s say a Toyota Corolla. Now, out of all the different ways you could arrange the atoms in that car, very few of them will get you from point A to point B as fast as the Corolla. It’s not the Corolla is optimal, but that if you consider the size of the brute force search space, as a means of travel the Toyota Corolla is clearly a very small point in that search space. From what I’ve told you so far, we cannot infer the presence of intelligent design because this is also true of a horse, which evolved. But we can infer the presence of something that we might call “optimization.” And if you wanted to measure optimization quantitatively, suppose you already know your utility function or your preference ordering over the solution space, then you can count up all the solutions that are as good or better than the solution that you have versus the total size of the solution space, divide one by the other, take the log base 2. That will give you the optimization in bits.
But wait, there’s more. If you’re familiar with the minimum message length formalism, the idea is that a hypothesis is useful if it lets you compress your description of reality, so here the notion is, let’s say you’re meeting aliens or some kind of completely alien artificial intelligence and you don’t know its utility function, but you find that it’s easier to describe them if you transmit a description of their output. You can compress the description of its output, of what it created or what kind of plans it took, by describing its utility function first and then describing how good of a solution it got. “Water wants to flow downhill” does not compress the message, because water flowing downhill is not a solution to any problem that can be described more quickly than saying “Water flows downhill.” On the other hand, a car is a very complicated solution to a problem of going from point A to point B that I can describe to you much more quickly than I can describe all the elements of the car design. That is what makes it useful to describe the car as being “optimized.”
Now, the car might still be evolved, so I would say that what distinguishes intelligence from evolution and other mere powerful optimization processes that are not intelligent is that intelligence is a lot more efficient than evolution. The characteristic speed of evolution in mammals, for example, might be one bit of information in the species gene pool per generation, because two parents have four children and then two of them die. Or two parents have eight children and then four of them die. That respectively gives you one bit or two bits of information in the entire gene pool per generation. Whereas a human programmer by thinking abstractly about the problem, compressing regularities and manipulating concepts that describe reality modularly can generate a complex piece of code with hundreds of bits of information in a single afternoon. So I would say that human intelligence is computationally efficient optimization that works by modularly modeling the outside world.
How is that different from saying the visual cortex is intelligent?
The visual cortex is intelligent. It’s just that the whole human mind is more intelligent. It steers the future more powerfully than a chimpanzee who also has a visual cortex but does not have the other property. You will never find a car by chance because it’s such a small part of the search space. It can’t be hit randomly. It has to be hit by an optimization process such as evolution or intelligent design. We could define intelligence as optimization power divided by computation. So if we were to measure the intelligence of evolution, it would come out very low, even though its power as an optimization process is pretty high because it took millions and billions of generations to produce those cumulative selection pressures that produced the horse.
What would be the result of creating something with infinite intelligence?
I’m talking about very large, but finite intelligence. I come from that branch of mathematicians who would pretend not to know what you mean by “infinity” until you show me something infinite. According to our current understanding of the laws of physics, you can’t do that. But it probably does not matter all that much because if something is intelligent enough to take stars apart, it’s intelligent enough to yield interesting consequences for discussion. How much intelligence really matters to us relative to what we can understand as interesting?
When does the positive feedback loop of self-improving intelligence finally hit a wall?
When you’ve optimized all the atoms within reach to the optimal configuration for atoms that you can afford to compute, according to whatever utility function the AI had after self-modifying, which, if the programmers knew exactly what they were doing, might even be something humanly comprehensible, like help people. And if they didn’t know what they were doing, it might come out as, convert your future light cone to paperclips. Different optimization processes have different targets. You can see this in evolution, which is sort of blindly aiming for reproductive fitness of its genes, versus humans, who are trying to optimize hundreds of different goals simultaneously and can learn more by being raised with particular moralities. Depending on which of 2^billion possible AI’s you write, you end up with 2^billion possible futures. This is what makes the problem interesting.
