Controlled Ascent in AGI
Posted by Jeriaska on September 27th, 2007
SIAI Interview Series – Ben Goertzel
The following transcript of the SIAI Interview with Ben Goertzel has not been approved by the author. Video and audio are available at the Singularity Institute website.
“I think there is a problem any time you use English terms, or terms in any natural language, to describe technical concepts, because natural language terms are just intrinsically ambiguous. That’s the way human communication is.”
“We have not yet built any kind of tool, be it a computer program or a hardware system or a vat full of chemicals, we have not yet built any kind of artificial system that can achieve a variety of complex goals in a variety of environments. But a number of us who are AI researchers are coming to believe that this may change in the relatively near future.”
Controlled Ascent in AGI
“What is artificial intelligence?”
To answer the question “What is artificial intelligence?” we really have to start with the question, “What is intelligence?” This is something that psychologists and those who study human intelligence do not fully agree on, but the way that I have come to define intelligence in the context of my own work is that intelligence is the ability to solve complex goals in complex environments. Of course, that is a matter of degree. Humans are much more intelligent than cockroaches, while cockroaches are more intelligent than bacteria, and it is quite possible that there are other minds that are much more intelligent than humans because they can solve more complex problems and achieve more complex goals in different contexts.
If you buy that conception of intelligence, the next question is, “What is artificial?” An artifice is a tool, so the term artificial intelligence carries within itself the idea that we are building systems that can achieve complex goals in complex environments, which are built out of objects that we have created. This is something that we have had limited success at so far, in the sense that we have managed only to build systems that solve very narrow goals: play chess, drive cars, diagnose diseases. We have not yet built any kind of tool, be it a computer program or a hardware system or a vat full of chemicals, we have not yet built any kind of artificial system that can achieve a variety of complex goals in a variety of environments. But a number of us who are AI researchers are coming to believe that this may change in the relatively near future. We are nearing the day when we may be able to create artificial intelligence with a kind of generality of goal achieving behavior that we now see only in humans.
“What is artificial intelligence versus artificial general intelligence?”
I think there is a problem any time you use English terms, or terms in any natural language, to describe technical concepts, because natural language terms are just intrinsically ambiguous. That’s the way human communication is. So you are never going to get completely precise using a natural language term for a technical topic. The concept of energy in physics is a little bit different from the concept of energy in everyday life. So I think you have to take terms like “artificial general intelligence” in the same light. There are precise definitions for these things within the science of AI, and these definitions are given in mathematics. They do not necessarily match exactly with the English language definitions.
Having said all that, what I think of as artificial general intelligence is a system that can solve a variety of different problems, some of which may never have been thought of by the programmers who created the system. Just like when a baby is born and grows up, the baby can solve problems that its parents had never heard of and did not even know about. It is quite different from the AI programs that most AI researchers are playing with today. Now, if we write a program to play chess, what that program is going to do is play chess. The program will never play go. The program will never play fish or random chess or checkers, unless some programmer tells it to. On the other hand, if you create a system with a higher level of general intelligence comparable to that of a human, once you have taught the system to play chess, if you then try to teach the system to play checkers, the system is going to learn it more quickly than if it had never seen chess or any other game before. It is going to be able to generalize from one problem domain to another because it has a general intelligence that spans particular domains.
I think we have to realize that infinitely general intelligence is not possible in this universe. Infinitely general intelligence would require infinite computational power. We humans are not infinitely general. 30% of the brain is specialized for vision processing. Most of our brain deals with very specialized things: seeing, hearing, moving, mapping where our body is, recognizing deception in other people, understanding language. Only a very small part of our brain is focused on generalization and broad scope problem solving. And this, in fact, is something we are not very good at. Most humans cannot solve many problems beyond what they have been taught or have been evolved to do. General problem solving to an extent is a skill that humans learn through a lot of practice and education. So, I would not want to exaggerate the level of general intelligence that humans have. On the other hand, it is much greater than that of Deep Blue the chess program or the cars that drive in the DARPA Grand Challenge.
“When will we see advanced artificial intelligence ?”
There are two basic approaches that could be taken to making artificial general intelligence. One is to emulate the human brain in computer software, and the other is to create a non-brainlike AI system that achieves intelligence by principles somewhat different from those that the human brain uses. Technology today is preparing us in different ways to achieve each one of these different paths in the relatively near future. In terms of the brain emulation path, two things are required to achieve AI – better mapping of what is happening inside the brain and better computer hardware. Both of those things are happening at a pretty rapid pace, and this is one of the points that Ray Kurzweil has made very effectively in his book The Singularity is Near. Moore’s Law and associated patterns in the evolution of hardware technology show that the computers we get are getting faster and faster and faster, with more and more memory, very quickly. If you look at the growth curves for brain scanning, we are getting greater spacial and temporal precision in terms of our understanding of what is happening inside the human brain. If you put those two things together, you arrive at the conclusion that in a few more decades we will be able to map what is happening in a human brain and simulate that inside a computer.
Well, you might say, that’s not very interesting. We already have enough people. What do we need digital people for? On the other hand, think of it this way, what if you could take the hundred smartest people in the world, copy each of them a thousand times, and run each of them a hundred times faster than they naturally run, and observe everything going on in their brain while that is happening? This is going to lead to a lot of amazing scientific advances very quickly. So I would argue that once you get a good emulation of a human brain, you are going to get all kinds of non-human AI systems. Basically, the Singularity is near.
Now, for the other approach to AI, the approach where you are not emulating the human brain but are trying to make a thinking machine through different principles, there are also important advances that have happened that I think make it much easier to conceive of what an AGI is going to be like, and much easier to chart a path toward AGI now than it was, for example, ten years ago. The case is a little more indirect than in the situation of human brain emulation, but I think it is quite clear nonetheless. The first line of evidence I look at comes from cognitive science. If you try to make a flow chart of what happens in the human mind, we have perception, action, short-term memory, long-term memory, reasoning, learning, perceptual learning, motor learning, a couple dozen boxes in your diagram with connections between them. We have some understanding of what happens inside of each of these boxes. All this knowledge is incredibly more solid than it was 20 years ago, even more than it was ten or five years ago. We now have a pretty good understanding on the cognitive psychology level of what is going on inside the human mind. Now, this does not tell you in itself how to build a thinking machine, but it tells you a lot of what the high-level architecture is that you need to build a thinking machine.
Then, complementing that, we have an awful lot of advances in artificial intelligence algorithms and data structures. This comes from the narrow AI work that many AI researchers have done in solving particular problems, like playing games, diagnosing diseases, driving cars, predicting the stock market. While none of these things in itself is going to get you to general intelligence, we’ve got a heck of a lot of wonderful algorithms. We are accessing things from memory, recognizing patterns, doing reasoning, doing learning. The approach I am taking in my own work in artificial intelligence is to take the high-level functions of mind as understood by cognitive psychology and figure out how to realize each one of these high level functions, such as reasoning, learning, perception, action, using cutting edge computer science algorithms, so we then have a convergence of cognitive psychology and computer science algorithms. You get this integrative approach to artificial intelligence, and then, of course, feeding into that, you have Moore’s Law and associated regularities in the advance of computer hardware.
“What work are you doing in the field of artificial intelligence?”
In terms of my formal background, I have a PhD in mathematics. I was a professor for eight years in various universities in mathematics, computer science, and cognitive science. In 1997, after eight years in academia, I joined the software industry. I have been building a number of narrow AI applications in a variety of areas including bioinformatics, computational finance, natural language processing, and data mining. In parallel we are doing this applied narrow AI work. I have been working on my own grand scheme to create a thinking machine, which I call the Novamente cognition engine. “Nova Mente” for “new mind.” It’s a big project. It has been going since 2001 at a moderate rather than rapid pace just because of the modest funding attached to it. We have been funding the work from our own AI consultant company. But my work on the Novamente cognition engine is what really convinced me that artificial general intelligence could be as little as five years off from now if a really concerted effort was put into realizing the Novamente system or some other analagously workable AGI architecture.
“What is your interest in SIAI?”
I have known Eliezer Yudkowsky for a number of years now, since maybe 1999. We have gone back and forth over the years about our sometimes different, sometimes similar theories of artificial intelligence, the Singularity, the ethics of AI development. I have followed with interest the development of the Singularity Institute from the very beginning. Last year, in 2006, I began talking with Eliezer and Tyler Emerson about joining forces in some way, so as to better realize our common goals of advancing safe and beneficial artificial general intelligence in maximizing the odds of achieving a positive technological singularity. Eliezer had been working for a long time on his own approach to Friendly AI: making artificial intelligence systems that are in some sense guaranteed to be nice to human beings and to obey positive moral values as they progressively get smarter. I had been working on my own approach to artificial general intelligence, although with a strong ethical interest in mind, in terms of making safe and beneficial AI, but with more of a practical engineering focus. Eliezer has been mostly doing theory. I have been doing some theory, but I have also been working with a team of programmers to build systems. To make a long story short, I became involved with the Singularity Institute as the director of research, my goal being to lead a more practical AGI research program within the Singularity Institute to complement the more theoretical work that Eliezer had been doing.
One of the things I would like to do is to lead the development of open source tools for artificial general intelligence. So, I would like to take a number of the software tools we have developed within my AI company Novamente and, with the cooperation of the Singularity Institute, release some of these in an open source format, so that AI programmers in academia, industry, and independently can collaborate in developing tools that will help to develop safe and beneficial AI. The role of the Singularity Institute there would be to serve as the ringleader and orchestrate the work done by various open source contributors, and also to provide guidance to be sure that the development occurs in a safe and beneficial way. My thinking is that getting more of an active collaboration with the Singularity Institute on the part of the overall AI and AGI research community should be a very good thing. It can accelerate our progress toward AGI, while being sure that SIAI, which has developed a lot of insight into how to keep AI on the right ethical path, can play a role in guiding what happens as the path to AGI unfolds.
“How can ethics be programmed?”
To ask how you can translate ethics into ones and zeros in the mind of an AI is much like asking how you can translate ethics into action potentials and dopamine levels in the human brain. When I teach a child ethics, as I have done with my three children, I don’t try to figure out how to translate the rule “Don’t hit people on the head” into a certain collection of action potentials in their brains. I instruct them by example, through language. That instruction works in part because their brains are architectured so as to receive those instructions and react to them appropriately, because the human brain has a very powerful in-built module for social interaction, for emulating and empathizing with others. Similarly, in order to inculcate appropriate ethics in AI, one is going to need to combination of explicit instruction and an appropriate architecture designed to receive that instruction. I do not believe the best approach is going to be to program in ethics. In that case, you would have a series of very rigid ethical injunctions, which would not be organic, flexible and generalizable enough to guide the behavior of a generally intelligent system in the real world. But I do believe it is important however to build an AGI architecture with ethical behavior in mind at the time that you build it so that it will receive ethical instruction in an appropriate way that will really wind up guiding its behavior. We have followed that approach in designing the Novamente cognition engine. We will also follow that approach in any open source AGI software that the Singularity Institute releases.
“Must AI’s understand natural language in order to be ethical?”
I think that in order to understand human ethics, an AI will have to be fully involved and embedded in human culture. I think that the notions of good and bad, violence, aggression, freedom, choice, self, will, none of these are mathematical formal concepts. These are all cultural concepts that derive their meaning only from the rich web of interactions that human beings have with each other, with our bodies, with our world. An AI is going to have to be enmeshed in that world in order to really understand the human sense of ethics in a practical way that is going to guide its behavior. But I do not think that is enough. Of course you could make an AI fully embedded in the human world and it could wind up doing some very bad things. It has to have that embedding, that teachability, and it has to have the right architecture. This is one area where I believe a non-humanlike AI can be fundamentally superior to a humanlike AI, because human beings are simply not that ethical. I have known a lot of very unpleasant human beings in my life, and I have also observed that some of the sweetest, nicest, most ethical human beings I have known can become really rotten bastards in the wrong situation.
“Assuming strong AI is likely, do you anticipate a hard or soft take-off?”
I don’t pretend to know whether there is going to be a soft or a hard take-off. I think that anyone who does think they know that is fooling themselves. I think the potential for a hard take-off is very real. It would be quite possible to create an AI system in such a way that it could modify its own underpinnings recursively and rapidly so as to in five minutes go from human level intelligence to godlike intelligence. On the other hand, I think that is very unlikely to happen by surprise, unless you have built your AI system very sloppily. I think that if you build an AI system carefully, as we have done with Novamente, and as we will do with any Singularity Institute AGI projects, open source or otherwise, if you build your AI carefully, you can control the rate of improvement of intelligence. You can achieve what I would call a controlled ascent, whereby the level of intelligence of the AI system will improve only a certain amount per year. My own tentative view is that this is the most ethical and sensible way to proceed. Unless you are under the threat of invasion by superintelligent aliens from space, or some horrible situation where you need a god immediately, much better to proceed relatively slowly over some period of years, maybe even decades or centuries. Let the human race adapt to the new situation, and let things happen in a way that is comprehensible and humane on the human timescale.
I think this should be possible, assuming your AGI system is built correctly. But this leads to all kinds of intriguing and difficult ethical questions. What if the Singularity Institute chooses in the future to take a path of controlled ascent? What is to stop some other guy from making an AI and trying to jumpstart a hard take-off? Do we somehow need an AI thought police to stop AI’s from ascending too quickly? If so, how would you create that? Is that a problem that you pose to your first AI as it is ascending in a controlled way? Once it is merely twice as intelligent as human beings, do you tell it, okay, your job is to figure out how to stop any other AI from advancing faster than you? I think these are very difficult questions. They are very deep, very important questions. There are a lot of possible answers. I don’t claim to have all the answers. I don’t think anyone at the Singularity Institute does claim to have all the answers. One thing that we would like to do at SIAI is to involve the broader community – AI researchers and human beings at large – in an intensive and profound dialog on these issues. What do you really want to do? How fast should we ascend? How fast should anyone be allowed to ascend? How slowly should anyone be allowed to ascend? What restrictions could it make sense to place on AI’s that are hard take-off capable? These are deep issues, and my own view is that no elite group can fully be trusted with these things. I think it has to be a society-wide process of decision and understanding. But I do think that a group like SIAI can be critical in catalyzing this kind of societal understanding process.
