Future Current

SIAI Interview Series - Eliezer Yudkowsky

The following transcript of the SIAI Interview with Eliezer Yudkowsky has not been approved by the author. Video and audio can be found online at the Singularity Institute website.

The Space of Possible Minds

I’m Eliezer Yudkowsky, a research fellow of the Singularity Institute for Artificial Intelligence, and I helped found the organization back in 2000. As a research fellow of the Singularity Institute, I’m supposed to first figure out how to build a Friendly AI and then, once I’ve done that, go and actually build one. In practice, this currently consists of about half my time reading math books and the other half of my time working with research associate Marcello Herreshoff trying to line up all the major unsolved problems and at least figure out approaches to them.

The Singularity has been given a lot of different definitions. The three major ones are: accelerating change, the breakdown of the model that happens when we’ve got AI’s running around that are smarter than we are, and the intelligence explosion, which is what happens if a smart mind can improve itself, and make itself even smarter, and improve itself even further. Later, rinse, repeat. The intelligence explosion is usually what I mean when I say, “The Singularity.”

The aftermath of an intelligence explosion is a bit difficult to predict, because it involves things that are smarter than we are. I think it depends very strongly on the initial conditions. In other words, you have to set it up in exactly the right way, and then nice things happen afterwards. if you don’t set it up exactly the right way, you’re doomed.

The very simple existence proof is, let’s say you have Gandhi. And you offer Gandhi a pill that makes him want to murder people. Gandhi will refuse this pill because presently he likes people, and therefore he doesn’t want people to die, and he knows that if he takes this pill he would kill people. So, that’s the sort of very simple demonstration that if you have a mind and it starts out wanting to do something, it’s not going to want to modify itself away from wanting to do that. Unless something odd is going on.

It seems reasonably likely that the whole fate of the human species is going to end up depending on this as a critical point. In other words, if we get this wrong, it doesn’t matter what else we got right. And, perhaps because it is so important, no one is paying any attention to it, with the exception of a very small handful of individuals, and that’s what the Singularity Institute is about. To get organized, full-time, dedicated thought focused on just this one problem: How do we come through this safely.

The problem is, of course, that you’ve got this potential, that you have computers lying around, and someday we will figure out how to program them. And even if we don’t, we have humans lying around, and someday someone will figure out how to upgrade them. And so you’ve got the potential for an intelligence explosion for recursive self-improvement: minds making themselves smarter and smarter. That’s the challenge. That’s what we have to come through, preserving everything that we care about.

Greater-than-human intelligence would be the result of an intelligence explosion. Probably vastly greater than human intelligence. If you look at human intelligence, it’s had a rather large impact on the world. Most of the things surrounding me here are the products of human intelligence. In fact, practically everything except the air molecules in the room and me are the products of human intelligence. Human intelligence had an enormous impact upon the world. We got skyscrapers, we got footprints on the moon, and that’s from essentially constant brains and mind power. So, if you go beyond human intelligence, you are taking a step beyond everything over the past 10,000 years.

All human minds are essentially the same make and model of car. We all have the same engines inside, we’re just painted different colors. When you talk about artificial intelligence, you’re talking about a space of possible minds that is enormously wider than what we are talking about when we talk about human minds. So, what happens as a result of these minds that are tremendously powerful, smarter than human, may depend on exactly which mind we’re talking about. Where in the space of possible minds we are. And our mission is to reach into the space of possible minds and pluck out a good one.

Well, the existence proof that you can have a moral machine is a human being. We’re not tremendously moral, but we are, shall we say, lawful physical systems. Our atoms are obeying physical rules, and yet, here we are. We think, we choose, and we seem to spend a lot of time thinking about the stuff called morality. So, that’s an existence proof that it can be done. It’s a sad but true fact that the things that get the most media attention are not always the most important. For example, I’ve read that the Michael Jackson trial received 133 times as much media coverage as the Darfur genocide. And this is not a mainstream issue. And it’s probably more important than a heck of a lot of mainstream issues out there. In fact, I would say that it is the most important thing, and that’s why I’m here working on it.

It’s the most important issue because it has a potential impact larger than everything the human species has done up to this point. We’re talking about going beyond what human intelligence can accomplish, and all that power will have a shape, and that shape will determine the impact. And that determines whether everything we’ve done in the last 10,000 years adds up to something for the future, for the galaxy, or just ends up as nothing.

The obstacle to creating strong AI and smarter-than-human intelligence is a piece of knowledge. And it’s very hard to predict when you are going to get a piece of knowledge you don’t have. It could be in a few years. It could be in a few decades. It could even be in a few centuries, although I very strongly doubt that. But the main thing to remember is that we don’t actually know how this is all going to play out, and therefore we should widen our confidence intervals. We should put wide binds around the problem. It doesn’t mean pushing it further away. It means not saying it’s going to happen in this narrow little interval. It could happen very soon. It could happen very late. That’s what you’ve got to say when you’re uncertain.

Most futurism is driven by the marketing departments, trying to set up their products as futuristic. They want you to believe the future consists of a bunch of shiny gadgets. Well, shiny gadgets are not as powerful as human intelligence. They’re just sort of byproducts of it: smoke trails that we leave behind us. I think that right now the human species is stuck in a sort of awkward phase where we’re smart enough to make tremendous promise for ourselves and not quite smart enough to solve them. The promise of artificial general intelligence is that if there is any problem that human beings have bounced off of, you can reach for a bigger hammer. And that would include the sort of reflex problems that everyone automatically thinks of, like world hunger, and AIDS, and global warming. And it also includes much deeper problems, like a lot of people being quietly, desperately unhappy most of the time. And it includes problems that we haven’t really thought about because we’ve never seen an alternative to them: like death.

The thing that I’m most worried about is not that somebody is going to maliciously and deliberately build an artificial intelligence that kills people. The thing I’m worried about is that it’s going to take very deep understanding and very precise work to actually reach into mind space and pull out a helpful mind. And I’m worried that someone’s going to underestimate the difficulty of this problem and proceed on a vague theory and get some exciting results, and encouraged by this, rush ahead, and doom us all. To put it bluntly.

The Singularity Institute was founded on the theory that in order to get a Friendly artificial intelligence, someone has got to build one. So, we’re just going to have an organization whose mission is: build a Friendly AI. That’s us. There’s like various things that we’re also concerned with, like trying to get more eyes and more attention focused on the problem, trying to encourage other people to do work in this area, but at the core the reasoning is, someone has to do it. That someone is us.

I think that a lot of AGI research has fallen into two modes. There’s a mode where someone is just starting out and they’re all bright and enthusiastic, and they attack the problem, and they bounce off it, and they give up on it, and they go back into the narrow AI business. And there is a pattern where people attack it, and bounce off it, and they don’t get discouraged, and they come up with another solution, and they attack it again. And I think that what we need, actually, is the ability to say, We don’t know how to solve this problem right away. We want to solve it. We’re going to sit down and do some deep thinking about it, and see if we can break it down into pieces and tackle the pieces. In short, conduct long-term research on it. Not just run out and try to do it right away, or alternatively give up right away.

On the research side of things, there’s me and Marcello Herreshoff sitting down and trying to break the problem down into pieces, and looking for approaches to the pieces, and then going off and studying existing work and math that’s been done that looks like it might have an application to the pieces. And that can be scaled up as fast as you can find sufficiently brilliant people with a sufficient background who can work on the problem.

On the non-research side of things, there is bringing the issue to the public, bringing the issue to concerned donors, trying to get more funding to put more eyes on the problem. With enough funding we will be able to launch a research grants program that will be able to get people to work on the pieces of the problem that are small enough to be done with in the standard scientific process and even get funded maybe within the standard scientific process if we can just sort of bootstrap the field itself into existence and make it respectable.

In very broad terms, the research program has two phases, which is: figure out what to do, and do it. And right now, we’re in the figure out what to do stage. Right now, for example, we are looking at reflectivity, which is that humans seem to get a tremendous amount of mileage out of thinking about thinking. Modern AI programs don’t seem to get any mileage out of this. And therefore, humans are doing something important and useful, and we do not currently know what it is. We are trying to get that problem set up and knocked down.

I think that once we know more about that, we will know more about related questions like, ‘How are human beings doing mathematical proofs that are so much larger and more complicated than any mathematical proofs that computers can do?’ And questions like, ‘How are human beings writing programs?’ We can’t ask computers to do our programming for us. There are these programs that ought to be able to do it in principle, but they would require as much computing power as a small galaxy to do it. Well, it feels a bit funny to say it, and I think it feels a bit funny to hear it, and because of that it’s often neglected to mention that this is the most important problem in the world, and it’s drastically urgent, and something has to be done.

This entry was posted on Wednesday, September 5th, 2007 at 5:06 am and is filed under Singularity Summit, Eliezer Yudkowsky, Artificial Intelligence. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.