Future Current

Peter Norvig is the Director of Research at Google Inc, where he has been since 2001. From 2002-2005 he was Director of Search Quality, the manager of record responsible for answering more queries than anyone else in the history of the world. He is a Fellow of the Association for the Advancement of Artificial Intelligence and the Association for Computing Machinery and co-author of Artificial Intelligence: A Modern Approach, the leading textbook in the field. In his keynote speech at the 2007 Singularity Summit, he argued that the invention of new technology is limited only by the laws of science and by the degree of ingenuity in the lab. But the proliferation of new technology into everyday life is a complex social process involving entrepreneurs, venture capitalists, international corporations, politicians, consumers, and dumb luck.

The following transcript of Peter Norvig’s 2007 Singularity Summit presentation “The History and Future of Technological Change” has not been approved by the author. An audio recording of the talk is available at the Singularity Institute website.

The History and Future of Technological Change

Good morning, everybody. It’s great to be here. Thanks for coming. I’m going to talk a little bit about how to evaluate technological change. We have heard so many predictions. How do we know what to believe and what not to believe?

Yesterday we heard from these two great 20th century philosophers Berra and Bohr. “It’s hard to predict, especially the future.” I’ve got to say, coming here, it’s hard to fathom all the predictions that you get. I’m used to much smaller, simpler predictions. I’m used to going to conferences where, say, Chris Manning has a paper that says “my parcer got 92% correct compared to Dekang Lin’s, which only got 91% correct. That’s the closest we get to talking trash in the parsing business. We’re arguing over 1% there. At this conference, we are hearing things about 100% differences or more.

Here is Steve Kirsch on the left, who has made the prediction that within 90 years humans will be extinct because of climate change. On the right, Aubrey de Grey says he’s going to live to be a thousand years old. At least one of them is 100% wrong, maybe both. But how are we going to tell?

Last year, Doug Hofstadter said that artificial general intelligence is at least 100 years off, and Ben says it’s less than ten years. Maybe that’s not quite 100%, but there’s a big disagreement there.

We’ve all seen the Kurzweil graphs saying that progress is increasing exponentially, so on a logarithmic scale it should be linear, and boy that looks pretty linear. But other people look at the same data in different ways. Jonathan Huebner has this paper called “A Possible Declining Trend for Worldwide Innovation.” He looks at pretty much the same data, important events in the history of technology. He has one little trick, in that he looks at events per year per billion people, rather than the total number of events. But by his count, it peaked around 1900. We were inventing relativity, the automobile, stuff like that, and it’s all been downhill since then according to his analysis. Amazingly, by 2100 there will be zero innovation.

Again, one of these guys has got to be 100% off. Some predictions that are being made are already false. I guess it depends on how you evaluate these predictions, but for example, Wendell Wallach said yesterday that within ten years computer error would lead to a catastrophe with many deaths. He said, maybe it’s not quite as bad as 9-11, but it’s going to be bad. Actually, if you look at it, today in the U.S. there are between 100 and 200 deaths everyday from medical error, and many of these medical errors have to do with computers. These are errors like giving the wrong drug, computing the wrong dosage, 100 to 200 deaths per day. I’m not sure exactly how many of those you want to attribute to computer error, but it’s some proportion of them. It’s safe to say that every two or three months we have the equivalent of a 9-11 in numbers of deaths due to computer error and medical processes.

That’s already happening. It just does not seem quite as catastrophic because it is spread out over time, rather than occurring at one place, in one instant. Arthur C. Clarke got a lot of credit yesterday, I think deservedly so, for his predictions in 2001 and so on. But here is a novel from 1986, just twenty years ago, and it’s about 1500 years in the future, when scientists discover that the sun is about to go nova, and so they have to save all of humankind’s knowledge, send some starships off, and populate the stars. They go about collecting all the knowledge and scanning all these books, or maybe they are digitized. It doesn’t quite say. And they are actually throwing out novels that they say are not good enough. Maybe you keep all the Jane Austen but some of these other authors, we’ll only keep one of their novels, not all of them. This is supposed to be 1500 years in the future.

Meanwhile, today, at Google they ended up with a few billion pages when they were done with this sorting process. Today at Google we have tens of billions of pages, we keep every novel, and we keep tens of billions of pages of spam. We don’t care. According to Clarke, you could have one user searching that database and they would get back an answer in a few seconds. We have tens of thousands of simultaneous users and you get back an answer in tenths of seconds. So, here we are, a prediction about 1500 years in the future, but after twenty years we have exceeded the prediction by one or two orders of magnitude in pages, one order of magnitude in the speed, and about four orders of magnitude in the number of simultaneous users. I guess we can take the next 1480 years off, because we are so far ahead.

It’s always fun to look back at what happened 100 years ago and compare that to predictions we are making for 100 years in the future. Ladies Home Journal from December 1900, some things they did really well. They said cars and houses will be artificially cooled. I don’t know if it is artificial general cooling or just specific cooling, but they got that one right. They said there would be aerial warships, forts on wheels (which I took to be tanks), and giant guns that could destroy entire cities. Photographs will be telegraphed from any distance, so there will be faxes, or maybe internet. Photographs will actually be in color. They got that right. Then there were some bizarre ones. There will be no C, X, or Q in our everyday alphabet. They thought that the future couldn’t tolerate this kind of redundancy, and we’d use the other letters instead. Why that was one of the most important predictions, I don’t know. And then there was a bunch they got wrong. There would be no mosquitoes nor flies, no wild animals except in zoos, and they thought this was kind of a good thing. That was another bizarre outlook. Rats and mice will be gone. The horse will become practically extinct because we will have these motorized vehicles, and there will be fast electric ships going from New York to Liverpool in just two days.

So this is simultaneously over and underestimating. Of course, it takes us about a quarter of a day to get from New York to Liverpool by plane, but our best container ships take about four days. They knew there would be air machines, but they were only going to be used for blowing up cities. Overall, they got a pretty good mix. There were twelve predictions where they did not go as far as we actually have achieved, seven that they got about right, eight where they were too conservative, and three, like the C, X, and Q, that were just totally bizarre. I tried to look at how we can evaluate what predictions to trust and not trust. There is not very much literature on it. One book that I thought was pretty interesting is this by Philip Tetlock. It’s called Expert Political Judgment, and he actually looked historically at these people who appear on Sunday morning TV shows and make predictions about politics, and analyze when they got it right and when they got it wrong.

This hedgehog and fox analogy that I never understood, maybe I don’t know enough about hedgehogs and foxes, he says that people that were driven by a grand, unifying ideology mostly got things wrong. Whereas people who were able to look at lots of different data from different data points and compare one to another, and say, “On the one hand this, on the other hand that,” they had a much better chance of being right. He says, “Subject matter expertise translates less into accuracy than it does into overconfidence.” So it does not really matter all that much how much you know about the subject area. He also said that the more famous you are, the worse you do. Actually, he quoted Google in there and says there is a .26 correlation between overconfidence and number of Google hits on your name. If you see these talking heads on Fox News, a quarter of their incompetence can be explained by their famousness. I guess the other three quarters is innate.

I think one of the great morals of this is that your own opinion can be as good as the experts. You don’t need the PhD with twenty years of experience in the field in order to get these predictions right. Even having that doesn’t help very much. What you need is to be able to read the papers day to day, catch up, understand what’s going on, and be able to evaluate two sides of an argumen. If you can do that, you are as qualified as the experts. That’s good news for all of us. You don’t have to listen to the people on stage. In fact, there is a .26 correlation why you shouldn’t listen to the people on stage. Instead, you should evaluate all this stuff yourself when you get home based on all the evidence you accumulate.

That’s the moral about judgment. It’s kind of depressing. Now we have to go back and look at the data. The Kurzweil-type stuff seems to be inconclusive. Kurzweil looks at it one way, Huebner looks at it another way and sees different things. Maybe that’s not too surprising, because it relies on these lists of what is important, and that is all very subjective. For all I know, the dinosaurs 65 million years ago had a Kurzweil-style chart that says all the important innovations happened in the last few years: there was this new type of plant that evolved that’s really delicious. That was really the most important thing that has happened for millions of years. All these new things are happening really quickly. So, from any one point-of-view in time, it may look like important things are happening no, but you really want to see if they have an impact on the world in a more objective sense.

One of the things you can look at is gross domestic product. This is the total GDP for the U.S., inflation adjusted. Where I got this slide from, they were trying to point out these depression and recession points. The Great Depression in the ’30s, you can actually see a real drop there. This littler recession due to the oil crisis in the ’80s, you can see a little blip there, but in a couple years it’s gone. If you draw a straight line, it’s all pretty straight there. So, a definite dip in the Depression, a little blip in World War II where production was artificially pumped up by borrowing from the future, but again, a couple years after the end of World War II that blip is all gone. What this says is that there is constant exponential progress, but it also says that it does not seem that there is any great accelerating due to technology. There is no take-off in the ’60s after the space technologies, or in the ’80s after personal computers come in. It all seems pretty much flat.

Another way you can look at it is by looking at percentage annual growth. Here I looked at the world rather than the U.S., and I am looking at per capita rather than total, but it looks fairly constant. As you go across, it looks like most of the annual growth is between 1% and 2%. Certainly almost all of it is between 0 and 3% a year. It does not seem like there is any solid trend up or down. In fact, over this period from 1970 to the present the annual growth rate was 1.6%. World GDP is about $6000 today, so if this 1.6% continued, in the year 2100 that would give us $26,000 median GDP. If it dumped up to 3%, that would give us $92,000 median GDP for the whole world. Either way, that’s a pretty good advance that we are looking at if things hold along this course. It means most of the people in the world would be brought out of poverty up to a level that looks pretty good today. But again, we don’t see any real accelerating trend, we just see this constant 1.5% plus or minus one percent per year.

Another objective stat you might want to look at is life expectancy. Here, this is a linear model, not a log model. Now the progress is not as accelerated, not exponential, just linear progress. Going from 1950 up to 2005, going up in every continent except for Africa, which starting in the late ’80s had this hit, mostly due to AIDS. The other continents you see going up and squishing closer together. South America and the other continents are catching up to North America in life expectancy. But again, no market acceleration due to increased use of computers or any other kind of technology. It all looks pretty flat. It seems like no matter what you do, it’s not going to make a difference.

Now I want to talk a bit about artificial general intelligence. In order to do that, I’m going to make some analogies. First I want to talk about AGS, which is artificial general space exploration, a term I just coined. This is the analog mission at the Haughton crater in the Arctic in Canada. This is a mission that I was involved with in my former life at NASA Ames, where we sent up teams to simulate what it would be like being an astronaut on Mars. The conditions in this crater in the Arctic are in several ways a good analog to being on Mars. We wanted to understand what it was like for a team of astronauts to be there, to try to live, do exploration and go out, and do science everyday.

This was really AGS. We were trying to say, let’s attack the whole problem of space exploration. In particular, Mars exploration: see if we can get it done, understand the problems, and see how far that can go. That was an interesting project and it’s ongoing. It seems to me that if we get to Mars, that is not going to be the bottleneck. Things that come out of that are going to be very important in terms of getting the maximum return per hour for an astronaut on the surface and having them all get along and not kill each other. The step that is going to make the difference between going to Mars or not is not going to be this AGS, this generalized space exploration. It is going to be all the components.

Here is one example. This is my pal and former colleague Les Johnson from Marshall Space Flight Center. He’s holding a carbon fiber material that could be used to build space sails. Probably space sails are not the particular technology that is going to be the key breakthrough for going to Mars, but there is going to be a lot of them. They are going to have to do with propulsion, radiation shielding, control, launch, transport into low earth orbit for cargo economically. It is going to be these components that will enable us to do it. It’s not that we don’t know how to put it all together, it’s that we don’t have the components.

Here is another field I just made up: artificial generalized material science. Material science is a fascinating field this way. You’ve got these carbon nanotubes over on the left. This is the first biological film. A team of students at the University of Texas grew this. At the bottom we have this incredibly strong structure build out of atoms at the Keasling Lab at Berkeley. All this great progress in materials, but nobody in that field is saying what we really need is one thing that makes all these possible materials. That is not where the field is going. Instead, they are saying, there’s these exciting breakthroughs, let’s work on them all individually and see where we all end up.

By the way, I think this whole field of synthetic biology is absolutely fascinating. I had a chance a while ago to talk with Craig Mello, who won the Nobel Prize for RNA interference. Talking with him and picking his brain about how exciting all this stuff was, how I was interested in it and saw this amazing future for synthetic biology and so on, he told me that what he was really excited about was the future of AI. I thought that was a little ironic, but I took it as kind of a bad sign that each of us who was an expert in one area was saying this other area was the one that is really exciting. We’ll see how it all plays out.

Another thing I want to talk about is AGC, which is artificial general culture. I think that is something that is really important. If you look at these two guys, there is really not that much difference between them, I think. Sure, if you took them into the lab you could say this chimpanzee of the left seems to be stronger, and this Homo sapiens on the right seems to be a little bit smarter. If we had the third kind of chimp, the bobobo, up there we could say they seem to be having more sex. There are these individual differences but mostly I think they are more similar than they are different, in terms of their overall abilities. If you plunk one down in the middle of the savanna and look at what they are able to do as an individual.

It is only when you compare the cultures that you see this fantastic difference. There you can really say the chimps are limited. They have these great family structures, but it does not really go beyond that. Whereas, look at what the humans have been able to do as a culture. I think that is the exciting promise of what humans are. It is not their individual intelligence, it is their collective cultural intelligence. If anything, we should be looking at that and trying to build this culture rather than trying to build an individual.

We are back to AGI. Is there anything different about this point in time right now, compared to points of time in the past? I tried to think about how to evaluate that. I was inspired by Marti Hearst, who had done some work looking at particular keywords to do extraction from natural language text. I thought a good keyword to look for is the phrase “unlike previous.” So I did a Google scholar search for “artificial intelligence” and “unlike previous,” and limited it to different age ranges so I could say what people are saying that is different about right now at their point in time compared to previous systems in the past. This is what I came up with.

In 1968 somebody said, “I have this system that, unlike previous systems, learns from examples.” Notice in 1998, we had the exact same thing. We’re also saying in 1988, say, being able to represent different states simultaneously, and so on. I looked at a couple hundred of these, and I can’t tell the difference. It seems like at all these points in time, people are saying “unlike previous systems, I’m building a new one, and it’s got something slightly different.” But there is no feeling that right now there is anything that is more privileged about us being on the verge of discovering something, as compared to ‘97 or ‘87 or even ‘77.

What do we need to make progress? I think it’s really two things. We need more data and we need more models of what that data does. Both of them are important, and I think it was interesting that in SIGGRAPH this year, there were two papers that came out that got a lot of press, Slashdotted and so on, and they took two extremes on this pole. One of them here is this work by Avidan and Shamir on image resizing. They show how you can take an image and squish away some of the pixels or add in some pixels and get back another image that looks really good. They are able to do this without any model at all of what the pictures are actually about. All they are doing is saying, “Take the pixels that are most like the pixels next to them, and they’re not adding much information, so just throw those away.” And then there’s a little bit of clever, dynamic programming. No knowledge about the world at all, but you can see here, you still get pretty good results. You can squish that picture in, or you can extend it out. It looks great. It’s filling in the right stuff without knowing anything at all.

Compare this to another paper that is pretty exciting, by Hayes and Efros. They say, say you took the picture on the upper left. Half that picture is goo, but it’s got those rooftops in there and you don’t really like them. So you go into Photoshop and you etch out what you see on the upper right, and now you say, I want to fill it in with something. If you are an experienced Photoshop user, you could actually clone out some of the water and put it in there, but it would take you a while to do that. What they said is, instead, let’s just go against this database of photos, find some that look similar, and fill it in with stuff from that. In the lower left you have this big collection of photos. In the lower right it shows one. You put it in and it looks just right.

The interesting thing about this was that there was a threshold. They started off with a database of thousands of images, and it didn’t do anything. It just put in junk there. But once they got past millions of images, then all of a sudden it started to find matches. If you have a few million images, you are going to find just the right one and match it in there. So here was this threshold of data where once you pass it, you get the right answer.

At Google we had a similar experience with our machine translation work. We are able to go from Chinese to English and do pretty well. There’s a few disfluencies there, but basically you are getting the right idea. In this case, there is both a model and data, but the model is a fairly simple one. We don’t have deep understanding of the world. All we have is a model of what words follow each other in English and in Chinese, and what words mapped to each other in pairs of text. We take a news story in English, a news story in Chinese, and say this story is a translation of this story, then build a probabilistic model that says, “Probably the first sentence is a translation of the first sentence. It might be one to two, or two to one. Spread the probability over all of that.” Then, within the sentence, you do the same kind of thing. ” This word might translate to this word.” If you only have a few million words of examples, you get nowhere. But once you start getting up into the billions, you start doing pretty well.

Here you see linear performance, not yet asymptoting out, up to 18 billion words of paired data between Arabic and English. In the background there is also an English model of a trillion words. That seems to help a lot.

So, we are changing the way we look at where the trade-offs are between data and models. If you can solve a lot of the problems with the data, go ahead and do that. Only introduce the models when you really need it. This changes the whole way that we think about how we do what you think of as everyday programming, as well. You look at doing unit tests, what is the API for jUnit. You can assert things are true, false, or equal. When we go to test Googl, this stuff does not really help just by itself. It’s not enough. Sure you can break it down into “we want this part to be part of our code, we want this assertion to be true.” But really, the important things we’re saying are not logical truths, they are probabilistic ones. They’re saying “we want over 90% of the time to get a good answer.” And we want that to be against the current state of the web, which changes everyday. It’s probabilistic and it’s online. It changes this whole idea of how you do testing.

It used to be you build the code once, and then you freeze it, and then you test really hard and make sure it is correct. You can’t do that. We are going to continue this process of updating, so we are testing this code that is essentially alive and continuously morphing, and we are testing it probabilistically. We are still trying to get some assurance that it does the right thing.

That changes the way we work. Let’s get back to AGI. I said that I thought that the components were more important than trying to solve the whole problem at once. Once you have the components, then you can solve the whole problem. What do I think are the main components? Well, I’m an expert so don’t trust me, but these are the things that I think we need progress on. There is progress currently being made, so I am encouraged, but we still have a ways to go. I would like to have probabilistic first-order logic. A lot of when you interact with the real world, as opposed to an artificial world, you are dealing with uncertainty so you have to deal with probability. Traditionally, most of the probabilistic models have been propositional rather than first-order. That means you can talk about the probability of “it’s raining,” but you can’t express complex things like “for all positions, if it’s raining now, then tomorrow it won’t be raining.” The ability to quantify over multiple states, rather than talk about individual states. That’s going from propositional to first-order. It has been difficult to combine those, but groups at Stanfor, Berkeley and other places are making good progress.

We need hierarchical representation in problem solving. We have a lot of ability to do planning and problem solving, we have ability to come up with different representations, but the real question is making this hierarchical. To be able to solve the vision problem, we need to build up in a hierarchical fashion going from pixels to lines, faces, and people in a crowd. That’s difficult to do. Then you need to do learning over all of these. You want to be able to learn from the data rather than having to program it by hand, or we’re never going to get anywhere. You want to be able to do that with lots of data in an onlie fashion, continuously updating, and do it efficiently. If you can do all that, then once we have those components, then I think the time is right to start thinking about AGI.

This entry was posted on Wednesday, November 21st, 2007 at 2:51 pm and is filed under Space, Singularity Summit, Artificial Intelligence. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.