Global Catastrophic Risks: An Overview
Posted by Jeriaska on December 15th, 2008
Global catastrophic risks are risks that seriously threaten human well-being on a global scale. An immensely diverse collection of events could constitute global catastrophes: potential factors range from volcanic eruptions to pandemic infections, nuclear accidents to worldwide tyrannies, out-of-control scientific experiments to climatic changes, and cosmic hazards to economic collapse. Anders Sandberg, James Martin Research Fellow of the Future of Humanity Institute at the University of Oxford, provides an overview of aims of the Global Catastrophic Risks conference series at GCR08, a meeting in Mountain View, California in November of 2008.
The following transcript of Anders Sandberg’s GCR08 presentation “Global Catatrophic Risks: An Overview, and Caution about Risk Assessments” has approved by the speaker. Video and audio are also available.
Global Catastrophic Risks: An Overview
I am having a lot of complex affiliations. Usually when people come to us at the Institute and try to figure out where in the organization we fit in, we have to fill an entire whiteboard with weird ven diagrams to show the overlap—it’s pretty messy. This is of course a good thing, because the philosophy department allows us to study anything we are interested in. Our four main areas—human enhancement, global catastrophic risks, wisdom and rationality, and emerging technologies—covers almost anything.
Now I am going to do a kind of impression here of Nick Bostrom, my director and friend. Unfortunately, he could not come. He sends his greetings, so you will have to make due with another Swede from Oxford. We are hopefully sufficiently alike, except that he has got much less of an accent than I have got.
Without further ado, let’s get on with the interesting and disastrous stuff. Global catastrophic risks are really the big threats against human well-being. One way of trying to sort things is by the intensity of something going wrong and by how large a scope it is. We can start out with the minimum possible loss, such as the loss of one hair. Unless you happen just to have one hair, that is probably nothing really you care about. Then of course you have all the threats against our personal lives, which are pretty important, the more intense they become. Even imperceptible threats on a large scale can actually be problematic. The loss of one beetle species, nobody might notice it, but it is a permanent loss. Given how interesting a lot of beetles are, we actually should feel sad about the loss.
As we move up toward the corner, we get the nastier ones. There are certain threats that we might be able to endure, but they are globally very bad for us. Losing the ozone layer, losing a lot of beetles, that’s pretty bad. We also get this weird case of terminal threats, which we might actually have on a global level but we don’t pay much attention to, such as aging, for example. Quite a lot of people are surprised when they see “aging” up there. I don’t think for this community it needs to be pointed out why actually this should be regarded as a global catastrophic risk. It’s just that it is a risk that is very certain right now, and is something we just might be able to deal with.
Then, up in the red corner is of course the threat from human extinction. Existential threats are something that threaten the well-being of humanity terminally. We could imagine, for example, the 1984 scenario of eternal dictatorship as an existential threat to humanity, but of course the classic one is that we’re all just going to get wiped out. There are lower levels of threat that are still pretty bad, and I think we can do a lot about them. I hope that today we are going to be able to find ways of reducing the orange corner here.
In general, I was trying to get a look at what really big disasters there were. We selected some at the random level of ten million. One of the interesting things about compiling this kind of list is to see what is on there and what is not. For instance, a lot of natural disasters are not on this list.
The biggest flooding disasters have hurt hundreds of thousands, up to even a few million people, but they still cannot reach the level of pandemics and human cruelty. Also, you might notice an overabundance here of stuff happening in China and India. This is because it is at a scale affected by the number of people. Since there are a lot of people in China and India, bad things happening there will kill a lot of people. This of course creates a bias toward relatively recent events, because there are a lot of people around. If you go back and think about the Mount Toba eruption 75,000 years ago, which might have been very close to wiping out our species, at that point it probably did not kill off that many people—on the order of perhaps tens of thousands of people. From the genetic evidence it seems that we were down to a population of about two hundred females, which was very, very close for our species. Still, it would not have been able to get onto the list.
Similarly, certain events of the utmost cruelty like the nasty genocides are still below here. We also get overlaps. World War II, surely we should not lump that together with the nasty genocides? What about the stuff Stalin did—should that be put under World War II? There is a lot of overlap. There is an entire community looking at this and having some fun quarreling about the proper way of measuring atrocities. From our perspective, we just want to get rid of atrocities, but it is an interesting problem to see that even defining it is very problematic. It is also interesting to see that we are having these biases and uncertainties about very direct disasters. There are reasons to mistrust actively what is going on in the data in some cases.
What kind of global catastrophic risks have we come up with? Here, I ran out of space, but we could add a lot more. Some of these are pretty obvious, and then there are others like supervolcanoes which I think teach us an interesting lesson. The term “supervolcano” I think was invented in the ’90s. Before the ’90s, we did not know that supervolcanoes could exist. We suddenly discovered a natural disaster—that if a Wyoming supervolcano would erupt, we would be in very, very deep trouble. That is troubling, that we are still discovering new kinds of disasters. If you go back a hundred years and think about the disasters that people were considering, that list was much shorter—not just because they could not make nuclear weapons, but mostly because we did not know enough about the world.
That should make us aware that the category of unknowns contains probably quite a lot of things that can have a big probability mass and can be fairly nasty. Similarly, we have also weird new things which might or might not happen. That makes them even harder to study. We don’t have any experience with the technology of the future. We are very bad at discussing it, and even worse at predicting what we can and cannot do. At best, we can make some reasonable assumptions based on the known physical laws, but we have a lot of trouble saying something sensible about it.
Usually, after we develop the technology further we realize we were pretty silly in our concerns or missed out the really bad side effects. That means that some of the new technologies are going to pose very deep challenges. We just know that we are going to be getting a lot of new technologies, some are going to be very powerful. Since everyone is looking at superintelligence, that might mean that we are going to miss some of the others, something similar that seems much more innocuous. No one really has considered threats from quantum computers, because they don’t seem to be dangerous. How do we select rationally what seems to be dangerous or not? Looking at history, we have been pretty bad at that. Again, this issue is something we should be concerned about.
Exogenous events have a very nice property: they are not that dependent on what we humans are doing. This means that the probability stays roughly constant over time. We can make it much worse, of course. Flooding becomes worse if you chop down the woods, and we can make ourselves more vulnerable by creating technological infrastructures that are vulnerable. However, a lot of big threats are human-made. We also have systemic threats, where we set up institutions that even make attempts at fixing threats, but in themselves can cause problems. Problems of financial markets are probably intrinsic from doing finance, and similarly we might have problems managing risks that are intrinsic from what we do.
I think most events tend to be accidental, but deliberate threats are much nastier because they can be fine tuned to get around our defenses. One interesting thing that came up in one of the discussions at Oxford last summer was there are a lot of common pathways. Quite a lot of disasters end in the same way. For example, with really big ones we might risk social collapse. That suggests that fixing social collapse might be a very valuable thing, because there are a lot of different kinds of disasters. Very many disasters depend on a certain technology level, but international tensions might lead both in war to the use of nanotechnology or nuclear detonations, depending on what nations are in these situations, but we can be very clear that we can deal with a common cause here.
Then there is correlation. Some disasters tend to be followed by others. Quite a number of wars have led to pandemics, because it leads to a lot of stress, a lot of people are living in bad conditions, and after war a lot of soldiers go home with new germs they acquired somewhere else. Then there is predictability. I think Rumsfeld’s classic distinction of known knowns, known unknowns, and unknown unknowns is a very good start, but there is also the nasty fourth category of the unknown knowns. That is stuff that you actually know but choose to ignore. Unfortunately, I think there is a lot in that category, and we are very good at ignoring stuff in that category.
The good news is that we seem to have reduced a lot of the small and medium risks recently. We still have a lot of problems, so we should not forget that we have some ongoing horrors in the world that also need some fixing. The biggest risks seem to be anthropogenic, which has a good side. We can actually do something about it, since we are deeply involved in it. The downside is that is going to mean getting a lot of people to cooperate, which is usually quite hard. The really big and weird ones are decades away.
We have some time, but that also means we don’t know much about them, too. We know that in the past when people have been trying to predict the really serious threats, people have done really bad. For example, discussions about the future of humanity from the turn of the last century, people were very concerned about dysgenics—the good people aren’t having enough children and the bad people are having too many children, so what is going to happen to the race? Fortunately, we know that that seems to be a very minor problem, even if you are a very pessimistic geneticist you see that the negative selection pressures are so weak that we have much bigger problems to be concerned about. There were actually a lot of things that people did in order to fix the dysgenics that turned out to be very, very bad indeed.
We know we have been bad in the past at doing predictions. I am going to return to this point again and again because this is one of the main things I have ever learned at Oxford. Very smart people make very stupid mistakes, and they do it with surprising regularity. The trick should be figuring out how to avoid doing them regularly. While I am just throwing out interesting challenges here dealing with going risk assessment the right way, estimating mistakes is a wonderful philosophical problem. This is a great thing to invite philosophers out to the pub to figure out. What is it worth to protect six billion lives? How much are the future generations worth? Here, some people think you should have some kind of discounting. Then, someone else gets very upset—there should be no discounting whatsoever, because all of the future is dependent on our actions.
This can be extremely cumbersome, because if you believe that we might actually have a chance at a glorious posthuman future, spreading throughout the galaxy and colonizing hundreds of billions of years, or existing as even more dense forms, it means that the future would be so enormously vast that the total value of that future would totally overshadow anything we could do today. We should perhaps sacrifice everything, from that point of view, for making sure that that big posthuman future happens. At that point a lot of people become very skeptical, because it is just a possibility. We don’t have all that much evidence for it. Even if we just agree on 6.5 billion lives, that is worth quite a lot. Even that could lead to some pretty surprisingly big actions we might need to take.
Then there is the whole mess of probability. Once upon a time I felt that probability as a study was pretty easy, but dealing with actual probabilities in real life is a completely different matter. The most pessimistic guy at our institute is from insurance. He has been handling real probabilities for many years and argues that philosophers actually get it all wrong. He has a lot of horrible examples showing that yes, indeed one in a million chances happen far too often. One reason is that we are simply making the wrong kinds of estimates. Another is that one-in-a-million chance costs so much that it actually does not matter that it occurs very rarely, because we still get completely swamped by those really rare events.
Thinking about probabilities, what is the probability of mankind being wiped out in the next hundred years? From the objective probability perspective, that is not even a real probability because it is not a frequency. Here, the subjective probability Bayesian people get into fisticuffs with the objective probability people. Then we have the problem that the threats are going to change. Right now, the threat from nano-assemblers I am rather happy to say is probably absolutely zero, or at least indistinguishable from it. That is probably going to change over the century.
Similarly, I think the problem of giant steam explosions from steam engines is going down quite a lot. It was never high, but it is being reduced. The chance of nuclear war in this part of the century might actually be relatively low, but it if happens it is going to preclude a lot of other technological advancements at the end of the century. If we are trying to figure out the probability of mankind being wiped out by nanomachines, if you think that war is relatively likely, you must reduce that other risk. At this point it becomes very tricky to fill out forms for probability estimates.
After our conference, we spent several evenings just debating what the meaning of these probabilities that people were giving was. I think the most sensible thing is to say it is not real probability in that sense, it’s concern. In reality, risk is not just the harm times the probability, but we also add in how much concern we have. A hellish threat might actually be worth taking more seriously than one where people died, but we were rather happy about it. There are not very many threats in that category, but they might exist. Then of course we have the interesting problem of anthropic biases. Our existence is already biasing the probabilities. We have not had a giant nuclear war over the last fifty years, because we wouldn’t be here.
Our continued existence actually affects probabilities. We cannot just think about probabilities without taking into account and correcting for existence. If we look at the fossil records, we will not find any meteors that wiped out all life on earth. This is simply because no observers on such a planet would exist. This actually produces some really weird things. I’m working on a paper with Nick Bostrom about this, where we actually find that you can correct for this, given some assumptions. It is going to produce a bit of weird problems.
Then there is of course really low probability issues, like a particle accelerator accidentally creating a black hole that wipes us all out. A paper demonstrating that the risk is zero is wrong. It is so much higher than an acceptable risk. If we are talking about fire insurance, it does not matter all that much because it is not likely to lead to a really bad event. If somebody says that it is a one-in-a-million chance, and whatever that was happens, it is much more likely that that occurred not because we were really, really unlucky, but simply because the estimation was wrong.
This leads to interesting problems. There were a lot of people who were very concerned about the large hydron collider destroying the world, and like all good physicists they just laughed it off. Of course they knew that intuitively it’s unlikely. That is never going to convince people. When the Brookhaven report came out about the risk of strangelets, Nature wrote that we always knew the probabilities were really low but it’s nice that somebody bothered to calculate them. When you think about it, it’s actually a very weird statement.
More recently, a marvelous paper demonstrating that there is absolutely no risk of black holes, but it is a fifty-page, complicated paper involving a lot of general relativity and other things. What is the likelihood of that paper being wrong? Looking at the scientific literature, we find that a lot of papers are wrong. A very conservative estimate would be that at least one paper in a thousand. This means that even if we accept that it is a very good paper, we still have one chance in a thousand that it is wrong. Another way around this might be to make several independent arguments. That leads to one chance in a billion that it is wrong. The crackpots who are really concerned are not going to read the papers anyway and are not going to get the probability argument. There are ways of handling very low probability. As Eliezer pointed out in his chapter in the risk book, if we could not handle really low probabilities, all the lotteries in the world would go bust.
Then there is the risk of cognitive biases. We are not very good at thinking rationally. Some of these biases are very big, some of them are very common, some of them are very confusing. If I show you a lot of examples of what could go wrong, you are very likely to increase the likelihood that you assign to them happening. That is actually one of the risks of attending meetings like this. You are probably going to go back home thinking that the risks are slightly higher than before. Rarely you might get a good argument that the risks were actually overblown.
Similarly, we know that we have availability biases. Conversely, if something has never happened people tend to assign far too low a probability to it. Then there are the social biases. A lot of the things we are interested in here tend to be regarded by people as silly. I have been thinking for a long time that I ought to write a long paper just analyzing sillyness. It has a lot of interesting categories. Some things are silly because they are unlikely; some things are silly for social reasons. The people in one area are regarded as crackpots for social reasons, which can of course bias a lot of research in very bad ways. On the other hand we have millennialism. On the one hand, people do not want to appear as millennialists, so they will underplay a lot of risks. On the other hand, it is very fun being a millennealist. A lot of people get a lot out of that, so they might be overly concerned about risks, and perhaps about the wrong risks.
What would be the rational way of spreading our effort? We might go for the biggest threats, but on the other hand earlier threats may be more important. There was an editorial on the book in Nature, where the reviewer took Nick to task for not dealing enough with climate change. Well, it was discussed in the book, but it was one chapter among others. The reviewer felt that since climate change is such an urgent thing, we really should focus on it. Maybe given the financial crisis, we should not concern ourselves with climate change and focus on the financial crisis… that kind of thinking does not really work. Obviously, we need to distribute our effort in a rational way. Given that we have a hard time even agreeing on probabilities, we need to figure out ways of dividing the labor.
Similarly, there are a lot of threats we don’t know very much about. Here, a little bit of research effort can get a lot of information. For instance, the threat from asteroids seems to have receded quite a lot thanks to the Spacewatch study because we actually found that there are rather few asteroids that pose a threat. A lot of uncertainty disappeared and a lot of potential threats evaporated. We still should, of course, watch the skies, but we now know that we don’t need to watch for certain things.
Then there is of course that some threats are more political than others. This gets very much into the debate of climate change and nuclear disarmament. Meanwhile, nanotechnology is not a political question yet for many years, so it might be very hard to deal with that threat. Again, we might think again about where to put our efforts. We should think about the best levers and try to find a way of wiping out emerging threats before they hit us. Thank you.
