Preventing Technological Armageddon
Posted by Jeriaska on March 27th, 2008
Brian Wang and Michael Anissimov at the 2007 Foresight Vision Weekend
Michael Anissimov co-founded the Immortality Institute in 2002, a life extension advocacy organization that today includes hundreds of members and an online community. In 2003, he founded the SF Bay Area Transhumanists. He has also written numerous articles for the Q&A site WiseGEEK, offering “clear answers for common questions” on technology. Michael is Fundraising Director, North America for the Lifeboat Foundation, and serves on the Global Task Force for the Center for Responsible Nanotechnology. At the 2007 Foresight Vision Weekend, he gave a presentation on the measures currently being taken by the Lifeboat Foundation to address existential risks, or possible human extinction event scenarios.
The following transcript of the Foresight Vision Weekend presentation by Michael Anissimov titled “Technological Armageddon” has not been approved by the speaker. Video is also available.
Preventing Technological Armageddon
I’m with the Lifeboat Foundation. Our website is lifeboat.com. We are the only organization that currently exists that is taking a look at all risks to human extinction in the next century. I’m just curious, who among you thinks that it’s plausible that humanity could extinguish itself in the next century? How about over 25% chance? Over 50% chance? Okay, well we agree that there is some probability, which is what the Lifeboat Foundation is about.
If you go back to Engines of Creation by Eric Drexler, he is talking about grey goo in that book. This is one of the things that helped inspire the foundation of this organization. The Lifeboat Foundation has delineated several categories of risks. In increasing implausibility they go robotics/AI, nano risks, and bio risks. We find it easiest to get people to think about the bio risks, because if you look at Craig Venter, he is talking about creating synthetic microbes. It’s really in people’s minds right now. Synthetic life is coming. It looks like in the next five to ten years we are going to have real synthetic life. Now, say it becomes easy to make synthetic life, and people are printing out genomes on home printer systems. You start to have a larger possible space of microbes, some of which could be tremendously useful, but even if you are talking about a single microbe that is potentially dangerous, it can cause a disaster. Just like how gene therapy was set way back because someone reacted negatively and died due to a gene therapy event.
We dealt with bio this century. For instance, the Spanish flu killed off a hundred million people. That was natural. Now we’re going to be talking about people with the capacity to study bio genomes in detail and synthesize them on their own. How about a bacteria that is immune to bacteriophages? Because when a bacteriophage virus tries to insert its own DNA, instead of pumping out more viral proteins it just fails, or pumps out something that’s not the virus. Then you have a bacterium that is immune to bacteriophages. One of the things that keeps bacteria in check on this planet and makes it safe for people like us humans and other large organisms is that these viruses are killing these bacteria at tremendous rates. If you have a bacteria that is immune to these bacteriophages, then that could be a risk right there.
Nanotechnological risks. As you go up this risk, it becomes a little bit harder to get taken seriously, which is why sometimes we focus down here on asteroid impacts. That is something that is statistically measurable, because there is an impact record in the form of craters on the planet, and it looks like there is a species-killing asteroid that hits every ten to hundred million years. You can then test statistically for what is the probability that I will die from an asteroid impact. It turns out it’s pretty damn small, even if you consider that everyone would be killed, it’s pretty damn small. The thing that is valuable about the asteroid risk is that we can point to it and say this is one thing that could threaten humanity as a whole. So people loose their incredulity.
When you go into bio, nano, robotics/AI, it’s a bit more of a challenge because scientists have an interest in protecting their projects, they have an interest in having a good reputation in their field of research, that’s why you saw a strong backlash to the grey goo idea. If you’re unfamiliar with the idea, it has to do with out-of-control self-replicating nanobots consuming all biomass on the planet. It turned out that that’s not too realistic because instead of creating free-floating, self-replicating nanobots, as Drexler first envisioned in Engines of Creation, we could do more of a nanofactory thing where everything is neatly stapled down. There needn’t be nanobots running around. We could deploy nanotechnology in a safe way, but does that preclude the possibility that someone will deliberately engineer it?
Audience: This is about asteroid impacts. I just read somewhere that the latest evidence is that the dinosaurs were actually killed by volcanoes.
I saw that press release, but I think consensus is still pretty strongly on the Yucatán impact crater, because of the irridium that they found.
Audience: I’m wondering if volcanoes might be an existential risk.
It’s true that Yellowstone is a supervolcano, and there is a five-mile thick barrier of rock and a gigantic magma upflow from the mantle in that region. That’s why I was thinking, maybe if you threw enough hydrogen bombs down in there, maybe you could uncork that baby, and cause a supervolcano eruption. We don’t know. But that would be an example of an existential risk.
In many cases, the communities that engage in these fields of research like to self-regulate. The reason they self-regulate is because they are afraid of the government coming down hard on them. When recombinant DNA technology started to unfold in the ’80s, people got together and started talking about the ethics involved. They don’t have an interest in going too far, because they don’t want to restrict their own research, but they do have an interest in making it look like they are self-policing, which is valuable.
A nano risk would be something that consumes biomass and can survive out of water. Something about biology is that it usually requires water to keep going. But if you were able to engineer something that is similar to a microbe, like a microbot, that can self-replicate using things found in nature, it would be challenging to create such a thing but there are incentives. For instance, there are military incentives and also productivity incentives. Instead of rolling out solar panels in the desert by doing it yourself, it would be a lot easier to create self-replicating robots. There is an economic incentive to build things that aren’t necessarily existential risks, but on the edge of it. If someone had a negative enough inclination, they could take that one step further and create something more dangerous.
Robotics/AI. That’s another. Steve talked in detail about how an AI would have as a convergent subgoal to acquire more resources and power in order to achieve its goals. Say we take an artificial intelligence that is built in order to pick stocks, but it’s the first artificial intelligence that crosses the threshold between human and superhuman, then it might go beyond the bounds that we originally set for it. For instance, it could try to put its fingers into world politics. It could try to manipulate select individuals in order to make its stock picks go well. From that you could get into a situation where AI’s are able to manufacture robotics.
One thing I have to emphasize is that I don’t think that today there are any risks that could wipe out all of humanity, which is why it is challenging to talk about it. Does anyone think that today there is a risk, besides asteroids and supervolcanoes, a techno-originating risk that could wipe out every human being? Humans are widely distributed. Half of humans live in cities, and half don’t. There have been studies about the effects of nuclear weapons. It would be devastating if there were a full-scale thermonuclear war.
That’s another thing that you can talk about as an existential risk because we’re familiar with it, but the key is that nuclear war and asteroid impacts are not the whole thing. Synthetic biology is a big one, because it starts to explore new areas of design space that natural biology is not capable of exploring. It gets dangerous when you start talking about self-replication. What I consider to be foundational existential risks are effects that are either self-replicating or self-amplifying. In the first category you would have self-replicating microbots. In the second category you would have artificial intelligences that are capable of adding to their own computing power or robotic capabilities.
We’re not there yet. I think we’re talking about 2020 to 2030. That’s the key timeframe that we should be looking at, and maybe even a little bit before. I’m talking about microbots that can self-replicate using any biomaterial. I think it’s worthwhile to start preparing in advance, because even though lots of these risks we are talking about are from 2020 to 2030, many people such as those from conferences like this that are forward-looking think about these issues. It’s really valuable when people like Steve and Eliezer, others at the Singularity Institute, are thinking about what moral systems we should instill in artificial intelligences so that they don’t threaten our existence. It’s a serious question.
Each one of these is clouded by its own issues. Robotics/AI is clouded by the fact that most expositions of AI scenarios are fictional, like Terminator etc. People consider that to be part of entertainment, entertainment is funny, so the idea of an AI wiping out everyone is to most people humorous. It’s in that category of things that are fictional.
For those that are not familiar with it, the function of the Lifeboat Foundation is to bring together scientists, visionaries, futurists, anyone who has thought about these issues. It’s really good that we have people like Stephen Hawking and Warren Buffett, who are truly concerned about these widescale risks. It’s easy to find quotes where people say that maybe in the 21st century we’ll make the last mistake of our existence. Since the Cold War, we’ve been talking about the possibility of self-annihilation. Whether or not nuclear war would be a route to that, we are familiar with it on some level. I think that the moral stakes are so huge that it’s worth specifically looking not only at technologies that can cause disaster, like even on the hundred millions scale, like nuclear war, but technologies so powerful that they could wipe out every single human being.
It’s really overwhelming for a lot of people to look at. People keep trying to change the subject a lot of the time. What the Lifeboat Foundation does is bring together scientists. We have a huge, maybe unnecessarily large scientific advisory board of about 400 members. We are getting these people talking to each other on our mailing list. We’re going to have a Second Life conference. One of the key difficulties here is that there are so many disciplines involved in each of these specific things that no one mortal human being can study them all. When I’m assessing the risk of synthetic biology, I’m not a synthetic biologist, so I have to talk to specialists, and sometimes the fields don’t even exist yet. Who is an expert in molecular manufacturing? No one. Because we haven’t achieved molecular manufacturing yet. Anyone who thinks they are an expert in the field is just the best we have now, but not really what we’re going to need.
The Lifeboat Foundation, we like to focus on areas that are most neglected. Areas like artificial intelligence safety already are being addressed by organizations like the Singularity Institute. So, we say that our policy is to support the Singularity Institute. In nanotechnology, there is the Center for Responsible Nanotechnology and Foresight. What the Lifeboat Foundation likes to do is encourage people specifically to look into the human extinction risks, the worst possible catastrophes. It’s hard to get people to talk about it, because it’s so damn depressing.
A quick show of hands. I would like to know which of these three risks you guys think is the greatest. Some people have said that it’s plausible that humanity will wipe itself out. Who thinks robotics/AI is the number one risk in the next fifty years? We have two. Who thinks that nanotechnology-related risks are the biggest risk in the next fifty years? Okay, two. And who thinks synthetic biology-based risks are the biggest? Quite a few. Five. In synthetic biology there is Leon Kass and there is a focus on anti-transhumanism, preventing us from modifying ourselves. There is not as much focus on things that could wipe out everyone.
It’s partially because it is a challenge to wipe out every human being. There are six billion of us. We’re distributed to every single crevice of the planet. Even an asteroid 30 kilometers in size wouldn’t kill all of us. It would create a lot of problems and kill maybe 100 million people, but that’s not everyone. Looking at the fossil record we see that there have been only six major extinction events in the past billion years. That’s why I think it’s critical to put the focus on technology risks, not natural risks. If natural risks were a problem, we would have been wiped out by them long ago, because the hominid line has been around for millions of years.
Many people here are into sci-fi. We think that we are going to expand beyond the planet, right? Can you imagine if we were to colonize the entire Milky Way galaxy and everyone were having a good time? That’s a tremendous amount of positive utility. You could be talking about quadrillions of human beings here. Say we develop the technology to travel to other star systems at .1c in maybe one hundred years. The galaxy is 100,000 light-years across, so in a million years you are talking about the entire galaxy being colonized. That’s pretty damn exciting. A million years is just an eye blink in the rest of the lifetime of the universe. Even the most pessimistic physicists think we have another 15 billion years to go. Then you’re talking about colonizing the entire universe.
Now, say there is a scenario where 90% of people die, I still see us eventually reaching that happy zone where we colonize the entire Milky Way and everyone’s having a good time. But when you’re talking about existential risk, where every singe human being dies, it’s all gone. There’s no more galaxy full of happiness. To me personally, as someone who looks further into the future and wants to live for a really long time and cares about the long term future of humanity, the difference between 90% of humanity getting wiped out and 100% is huge. Even 99% versus 100% is huge. I think there needs to be an organization specifically focused on human extinction risk, which is what the Lifeboat Foundation is for, and there are a lot of people that agree with us. David Brin, who wrote The Transparent Society, is writing his next book focusing on the Lifeboat Foundation, so we should get some publicity from that.
Let’s brainstorm a little bit. How about some counter-measures to risks?
Audience: Don’t have all your eggs in one basket.
There you go. Not having all your eggs in one basket, space colonization, is one of the key initiatives of the Lifeboat Foundation. The Lifeboat Foundation is sponsoring an electromagnetic launch competition that’s happening in Arizona later on this month for the X Prize Games. It’s a competition to see who can launch a payload to the highest possible apogee using a thousand dollars worth of equipment. It’s a student-based competition to gain progress in cheaper launch strategies. Right now it’s pretty damn expensive to get anything into space.
Any other ideas? How do you prevent Armageddon?
Audience: Bill Joy’s idea. Cease and desist. Knock it off, fast.
Relinquishment–does anyone think that would work…? No one thinks it works, because this is a Foresight Conference, and we have enough Foresight. If some group had exclusive access to nanofactories then they could have a good shot at establishing a totalitarian regime. What if there is some group that wants to make it so that they are the only ones that have access to the nanotoys and they get to hand them out on their own? If I were a Luddite and I wanted to stop all technology development, that’s what I’d do.
Does anyone else have any other ideas? I’m going to say one. Transparency. Phillipe can Nedervelde, who is the head of Foresight in Europe, as well as the spokesperson for the Lifeboat Foundation, has a plan he calls 4E: Everyone has Eyes and Ears Everywhere. That would be facilitated by smart dust, sensors everywhere that are so ubiquitous that you can’t get them out of your clothing. If groups were distributing this smart dust, which would be tremendously cheap, especially through nanomanufacturing, so there isn’t a single cellar or closet in the world that doesn’t have some of this smart dust.
Transparency would not be a panacea. What Lifeboat Foundation is advocating, Everyone has Eyes and Ears Everywhere, is what others have called a participatory panopticon. It would be sousveillance as well as surveillance to keep it fair. There’s a site, livelink.com, where you can see a video of cops beating the crap out of people. These cops don’t know they’re being watched. They might be canned, they might be ridiculed. Sousveillance is already beginning to lead to beneficial effects. The most beneficial possible effect of all would be preventing the extinction of the human species.
If there is a hassle that has to be done while we’re vulnerable and on this little rock, then it might be a necessary evil. Obviously, everyone has to buy in, or a significant majority has to buy in, or it’s not going to work. No one organization can unilaterally instigate something like that. If there were one organization that were distributing smart dust everywhere against the wishes of, for instance, the United States federal government, then they could probably come down pretty hard.
Audience: There’s transparency, but then once you see something that you don’t like, you have to be able to stop it.
The original definition of existential risk given by Nick Bostrom was something that either eliminates humanity or permanently and drastically curtails our potential. A self-perpetuating totalitarian state of immortal dictators would qualify as an existential risk. Any others?
Audience: Ruggedization. Make humans less extinguishable. Greg Egan is writing about the kinds of transhuman upgrades where you can eat tires and survive extremes of temperature.
Totally. If we could create artificial white blood cells, artificial antibodies that could adapt to new pathogens as they are being invented in this synthetic biology, then we could nail synthetic biology. There would still be the risk from nanopathogens possibly that could overcome our defenses. If it would also be distributed ruggedization available to everyone with upgrades, then we would be in pretty good shape. There is also a challenge to selling this to the powers that be. You can’t be too transhumanistic about it. This organization has a lot of transhumanists involved in it, but it’s explicitly not a transhumanist organization, because we don’t want to socially sideline ourselves.
Any others? How do we prevent extinction? Another thing that could help us is the creation of what Nick Bostrom calls a “singleton,” a single decision-making agency on the top of everything. For instance, a world government or an artificial intelligence that is the most powerful entity on earth at any given time. Is there anyone who thinks it would be possible to create an AI that is both very powerful and does not abuse its power? I would trust for there to be an artificial intelligence that is more powerful than any entity on earth at any given time. I think it’s possible that out of a huge space of possible AI’s maybe one we could trust enough. I think that AI is the greatest risk and also the greatest passage out of risk. That’s why I support the work of organizations like the Singularity Institute, who is trying to create Friendly AI.
Lots of people disagree with me. For instance, most people here raised their hands for the bio risk. This is important because it’s coming up real soon here, like five to ten years. You don’t need to sit down and argue at length, which is what you need to do for robotics/AI. Some people don’t believe that human-level artificial intelligence is possible at all. So what we like to do is start down at bio risks and slowly work our way up as high as we can get until people start giving us weird looks and then we go back down. You have to maintain people’s credulity, otherwise they dismiss you as a nutcase.
There’s a Tragedy of the Commons issue here because every nation has the incentive to cover their own ass. Many nations promote international peace to the extent that it’s necessary for trade. When we awarded the Dalai Lama with that congressional medal, it made China really pissed off. Was that a smart thing to do, even if we do respect the Dalai Lama? Maybe in the interest of international peace we shouldn’t have done that. This is the kind of thing where if there is a gigantic war that kills off 99% of humanity, we’ll be able to say in retrospect, “Oh, that was a bad idea.” But now we’re in a position before we’ve gone into World War III, where we need to look carefully. For instance, this is relevant to all the decisions we are going to be making for who to vote into president next time around. I don’t know how many people here buy the exponential curve, but as the technology ramps up it gets more and more dangerous. That’s why I’m glad George Bush has the time where there’s less technology so if there’s a backlash against him, hopefully we won’t have another president like that. When the dangerous technologies start showing up we’ll have enough people that have their wits about them and aren’t engaged in cowboy diplomacy, ready to instigate World War III.
Any other potential solutions here? For those that don’t believe there are risks that could extend beyond the envelope of the planet earth, it seems like space colonization is the magical cure. It would allow us to escape any dangers that are on the surface of this planet. But if you start talking about advanced artificial intelligence trying to fulfill goals by reshaping matter around it, and also improving its own capability do that, it could potentially spread out in a sphere at the speed of light from its point of origin. Of course, when you start talking about that, there are some groups that you can’t talk about this sort of thing in front of, because they will look at you like you’re crazy. This is one group where you can.
With this issue of human extinction risk, potentially Tragedy of the Commons effects come into play. Every nation has an interest to protect themselves, but we don’t have as much of a global view. I think efforts to fight global warming are really important because it’s an example of people coming together and trying to work together about a global issue. I think that some of these existential risks will be global issues as well. One solution to this that we didn’t talk about would be global governance, having a global congress of some sort, because then they would have more of an interest in implementing these solutions. For nanotechnology regulation, regulations in one country might set a good example but they aren’t ultimately going to be effective unless they are global. Someone can run away to Cambodia and make their own doomsday nanotech device that is not under the purview of the United States or the European regulations. So, it’s a big issue.
I brought a book today that I think is pretty important. It’s kind of obscure. It’s called “Military Nanotechnology” by Jürgen Altman. It’s $120. It’s one of those academic books that costs a huge quantity of money and barely anyone reads. The reason why they are even able to publish it at all is there are certain libraries that buy almost every single academic book to put on their shelves. I think this entire book is really important. It looks at the combination of nanotechnology with various military applications. You’re talking about missiles that are this big, are tremendously deadly, and have the ability to navigate to their target. Those aren’t existential risks, per se.
I was also going to bring books like Unbounding the Future by Christine Peterson and Gayle Pergamit, who are here. Books like that I think underemphasize the risks. I think that because of the backlash due to grey goo, nanotechnologists have gotten into a dismissive attitude about risks. I think that we are not looking closely enough. Grey goo, we acknowledge that that was a little bit silly. We’re in a lab, working with these nanobots, and suddenly they get out of control and devour the entire biosphere. That’s a little bit implausible. When you’re talking about people creating it intentionally, it becomes more plausible. Any comments?
Audience: Could you say a little more about the book?
It talks about making everything smaller. It splits it up into a section on nanotechnology proper and molecular nanotechnology. Nanotechnology itself generally refers to miniaturizing things. I think it’s important because in winning the PR war, nanotechnologists tend to over-focus on the benefits. For instance, I’m a transhumanist, but that does not necessarily mean that I would condone every single transhumanist modification. For instance, what if someone were a wirehead, someone who only self-stimulates their own pleasure centers, that wanted to kill other people to ensure that no one else can bother it, and it can wirehead itself for the rest of eternity. It may be like a savvy wirehead. I see artificial intelligences could be that way, as well.
Audience: Are you aware of The Millennium Project?
I was really inspired when I first read Marshall T. Savage’s book on space colonization. But that organization is pretty much defunct.
Audience: I would add attaining “inner peace” to the list.
It’s problematic because human beings evolved in a zero sum society, where if I win it means you lose, and if I lose it means you win, because generally there is a finite amount of resources. Human beings, just like the rest of the animal kingdom, have these aggressive tendencies. When you fundamentally get down to it, when you talk about human beings with molecular manufacturing, it’s kind of like handing a kindergartener an assault rifle to play with. I think that some of these technologies are fundamentally too powerful for humans to handle. That’s a really controversial thing to say, because some people think that human beings are indefinitely improvable and we can figure out the right thing to do. I think that inner peace is pretty important, but human nature goes against it.
Let me put down a few probabilities of human risk from famous well-respected people. Martin Rees, the British Astronomer Royal, came out with a book last year called Our Final Hour. He said there was a 50% chance of humanity surviving for the next century. I think that’s a little pessimistic, but it gets the point across that there are people who are really serious about this. Nick Bostrom, a philosopher at Oxford, said it would be irresponsible to give it a probability of less than 20%. Stephen Hawking has come out lately, and I think some of his suggestions are a little half-baked. For instance, he says that in order to protect humanity against AI’s we need to use genetic engineering. That’s a little silly because genetic engineering doesn’t work fast enough. It’s not really going to help if you’re still built out of proteins.
Audience: Hawking was misquoted. He wasn’t quite as stupid as he sounded then.
That’s good. Hawking is big on space colonization. If you look on some of the Wikipedia entries for space colonization, the number one rationale they put down there is because Stephen Hawking said it would be a good idea to preserve the human species. I think Stephen Hawking said there would be a roughly 50% chance of survival in the next century. He’s pretty pessimistic, and I think he has good reason to be. Something about existential risk is that you can’t come back from it. If the disaster happens, it’s over. I think it’s worthwhile to be excessively prudent, because once we do spread out into the cosmos, the risk goes way down. Once we do have surveillance everywhere, the risk goes way down. So I think we’re in a critical period right now.
Eli wrote a paper on cognitive biases and how they come into play specifically when considering human extinction risks. I think those are really important papers. There is a huge neglect in academia of these risks and it’s mainly because they involve technologies that are advanced, technologies that are considered futuristic, technologies that don’t have working examples yet. I have mixed feelings when I read about synthetic biology coming in the next five to ten years, because on the one hand it gives me and others who are concerned about risk more ammo when there is actually a pond with synthetic biology in it converting sunlight and basic nutrients into biofuel. What if there were a dangerous variant of that. You could speculate all day about it, and science fiction writers have got various disaster scenarios. It’s not hard to come up with some. I just hope that I can use the technology that’s out there to argue for preventing the biggest disasters before they actually happen.
Audience: You are too optimistic. The very instant that someone creates artificial life in a pool, it will cease to be shocking and futuristic. They will say, “Oh, I could have predicted that. Now, this stuff you’re talking about… is out there.”
Right. One of the things about AI that’s so frustrating is that this is the one that people consider to be the least plausible existential risk, but it’s also the one that could come most suddenly. People have talked about a hard takeoff scenario where you have a roughly human-level AI that is recursively self-improving goes superintelligent in a period of days or even hours. It’s hard to make that case because most people don’t consider that possible. What I think is worth doing is just arguing as best we can. If people think AI isn’t an existential risk, then let’s argue about nanotechnology being a risk. If people aren’t willing to believe that, let’s talk about synthetic biology.
I just want to get the conversation going. There’s a lack of it. People have Cold War malaise. They’re tired of it being said that we might all destroy ourselves. They’re just like, “Who cares? I just want to live my current life.” But if you care about that human future of a colonized Mily Way versus a dead Milky Way, then it’s a big deal to you now even if you don’t even plan to survive until then. It’s a responsibility to our children, our grandchildren, and if we can continue to live, even to ourselves. There is huge neglect in that area. I want you all to go home and think about the risks that we face in this coming century and how serious they are.
I personally believe that the human species could be the only life form in the entire universe, so if we mess this up there is no going back. So think about it carefully. We have the entire universe and we’re just little human beings on this rock. So take advantage of this opportunity.
