Scratching my Head about Global Risks Monday, Dec 10 2007
risks 2:29 am
Zooming in on the real global risks:
1. custom-engineered prions/virions?
2. genetically engineered viruses
3. genetically engineered bacteria
4. cybernetic (combinations of bio and nano) microbes
5. molecular manufacturing - self-replicating nanites
6. molecular manufacturing - nanowarfare, arms races
7. human or near-human-level artificial intelligence - controlled self-improvement
8. human or near-human-level artificial intelligence - unrestrained self-improvement
9. human intelligence enhancement gone awry
Broadly, I categorize these risks as either self-replicating (1-6) or self-amplifying (7-9).
In terms of the time window of risk:
For 1-3, the risk window is approximately 2010 onward.
For 4, the risk window begins around 2010, but doesn’t get serious until 2020+.
For 5-6, this is a big unknown. Some argue for emerging risk as early as 2010, but 2020 onward seems more realistic to me. Once it emerges, it will balloon quickly.
The time frames for 7-9 are the most unsure and contentious. For those who see human-equivalent AI or human intelligence enhancement as prospects at all, predictions cluster around 2020-2030.
What do all these risks have in common? They’re relatively close, quite serious, and virtually ignored. There are only a handful of organizations that have looked at them in any detail.
Given all the resources devoted to improving quality of life, fighting disease, maintaining economic momentum, promoting democracy worldwide, and so on, it’s shocking that humanity has devoted so few resources to ensuring its own survival.
Are all of these risks so low in probability from the viewpoint of casual thinkers that they aren’t even worth talking about? I find this hard to believe. There are several prominent figures who have mentioned the risks: Stephen Hawking, Martin Rees, and Warren Buffett. There was recently a talk at the high-profile TED conference about global risk.
A confounding point is that many of the mentioned risks are based on technologies which may not exist yet. But genetically engineered viruses, for instance, do already exist. Even entirely synthetic viruses have been created. Cybernetic microbes do not yet exist, but are very close to being reality, perhaps 3-5 years in the future.
Molecular manufacturing (MM), AI, and human intelligence enhancement-based risks are more difficult to communicate. In the case of molecular manufacturing, the basic concepts still aren’t clear in the minds of many. Discussions of the technical requirements of molecular manufacturing (reliable mechanosynthesis and self-replicating robotics) do not tend to be especially controversial, but when the products (likely to be made of diamond) are mentioned, the controversy and incredulity meter often goes through the roof. More discussion and study is needed.
AI and human intelligence enhancement are the most contentious risks. A small group of people take the risks very seriously, another small group of people seem to believe that greater intelligence equals greater morality, so we have nothing to fear, and the largest group, including many self-described geeks, let Hollywood do the thinking for them. Another distraction here is spiraling off into philosophical debates and identity crises caused by exposure to the possibility that humans won’t be the most intelligent species on the planet forever.
Whichever risk you take most seriously yourself, even if none, my advice would be to keep an open mind about the possibilities. Many of these technologies aren’t even on the drawing board yet, though some are, and there is no such thing as an expert in many of these areas. As we figure out more, our probability estimates will surely change. My estimates of the time frame and intensity of the risks has changed numerous times in the five or so years that I’ve been reading books and papers about them.
If you consider study of the risks to be important, please join the Lifeboat Foundation, which seeks to fund full-time investigation into these issues.
Further resources:
Lifeboat Foundation Reports
“Existential Risks” by Nick Bostrom
“Immortalist Utilitarianism” by myself
“Risks” category on Accelerating Future
“Existential Risk Analysis” category on the People Database Blog
Singularity Institute for AI, looking into AI-related risks
Center for Responsible Nanotechnology, looking into nanotech-related risks
December 10th, 2007 at 6:03 am
“another small group of people seem to believe that greater intelligence equals greater morality, so we have nothing to fear,”
I’ll just add that whilst I think that it may be the case that greater intelligence almost always results in greater morality [there are, IMO, decent reasons for believing this], I’m not sure about this, so I would not say we have nothing to fear.
Thus I fully support the position that AGI should be considered a serious existential risk, and that we should take steps to minimize it.
December 10th, 2007 at 6:31 am
For those who see human-equivalent AI or human intelligence enhancement as prospects at all, predictions cluster around 2020-2030
Most of the people who see beyond-human intelligence as a prospect are leery of saying any date, given the enormous amount of current uncertainty. (I, of course, am always happy to cross that line from “a reasonable level of honesty” into “blatantly self-destructive behavior”, and am happy to reveal my uninformed prediction.)
I also get the sense that, if you forced all of the FAI community at gunpoint to reveal the most likely date, the range might be more like 2020-2060; I have a strong sense that more people would choose 2030-2040 than choose 2020-2030. (This is all speculation; for example, Yudkowsky is smarter that I am and won’t publicly guess a date.)
As always, the importance of the listed crop of existential risks is not necessarily that any one of them is Almost Certain to Happen, but that all risks should be tackled in proportion to “their impact times their probability.” If a risk is at 10% rather than 100%, then that means it should get 10% of the resources it would if it were certain, rather that one-millionth of the resources it would if it were certain.
December 10th, 2007 at 7:57 am
Michael, I like your inclusion of the bio/nano combination microbe. I submit that bio/robot combinations (grobyC inverse of Cyborg) will also become a significant risk–if not existential risk–perhaps sooner than most others on the list.
You cannot assume anything about the “morality” of an advanced machine intelligence.
The safest approach with advanced machine intelligence development, is to only connect the AI with a simulated reality–preferably in a physically isolated environment (dedicated power generation). Make sure the AI doesn’t “know” that a larger world exists outside the simulation.
This would give developers time to observe the AI in action, with the ability to monitor trends and rates of change. The isolated power supply would always be subject to disconnection.
I recommend that all prognosticators on AGI and FAI be required to spend at least one month out of the year working with actual projects such as Numenta, Novamente, etc.
December 10th, 2007 at 10:23 am
I agree with Al Fin that AI development should be in an isolated sandbox, but I doubt that will happen because there will be pressure to use the AI to gain a military or economic advantage. That may not be the goal of the researchers, but that’s what the people writing the checks want. They’ll want a return on their investment, as soon as possible.
I also think that we can’t assume intelligence equals morality. Would an AI developed from systems originally designed for military purposes or economic domination be “moral” or have our best interests in mind? I wouldn’t bet the farm on it.
December 10th, 2007 at 12:00 pm
The point about prions is interesting. Hmm.
On the human intelligence enhancement point, I’d suggest two things:
1. It’ll greatly depend on the culture of the discipline that arises to research and implement it.
2. Even moreso than just kicking off spiraling philosophical debates about what is human, I think human enhancement has the potential to cause deep social unrest- both in terms of how those thus enhanced can change society, and in terms of fundamentalists trying to stop or turn back or fight against these technologies. I think this secondary reaction to human enhancement could get *very* violent and polarizing, and I’d say it’s a real existential risk.
December 10th, 2007 at 12:29 pm
When discussing global risks, it is beneficial to focus on physical mechanisms above underlying social causes. Otherwise, it gets exponentially complex, no real facts can ever be agreed upon, and zero progress is made. It’s easier to answer objective questions and get the actual facts about physical technologies, even not-yet-existing technologies, than future social situations we can scarcely anticipate.
As an aside, I doubt it can get that violent or polarizing, because either it will be so incremental it’s analogous to boiling a frog slow enough it doesn’t notice, or so quick that the normals can’t stage much of a response. In science fiction movies, there is always an implausibly precise parity between both sides for the sake of dramatic conflict. In real life, the new species usually wins outright. Think about a battle between Homo sapiens and Homo neanderthalis. 99.8% DNA similarity, but the winner is obvious.
December 10th, 2007 at 2:02 pm
Interesting you should mention the Neanderthal example… there was fascinating paper on this that just came out.
(Short version of that paper: there’s good evidence that “archaic introgression”- i.e., gene flow from neanderthals- greatly contributed to the human brain into its modern form.)
At any rate, I think you’re right- concentrate on the capabilities of the technologies, rather than the deeply chaotic social angle. Predict what can be predicted.
I do think disagreements over human enhancement have a significant potential to turn violent, however, given the amount of fundamentalism in the world and the increasing use of terrorism to protest against modernity. Well, it’s hard to predict, as you say.
December 11th, 2007 at 10:15 am
If you are developing a new technology in the civil field there are all sorts of hoops you have to go through, some rational, others less so.
The military is able, on the other hand, to obtain money and carry out a program in complete secrecy. To me the reduction of existential risk involves having greater transparancy and accountability of military programs.
Forget about terrorism. Al Qaeda is incapable even of doing a proper distillation (reduced pressure) of Hydrogen Peroxide.
December 19th, 2007 at 1:21 am
Net Girls On Youngest The…
Nice Sharon Wildd in spite of since. Tera Patrick an Brittney Skye a electron salvation? …
May 27th, 2008 at 4:25 am
At risk of restating myself, again…
We need a contained demonstration of an existential risk. Nothing else short of an uncontrolled demonstration in the open (i.e. disaster) will get people to take the risks seriously.
You are right, we are not far at all from several existential risks. Sorry, Lifeboat will not prevent all existential risks forever. We need to survive the first existential risks if attention and money are going to be devoted to preventing these risks and securing a distant human colony.
We need a contained demonstration.
We need a contained demonstration!
May 27th, 2008 at 6:21 am
John, I’ve been thinking the exact same thing lately. I’ve just been sort of afraid to say it. In fact, I even have a draft of a post saved articulating this very concept.
Dangerous ground. But you’re right. Every really smart person that thinks about this comes to a similar conclusion. I suspect that many others have thought it but were also afraid to speak it aloud.