Existential Risks in the News Friday, Aug 25 2006
risks 6:36 am
I recently found an article on existential risks in the Guardian from about a year ago. Of course, the Martin Rees book is immediately cited as justification for running the piece. In the article, 10 risks are summarized and scientists are asked to give a paragraph or two of commentary. They are, in no particular order:
1: Climate Change
2: Telomere erosion
3: Viral Pandemic
4: Terrorism
5: Nuclear war
6: Meteorite impact
7: Robots taking over
8: Cosmic ray blast from exploding star
9: Super-volcanos
10: Earth swallowed by a black hole
The likelihood of many of the above risks was exaggerated in the piece. Climate change could be troublesome, but it operates over long timescales, and couldn’t possibly kill us all. Terrorism and nuclear war wouldn’t kill everyone. Telomere erosion is just silly. Large meteorite impacts, supervolcano explosions, and cosmic ray blasts all happen only once every few tens of millions of years, so I think we’re okay for now.
A viral pandemic is a serious risk, though not as great as some others. Getting killed by recursively self-improving robots/AI is probably the greatest risk to our future, and the most poorly understood (thanks to Hollywood and our innate tendency to anthropomorphize and mechanomorphize). The Earth getting swallowed by a black hole or stable strangelet generated in a particle accelerator is one of those wild cards. The Brookhaven study rates the probability as negligible, but other studies still encourage caution. Anthropic calculations by Max Tegmark and Nick Bostrom give an upper bound of once per 10^9 years for the occurrence of such disasters. It is true that extremely high-energy cosmic rays slam into the moon regularly without creating stable strangelets.
Risks they missed: deliberate or accidental misuse of nanotechnology, badly programmed superintelligence, genetically engineered pathogens, repressive totalitarian global regime, take-over by a transcending upload, or something unforseen. See the classic Bostrom paper on the issue here. For organizations working on comprehensive solutions to address global risk, see the Lifeboat Foundation and the Singularity Institute.
August 25th, 2006 at 9:18 am
If recursively self-improving AI is the greatest risk, then we’re all doomed. A population of recursively self-improving AIs is a population of evolving agents, with complete control of their evolutionary tragjectory, and a nonzero probability of error. It’s only a matter of time before a critical mistake is made and the planet goes caput.
I’ve posted that criticism on SL4 several times, even directly to Eliezer, and never got a response. If my criticism is valid, then you’re wasting your time trying to build “frinedly” AI.
August 25th, 2006 at 10:30 am
An interesting topic, that has broad appeal. The Guardian enjoys treating this subject from time to time.
Peter Hamilton’s sf series that starts with “Pandora’s Star” is an interesting look at recursively self-improving entities and how quickly they can spread out.
Personally I think the greatest threat of this century to humans is religious and quasi-religious fundamentalism and the dumbing down effect that necessarily accompanies.
August 25th, 2006 at 1:20 pm
What happened to genetically modified organisms? Is it just me, or do we have possibly the same problems with these as we do with chemical pollution? Being taken over by robots is a little out there. This really wasn’t a very serious article, but it could have been.
August 25th, 2006 at 3:02 pm
If my criticism is valid, then you’re wasting your time trying to build “frinedly” AI.
So your criticism is that good AIs automatically turn bad or indifferent if you give them enough time, because error-free hardware is physically impossible.
This is an invalid criticism because the Friendliness isn’t supposed to be dependent on flawless hardware. Error tolerance and all of that. Note that there are altruistic humans despite evolution, errors, etc.
No need for a population of evolving agents - a single one to start is fine.
Or is your criticism the following - “as long as there is more than one being in existence, there will be competition, and when there is competition this puts an upper bound on how nice any agent can be”?
August 26th, 2006 at 9:14 am
Martin Striz :
It’s only a matter of time before a critical mistake is made and the planet goes caput.
Or the self-improving AI itself goes belly up, or it does ANY other kind of ridiculous blunder!
Why would there be a premium for “planet destruction” ?
BTW, there is a much more REALISTIC existential risk which has been missed by the Guardian first author (Kate Ravilious) by a narrow margin but reported by one of her colleagues (John Vidal) just one week later:
The end of oil is closer than you think
So close that it is already there as of 11 Aug 2006:
No more business as usual
Friendly AI, Singularity & whatever else need to go a bit in overdrive to catch up…
August 26th, 2006 at 9:20 am
Ahem, Michael, are you time travelling already?
Your linked article to the Guardian is dated Thursday April 14, 2005, LAST WEEK???
August 26th, 2006 at 9:37 am
Your linked article to the Guardian is dated Thursday April 14, 2005, LAST WEEK???
My bad. Error corrected.
BTW, there is a much more REALISTIC existential risk which has been missed by the Guardian first author (Kate Ravilious) by a narrow margin but reported by one of her colleagues (John Vidal) just one week later
Yes, there are many peak oilers about. But likewise, there are many others who disagree with the risk. “Existential risk” means “everybody dies”, not “civilization collapses”. I hope you don’t post peak oil links in every post I write on existential risks in the future, but if you insist, I think you can guess what I’ll do.
August 26th, 2006 at 12:40 pm
“Existential risk” means “everybody dies”, not “civilization collapses”.
Given the interconnectedness of the modern world a severe collapse can do as much as climate change, or meteorite impact, or supervolcanos.
You seem to count these as “Existential risk” but animals of similar size survived these events, WE survived the Toba eruption 70,000 years ago.
As well, neither Nuclear war nor Viral pandemic will NECESSARILY wipe out “everybody”.
Cosmic ray blast from exploding star probably happened too and some ascribe the Mycaenian civilisation disappearence to such an event (can’t retrieve the link right now but I could if asked).
And “everybody dies” from Terrorism is plainly a JOKE!
Why dismissing Peak Oil (or Peak Whatever)?
No singularitarian solution to these?
August 27th, 2006 at 12:25 am
No singularitarian solution to these?
Of course, the solution is to make ourselves smarter with technology, so that our problem-solving ability radically increases.
August 27th, 2006 at 4:18 am
Of course, the solution is to make ourselves smarter with technology,
Agreed, I would more precisely say AI but since I have been watching AI for more than 30 years I am a bit pessimistic.
There is another “weak point” I am worried about, techno-optimists don’t seem to have a real grasp of thermodynamics nor of the meaning of the word finite.
August 28th, 2006 at 10:25 pm
To the contrary, Jean-Luc, techno-optimists have invented virtually every meaningful invention utilising thermodynamics in the past two centuries. It is techno-pessimists who lack a genuine grasp of thermodynamics. They merely say the word and assume they have won some argument or other. Putting thermodynamics to actual use is the best way of demonstrating an understanding, not using it as a boogeyman.
August 28th, 2006 at 10:56 pm
There is another “weak point” I am worried about, techno-optimists don’t seem to have a real grasp of thermodynamics nor of the meaning of the word finite.
Whatever.
September 8th, 2006 at 6:09 pm
Sorry, I totally forgot about this post. To return to the topic…
So your criticism is that good AIs automatically turn bad or indifferent if you give them enough time, because error-free hardware is physically impossible.
Given enough time and/or a large enough population, (good) AIs that rewrite their own code with a nonzero probability of error will eventually start to diverge from their original goal-system and explore the behavior space. It may happen along a smooth gradient, or if a critical error is made, larger saltations. One way or another, it will happen. Not all errors will lead to catastrophe, as was pointed out. But the space of possible bad outcomes is bigger than the space of good outcomes.
This is an invalid criticism because the Friendliness isn’t supposed to be dependent on flawless hardware. Error tolerance and all of that.
It’s a valid criticism of the assertion that you can guarantee friendliness. If you’re merely claiming that you want to make a friendly AI to start with, but not that you can ever prevent them from becoming unfriendly, then I have no problem with your position.
Note that there are altruistic humans despite evolution, errors, etc.
Yes, and plenty of bad ones too. It’s just that none of them up until now have had the power that a future AI might have to inflict damage. Even the first AI that accidentally rewrites it’s own code to tile the solar system with smilie faces could be catastrophic.
No need for a population of evolving agents - a single one to start is fine.
Sure, but eventually there will be more. Even if you don’t want to build more, what if the AI wants to replicate because it surmises that that’s the best way to fulfill it’s goals? Remember, a smarter-than-human AI is not a tool or a toy, it is a person. It would have as much right to replicate as you.
November 1st, 2007 at 5:25 am
Xanax 2 mg bars….
Xanax 2 mg bars….
August 3rd, 2008 at 10:45 am
Dog and human love making….
Love my dog named sex. Baskin love me love my dog….