A Bare Minimum for Extinction Safeguards Wednesday, Sep 24 2008
risks 1:31 pm
Say that the world’s foremost microbiology expert announced that he or she was working on a synthetic virus with the ability to wipe out all human life. Or that a space-capable country, say, the United States, decided that its new mission is to locate the largest possible near-Earth asteroid and alter its trajectory so it impacts the Earth. Or that the world’s foremost expert on machine learning decided that he is tired of humanity and wants to create an AGI that shreds us all to pieces, to put us out of our own misery. (Yes, the technology to make these risks real may not exist now, but imagine 10 or more years down the line — the near future.)
I’d want to be able to say, “So what? Go ahead. You’ll never succeed, we have safeguards against that in place, and they’ve been extensively tested.”
But I can’t, and we’re always in danger, and will continue to be until safeguards are in place.
A key difference between me and many others who think about extinction risk is that I wouldn’t be unpleasantly surprised at a Doomsday Announcement by a leading scientist — surprised as in I’d feel sad about the announcement, but I would have accounted for the possibility in advance. I’d say, “yes, it’s sad you have to be this way, but I knew all along that we needed safeguards in place that would account for this eventuality”.
Of course, the available resources at my disposal are far less than is necessary to put such safeguards in place. However, it could potentially be done for lower cost than one might think. By fueling our watchguard systems with superintelligence rather than human intelligence, they might actually work, rather than failing when the guy watching the cameras goes to take a leak (”human error”). Recursively self-improving superintelligence might be accessible merely through a well-funded seed AI or human intelligence enhancement effort. A few million or even a few thousand dollars can go a long way here.
Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it? Say “we need to optimistic in life, we can’t always worry about risks, lol”? The problem with that outlook is that it causes you to die.
13 Responses to “A Bare Minimum for Extinction Safeguards”
Leave a Reply
You must be logged in to post a comment.
September 24th, 2008 at 3:22 pm
“Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it?”
This example is extremely silly, and you definitely should be surprised if such a thing happens. The technology for an extinction plague is far beyond current levels and faces major intrinsic challenges. An *announcement* would result in them being immediately arrested, and invasion of any country harboring them, and it would be ludicrously difficult to form such a conspiracy (intentionally omnicidal geniuses are incredibly rare, and each new member recruited provides another possible leak for the conspiracy).
An attempt to divert an asteroid would be trivially detected, and if the resources to move were to exist, then so would the resources to reverse the attempt.
There is more potential for an AI-driven extinction than the other two, but again that would need to proceed in a secret military project if elites believed that private civilian projects could feasibly produce humanity-destroying AI.
I realize that your point is that it would be nice in some ways to have defenses that would withstand such an unopposed effort, but the scenarios don’t really make sense, and so present a bad image. It’s like saying that we should develop and wear diamondoid skintight body armor so that we can defend against rampaging tigers without using offensive weaponry. Neat, but not the most cost-effective course to prevent tigers from eating people.
September 24th, 2008 at 8:33 pm
“This example is extremely silly“… I wish that were true.
September 24th, 2008 at 11:05 pm
Would you have to be able to anticipate all rapidly developing xRisks before hand?
After 27 years of the HIV infection, we still don’t have a useful vaccine.
I’m skeptical that we can anticipate all xRisks. And I am skeptical that we can respond quickly enough when a novel xRisk appears.
Defenses are fine but not sufficient. We also need to develop means of having at least a few people be able to survive and then thrive after an anticipated xEvent.
September 25th, 2008 at 1:44 am
Carl, I made a few changes to emphasize that these risks aren’t necessary technologically possible immediately, but will be soon. I disagree with your highly confident assessment that the technology to create an extinction plague doesn’t exist now.
For an announcement, you can think of it as an announcement to a small group of people rather than the world, to make it more plausible. Of course this is just a thought experiment. Regarding a cooperative group of 100 people, again I think you’re overly confident in your dismissal. Aum Shinrikyo, for instance, had over 50,000 members at one point, and used skilled scientists to research chemical weapons. They obviously did a good job hiding what they were doing, or they would have been caught before the sarin gas attack.
Mentioning asteroid risk is useful because one day many more countries will have space technology, and once an asteroid is put on course to hit Earth, it might be hard to change it, especially if it’s protected somehow. It’s also vivid. Many people believe that any risk whatsoever from AI, nano, or bio is silly, so sometimes I talk about side risks. It being a thought experiment, you can also subtract any of these risks and insert an “unknown risk” in their place.
John,
The point isn’t to anticipate all x-risks with certainty, but to anticipate and respond to as many as possible.
September 25th, 2008 at 5:51 am
“I disagree with your highly confident assessment that the technology to create an extinction plague doesn’t exist now.”
An extinction plague is different from a civilization-collapsing pathogen: it needs to reach isolated hunter-gatherer populations, almost universally defeat resistance, have a long latency time without enough variation to reveal its spread, not evolve into a variant that spreads more efficiently but lacks the the other needed properties (while conferring immunity), etc. That’s a lot of basic research to do, further hampered by difficulty in testing results without giving the game away.
“Aum Shinrikyo, for instance, had over 50,000 members at one point, and used skilled scientists to research chemical weapons.”
This is a fairly low level of ‘genius’ relative to ‘global expert with monitored bioweapons expertise.’
“It being a thought experiment, you can also subtract any of these risks and insert an “unknown risk” in their place.”
My main beef with the idea is that countermeasures including active surveillance and interventions can dramatically reduce the sophistication of passive defenses required to greatly reduce given risks, and we should pursue the most effective approaches.
For example, nuclear terrorism can be interfered with by helping to secure national arsenals and flooding markets for stolen nuclear material with fake buyers and sellers from intelligence agencies, and this is more cost-effective than passive measures.
September 25th, 2008 at 10:11 am
You might have some expendable or alias create a web community called ‘exterminating humanity’, add articles, sections and a forum and start discussing serious attempts to exterminate all of humanity. Seriously. Discuss viral epidemics, homegrown pathogens, dispersion, cobalt irraditation, climate terrorism, torching all abandoned coal mines to release massive abouts of CO2, rerouting asteroids - every single scenario - with the intent purpose to trigger the mass extinction of as much human beings as possible. Discuss practical approaches. Don’t get into traditional terrorism (that might cause you to get arrested) but only discuss the type of scenarios that are hypothetical. If some intelligence busybody questions you about your plans with nanoids you can always claim “but these are hypotheticals. science fiction” … This website will do a LOT more to activate and alarm people and policymakers than what the lifeboat foundation does now. Look at it as a devil’s gambit, or an antidote against the real thing.
These things are remote from policymakers and the swill of retarded voters. These people don’t get warnings. They need examples. They need a forum section where a guy explains with schematics he can in fact launch a rocket, can in fact change the orbit of a NEO and can in fact have the 17 mile asteroid rain down on earth in 33 years.
Then proceed to calculate the best impact region, causing the most death, economic disruption, atmosphere damage, biosphere die-off and resulting afghanization of remaining human societies. Then have people complain on the forum “97.9% deaths in one year? We are doing way better in the nanoplague section. Amateurism”.
Discuss overlapping, *intentional* massacre scenario’s in a cold, hateful and detached manner. Discuss creating humanhating AI’s and giving those AI’s startup resources to inflict maximum damage on humanity. Plan by giving the AI’s access to weapon’s factory by creating plans how to hack, intrude and divert resources. Then claim you worship Satan and are doing His will. Show pics of you with horns and a black cape. Call your organization SPECTRE.
Of course this whole idea is psychotic and unacceptable. But the alternative, not causing relevant people to acknowledge the actual risks unvolved is by far more ludicrous. I mean, most people are clueless.
September 25th, 2008 at 10:22 am
Carl, by “extinction plague”, I mean something that kills, say, 99% of humanity, or a suite of deadly plagues that overlaps to achieve 99% or so mortality. In combination with say, major nuclear war, this could be enough to cut humanity down to a society primarily consisting of hunter-gatherers and other extreme rustics, which then fail to survive through the nuclear winter.
By “safeguards” I also meant active interventions, not just passive safeguards. This is a language issue.
Dagon, good idea, and I’ve certainly thought about it. I’m somewhat worried that my reputation would be damaged if I started such a project, but my concern about x-risk is great enough that I may be willing to deal with the blowback.
September 25th, 2008 at 4:57 pm
Dr Panika (panic is right!!) is a jerk. A jerk with a Ph.D (gee, how many of those s.o.b.’s do we have wandering around these days…) but a jerk nonetheless. Thanks, IConrad, for reminding us of him and his utterly RIDICULOUS ideas and positions and proposals. I’m embarrassed that he’s ensconced in an institution of higher learning in my belove great state of Texas. With all due respect to the spot-on critiques of Tom Szasz and Pete Breggin, if any position would qualify as clinically (and, indeed, legally) INSANE, it’s Panika’s absurd droolings…
And, yeah, Dogon is right—Michael, please start another blog exclusively devoted to x-risk. You could either do it straight—OR, and this **might** work even better, make it a mock site wherein you mock-*advocate* all this extreme stuff, done in such a way as to highlight and make clear the utter ridiculousness of it all, but with dead-pan mock-seriousness. Hell, I’d contribute to such a site, for sure. These bacteria- and virus-hugging, “save-the-biosphere-from-human-scum” MAGGOTS really do piss me off… ;)
Death to all fanatics!! ;)
Shalom
September 25th, 2008 at 4:58 pm
Woops, DAgon, not DOgon—sorry, dear colleagues, can’t see or read worth a flip at this particular time, apparently!!
September 25th, 2008 at 11:32 pm
MCP2012, I just want to remind you that it’s Pianka, not Panika. :)
That being said… the Discovery Institute is also a well-known “Creation Science” group. But this in no way excuses the statements of Pianka and the reception these statements received at the Deep Ecologist movement’s participants’ hands.
September 26th, 2008 at 11:56 pm
I can do this and I don’t mind handling (any) controversy and I am available to act as Proxy, from the Netherlands. I have a clear idea on a mission statement - and we would draw a clear line in the sand between actual practical projects (though we can discuss them as hypotheticals) and stuff that will get me hauled off to Syria.
We seriously discuss strategies and actual working plans, budgets, aims - to sterilize the planet and cause irreversible destruction of humanity, ideally the sterilization of all life down to the microbial level.
We hide a page we are a sinister group driven by collective dream visions we receive from the demon Abaddon or something like that. Nyarlathotep. Whatever. To generate a suifficiently ludicrous yet compelling motive.
However I can’t handle the content and site design all by myself, I am pretty busy IRL. I can upload it, add articles, look at the forums occasionally - but I’d need a shadow writer team and someone who looks at security (government probes, hacking etc.)
Let’s call it the Humanity Eradication Project.
October 2nd, 2008 at 4:54 pm
Dagon; better, try the Involuntary Human Extinction Movement.
This would be a contrast to the Voluntary Human Extinction movement.
Start posting in online forums associated with the VHE group. See how many sign-ins and assignations you can get.
We can go from the pacifistic methods of human extinction to the absolutely horrific; the general idea being that human beings are bad for the world/universe, we are a virus, and we should be exterminated for the betterment of Nature And Everything. You Know. Stuff.
October 21st, 2008 at 6:31 am
Hi! cymbalta reviews