More on the Bioweapon Testing Idea

On Monday I proposed that a demonstration of extremely destructive self-replicating technology may be necessary before the world takes the risk of it seriously, and starts devoting the necessary attention and money to developing comprehensive safeguards. I proposed such a demonstration very hesitantly, in the context that it may be the only way to get global attention on this very important issue, but still, some said I had “lost it”, and that such talk “polarises opinions on Transhumanism”.

Two points:

1) Prevention of global catastrophic risks is only tangentially related to transhumanism. When I was thinking about the proposal, I had nothing about transhumanism in mind. My blog tagline is “Transhumanism, AI, nanotechnology, and extinction risk”, because these are separate topics — interrelated, maybe, but separate. Sometimes I talk about just transhumanism, sometimes I talk about just nanotechnology, sometimes I talk about the relationship between the two. On Monday I was talking about just extinction risk, and what can be done to lower it.

2) Extinction risk prevention is more important than transhumanism. If we don’t survive the 21st century, then not only will the future lack cyborgs, life extension, space stations, and all that other exciting stuff, but it will also lack such mundane things as smiles, good food, jobs, picnics, writing, and just about everything else uniquely associated with the human species. Based on all the conversations and reading I’ve done, I consider humanity’s risk of wiping itself out in the next few decades to be substantial, and therefore consider it worthwhile to develop outside-the-box solutions to counteract that likelihood. Transhumanism and biological modifications are one of those fun activities you get to do if your species survives.

That absolutely includes proposals to test bioweapons in controlled environments for the purpose of determining their destructive capacity.

Take a look at the situation. Two of the greatest living scientists, Stephen Hawking and Martin Rees, have spoken out on the extreme danger of human extinction, and both the public and academia have failed to listen. If the people won’t listen to them, who else is there? Paris Hilton? Wilford Brimley? Our effort to educate the public on the dangers of advanced technology is failing, and we need solutions fast.

No one could deny the destructive power of the atomic bomb after the test at Trinity. But people have difficulty imagining new, better technologies with even more destructive potential. Discussions surrounding the regulation and control of post-nuclear weapons of mass destruction should be front and center. This should be the first topic that gets brought up at Presidential debates, and all the leading universities should have institutes devoted to studying self-replication in biological and nonbiological systems, to understand how difficult these systems are to build and how to keep them in control in the instance of accidental release.

Instead, the possibility of out-of-control self-replicators is widely seen as a joke, a science fictional plot device. I disagree: the possibility is real, so real that it could threaten human life by the year 2015 or earlier. Attention must be focused on the issue immediately. I am merely brainstorming to come up with some way of doing this.

To temper my suggestion, I’ve realized it could be done in a way more limited fashion than currently envisioned. As John Hunt remarked in the comments thread for Monday’s post, the test could be carried out in a level four containment facility, or feature something more mundane than wiping out all the life on an island, for instance “having little robots cracking all of the eggs on an small island and using their juices for fuel”.

Unfortunately, the point here is to make people afraid. We have every reason to be rationally afraid. Being rationally afraid doesn’t mean panicking, it means taking global catastrophic risk seriously. If people aren’t afraid, then either they disagree with us on technical grounds (if so, we welcome these arguments) or emotional grounds (thinking about human extinction is too unpleasant). Even though representatives of the latter group are very common, I have no sympathy for them — if ignoring the prospect of human extinction is necessary for you to feel pleasant on a daily basis, then ignore it, but tolerate the brainstorming of your more pessimistic peers on how to lower the risk.


  1. Nick Tarleton

    IMHO, self-replicating factories building conventional weapons are no less of a threat than out-of-control replicators.

  2. Warren Bonesteel

    As a thought experiment, the idea is worth serious consideration and effort.

    Your original proposal to actually try this out in the real world is the very definition of sociopathic.

    Right up there with pulling the wings off flys and torturing the neighbor’s pet.

    If you haven’t noticed, a lot of people are already *very* afraid of the technology. Irrationally so. Inflaming those fears in such a way as was originally proposed is counterproductive to the stated goals of Tranhsumanism.

    Destroy a desert island just to prove how dangerous the technology actually is?

    You have no defense for that proposal, Michael. The general public will associate that proposal with Transhumanists. You’ve only further damaged the credibility of all Transhumanists by making such statements.

    Just apologize and keep moving. …and don’t do it again.

  3. It’s a nice idea, but I doubt the general public would respond in the correct fashion. People are severely irrational and often react to things in an emotional way.

    If some organization were to demonstrate a destructive bioweapon, I suspect the primary reaction would be (a) fear and knee-jerk legislation and (b) endless speculation as to the ulterior motives of those who proposed the test.

    Look at the US public reaction to 9/11 – the US decided to invade Iraq. Did Iraq have anything to do with 9/11? About 50% of the US public associated iraq with 9/11 – source

    When people get scared, they panic and behave irrationally. I think that averting global catastrophic risks with the support of an irrational, apathetic public is a tall order – my strategy is to work on AGI and ethics and hope that nothing bad happens soon.

  4. Conventional weapons can be addressed with conventional and nonconventional means. One very simple method would be to restrict the availability of the elements used in the creation of explosives to relatively low levels. There’s not much you can do with pure hydrocarbon explosives, but those leave specific traces which can be addressed in semi-nonconventional methods; such as a variation on Nanoshield that reacts to a certain density of standard compounds…

  5. Warren,

    An isolated island isn’t “the real world”. I proposed a controlled test in a quarantined environment. If an island turns out not to be quarantined enough, then a containment facility might be used.

    People aren’t afraid of out-of-control replication technologies because they don’t know about it.

    Destroy a desert island just to prove how dangerous the technology actually is?

    And who exactly would be hurt by this?

    You have no defense for that proposal, Michael. The general public will associate that proposal with Transhumanists.

    I have every defense for that proposal: no one would be hurt, and everyone would benefit. In any case, I’m not famous enough for the general public to be aware of my proposals, so no worries!

    Just apologize and keep moving. …and don’t do it again.

    Why don’t you apologize for calling me a sociopath, and keep moving yourself?

    I refuse to apologize, my proposal is entirely reasonable. I will keep bringing it up frequently, perhaps every week or more, because I think it’s a provocative and potentially useful idea.

  6. Nick Tarleton

    Conventional weapons can be addressed with conventional and nonconventional means. One very simple method would be to restrict the availability of the elements used in the creation of explosives to relatively low levels. There’s not much you can do with pure hydrocarbon explosives

    Don’t forget O and H. My concern is not with explosives in particular, but the implications of self-replication for war manufacturing in general, as explained e.g. in Nanotechnology and the World System.

    but those leave specific traces which can be addressed in semi-nonconventional methods; such as a variation on Nanoshield that reacts to a certain density of standard compounds…

    Yes, but Nanoshield has its own share of obvious, severe political problems.

  7. Michael,
    I respect your thinking on this issue and agree that we need to be thinking about the risks of various future and near-future technologies, but I think you underestimate the effect of a frightened public. We have a long history of panic in the face of what ultimately prove to be minor risks (Alar, DDT). You have no idea what kind of reaction would be evoked if a demonstration such as you propose actually scared people. It would be far worse than the demonstration effect of Hiroshima and would drive research underground rather than leading to a reasoned discussion of risk/reward assessments.
    You idea is excellent in the abstract and potentially disastrous in practice.

  8. Then conduct the demonstration in such a way that the public doesn’t get the importance, but most scientists do.

    Look, now wasn’t the previous sentence a decent idea? I never would have had it if I hadn’t proposed the broad idea in the first place.

  9. Matt Grimes

    I’m afraid this line of thinking is counter-productive to your stated goal of creating awareness of the “potential danger” of self-replicating technology. The goal is noble and deserves some serious thought but getting to the point you suggest equates to developing and demonstrating a military-grade weapons system. If the technology is developed to the point where you can actually pull of this desert island demonstration that technology should be highly classified and adequately guarded. It’s not exactly the sort of thing you want the terrorist organizations getting their hands on.

    I suspect you would be getting visits from several of the three-letter govt agencies long before you got close to executing your plan.

    Personally, I think that once the technology is demonstrable (in a non-destructive format) that there will be no shortage of journalists willing to provide media coverage. And ultimately, that’s how the populace will get their spin on it; from the media. So perhaps a better question is how can you systematically prime the media to make your case as we progress down the development cycle.

  10. John Hunt

    Actually, it seems that everyone in principle agrees with the concept:
    – “the idea is worth serious consideration and effort”,
    – “It’s a nice idea”,
    – “I respect your thinking on this issue”, and
    – “The goal is noble and deserves some serious thought”

    But nearly everyone shares the same concern about public reaction:
    – “people are already *very* afraid of the technology”,
    – “I doubt the general public would respond in the correct fashion”, and
    – “you underestimate the effect of a frightened public”

    The perceived consequences of an unhealthy public reaction are varied:
    – stopping the technologic improvement of humanity,
    – public engaging in conspiracy theories,
    – driving research underground (with less oversight?), and
    – terrorists getting their hands on the technology

    So it seems to me that the obvious thing to do is to figure out if the concept could be carried forward in such a way that:
    – the public reaction wouldn’t lead to legislation:
    – stopping the development of effective defenses,
    – preventing survival strategies from being developed,
    – stopping the technologic improvement of humanity,
    – terrorists wouldn’t get their hands on the technology

    Yet the demonstration would still need to be powerful enough to:
    – engage more scientists in giving serious consideration to existential risks (xRisks).
    There’s potentially many different paths to xRisks we need a lot of good minds working on this problem.
    – drive research funding to adress technological xRisks
    – drive development funding to construct survival strategies (e.g. underground ark, lunar base)

    Here are my suggestions:

    1) Do everything above board. There is nothing like doing things in secret to cause the public to suspect nefarious motivations and to overreact. Emphasize that we want to be very transparent about everything we do so that the public will support us as we develop strategies to counter and survive the risks.

    2) Invite and appreciate concern. The people who are concerned about the risks get the point. If they are concerned about what we’re doing then we can expect their support for the development of prevention and survival strategies.

    3) Avoid aligning one’s self with political entites such as the administration, the UN, some “internationalist” foundation (e.g. the Rockefeller Foundation), or even a big corporation. Doing so might fuel opposition from unreasonable people.

    4) Conduct a demonstration which effectively illustrates what an xRisk would look like but itself is unable to destroy humanity if it escaped quarentine. Also, such technology wouldn’t be useful to terrorists.

    Doing #4 might be sufficient to get private funding even if it doesn’t generate enough public interest to prompt government funding.

    If doing #4 fails to generate interest and funding then one could follow-up with a more serious but still contained demonstration.

  11. John Hunt says: “4) Conduct a demonstration which effectively illustrates what an xRisk would look like but itself is unable to destroy humanity if it escaped quarentine. Also, such technology wouldn’t be useful to terrorists.”

    that sounds like a good idea to me. Perhaps one could *combine* a large scale mock-up of the effects of a bioweapon with a small scale, quarantined demonstration that these effects were realistic. Some rich philanthropist could fund this, it should be do-able for a few hundred thousand dollars. Perhaps this could be the lifeboat foundation “killer app”?

  12. Matt,

    With imagination, one can come up with alternatives that are not actually deadly if they escape quarantine, but still demonstrate the point. For instance, the idea of a self-replicator that only eats the inside of eggs. That’s not military grade, now is it?

    The technology would be inherently destructive (it would need to be to make the point), but conceivably this destruction could take place in a container the size of a breadbox, if necessary.

    The idea here is to propose the general concept, then consider spinning it to the wider public, but I forgot that there are so many people here who are used to having all their information input already pre-spinned, that they have trouble looking at an abstract concept from several different angles, in such a way that some of those angles would be acceptable to the public and the gov’t, and it was a good idea to bring up the general concept after all.


    Thanks for the analysis and overview. Your suggestions totally make sense.


    I would estimate at *least* a million would be necessary (I figured we are talking about qualitatively new technologies, like synthetic biology, not just genetic engineering), but yeah. Maybe it could be the LF killer app!

  13. Warren Bonesteel

    “And when they’ve given you their all
    Some stagger and fall, after all it’s not easy
    Banging your heart against some mad bugger’s wall.”

  14. L

    Micheal, when you say things like:”Look, now wasn’t the previous sentence a decent idea? I never would have had it if I hadn’t proposed the broad idea in the first place.”

    It makes me wonder whether you
    a) ‘only’ proposed this radical idea to provoke the debate, which I would appreciate, even if you should’ve made it clear earlier…
    b) try turn this into a harmless brainstorming once you realized what kind of opposition you’re facing
    c) you just wanted to brainstorm all along, but have been un- thought/tact- ful in posting your idea initially…

    So which one, if any, is it?
    before I go on critizing any further, I’d like to give you the chance to clarify.

  15. I wanted to brainstorm, and consider that my proposal was appropriate all along. Military planners wouldn’t have been offended by my idea. Some people are too sensitive.

    The opposition reminds me of the people that called me “Hitler” when I posted my six places to nuke article.

  16. L

    I would like to not use “nazi”-accusations in a rational discussion, they usually fail any comparison.

    Sensitive or not, I fail to see how (potential) approval of military planners can sanction this, or anything really.
    Military planners, I assume, were also in favor of testing nuclear weapons. Yet all this has brought us was a nuclear arms race, with no signs of stopping. No country is honestly disarming their nukes, let’s be serious.
    Or can you see a way of how nuclear weapons tests have lowered the risk for global extinction?

  17. This is so great to see. We have do more things like this for the military families. They are protecting us and we need to support them in this way.

  18. Hi i attempt to spread out your blog within safari and its appears humorous we tink that the issue is out of your hosting or maybe from me but nonetheless youve got a good set up for the advertisements i composing in this article because you will see this when youre grading comments Continue the good function Andrei through Romania

  19. Hi there! This is kind of off topic but I need some guidance from an established blog. Is it tough to set up your own blog? I’m not very techincal but I can figure things out pretty fast. I’m thinking about setting up my own but I’m not sure where to begin. Do you have any points or suggestions? Thank you

  20. Hi I found this webpage from mixx Its not something I would normally read but I liked your perspective on it Thanks for creating something worth

  21. How do you keep blogger from counting your page views on your blog?

  22. mm it appears like your site ate my first comment (it was super long) so I guess I’ll just sum it up what I submitted and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to everything. Do you have any helpful hints for inexperienced blog writers? I’d definitely appreciate it.

  23. Thanks for an amazing publish, can examine your particular others content. thank you your ideas for this, I felt a bit made an impact to by this short article. Merit again! You wanna make a good time. Has great report here. I believe that in case a greater number of people thought about it like that, they’d have a very better time frame receive the grasp ofing the issue.

  24. Hello there! Do you know if they make any plugins to safeguard against hackers? I’m kinda paranoid about losing everything I’ve worked hard on. Any suggestions?

  25. Found your report quite appealing in fact. I actually loved looking at it and you also make really some great factors. I am going to bookmark this web site for your future! Relly great write-up.

  26. Found your post pretty intriguing indeed. I really loved reading it so you make quite some fantastic points. I am going to bookmark this web page for your upcoming! Relly wonderful post.

  27. Found your article really intriguing in fact. I truly really enjoyed browsing it and you simply make pretty some superior points. I am going to bookmark this web-site for that long run! Relly terrific content.

  28. Identified your write-up quite exciting indeed. I truly enjoyed studying it and you simply make pretty some good factors. I am going to bookmark this website for that long term! Relly excellent post.

  29. This reminds me of something I saw a while back. Good stuff!

Trackbacks for this post

  1. But don’t be bullshit» Blog Archive » I’m Skeptical about Robotic Nanotech

Leave a Comment

Awesome! You've decided to leave a comment. Please keep in mind that comments are moderated.

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>