Accelerating Future

“The main thing that stands between the human species and the creation of a supervirus is a sense of responsibility among individual biologists.”
– Richard Preston, The Demon in the Freezer, page 227

From a 2002 article by Danny Penman in The Guardian:

“A few months’ work in a makeshift laboratory is all it would take to produce a biological weapon that could kill hundreds of millions of people.

The scientific information is freely available and the raw materials easily sourced. The only difficult part would be mastering the necessary scientific skills, and they are taught on most biology degree courses.

One of the simplest ways of constructing a biological weapon would be to engineer an existing human disease and to make it even more lethal. Something as simple as the flu virus, when engineered with the gene for botulinum toxin, could wipe out a significant part of the human race. A low dose of this toxin is the main ingredient in cosmetic botox injections.

The genetic sequence for the toxin is freely available. This sequence could then be uploaded to a commonly available gene synthesizer, which would churn out millions of copies of the gene in a few hours. The flu virus would then be grown in the presence of this newly synthesized gene. As the virus reproduced, a few of the virus particles would absorb the gene. With a bit of luck, the budding terrorist would have produced a new biological weapon.”

Continue.

Human beings are inherently vulnerable to a significant number of lethal compounds. Given a highly contagious biological vector to distribute these compounds, the potential outcome is grim. Given adequate tools, knowledge, and time, groups could manufacture several such viruses and release them simultaneously in different areas, thwarting quarantine and antidote efforts. Governments know this — hence the Biological Weapons Convention. Bio-weapons are off-limits for warfare at the international level, but such rules might break down in the case of a large enough war, and will not be respected by rogue parties.

11 Responses »

Say that the world’s foremost microbiology expert announced that he or she was working on a synthetic virus with the ability to wipe out all human life. Or that a space-capable country, say, the United States, decided that its new mission is to locate the largest possible near-Earth asteroid and alter its trajectory so it impacts the Earth. Or that the world’s foremost expert on machine learning decided that he is tired of humanity and wants to create an AGI that shreds us all to pieces, to put us out of our own misery. (Yes, the technology to make these risks real may not exist now, but imagine 10 or more years down the line — the near future.)

I’d want to be able to say, “So what? Go ahead. You’ll never succeed, we have safeguards against that in place, and they’ve been extensively tested.”

But I can’t, and we’re always in danger, and will continue to be until safeguards are in place.

A key difference between me and many others who think about extinction risk is that I wouldn’t be unpleasantly surprised at a Doomsday Announcement by a leading scientist — surprised as in I’d feel sad about the announcement, but I would have accounted for the possibility in advance. I’d say, “yes, it’s sad you have to be this way, but I knew all along that we needed safeguards in place that would account for this eventuality”.

Of course, the available resources at my disposal are far less than is necessary to put such safeguards in place. However, it could potentially be done for lower cost than one might think. By fueling our watchguard systems with superintelligence rather than human intelligence, they might actually work, rather than failing when the guy watching the cameras goes to take a leak (”human error”). Recursively self-improving superintelligence might be accessible merely through a well-funded seed AI or human intelligence enhancement effort. A few million or even a few thousand dollars can go a long way here.

Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it? Say “we need to optimistic in life, we can’t always worry about risks, lol”? The problem with that outlook is that it causes you to die.

13 Responses »

As I see it, there are three main categories of risk: bio, nano, and AI/robotics. These man-made risks make up the vast majority of the threat magnitude over the coming century and deserve most of the attention.

Threats of low probability include asteroid strikes, supervolcano eruptions, alien invasions, simulation getting shut down, and many others. Though there is disagreement on whether nuclear war, particle accelerator disasters, or runaway climate change deserve to be counted as substantial-probability extinction threats over the coming century, I would say they are not.

A word on focusing on low probability threats alongside higher probability threats. Mentioning low probability threats just for the sake of comprehensiveness is rhetorically damaging. It distracts from the central thrust by introducing superfluous information. Worse, it can damage credibility of the entire message. Whether fair or unfair, we have seen the doom-worriers of the Large Hadron Collider heavily maligned by both scientists and laypeople in print and online. Even if an x-risk mitigator thought there was some probability of planetary doom due to the LHC, say one in a hundred thousand, the credibility sacrifice of pushing the issue is bound to detract from one’s ability to advocate mitigation of other, much higher-probability threats. So it should be avoided. Of course, if the LHC occupies a dominant portion of the risk pie in one’s personal estimate, it would be rational to devote attention to that, despite the credibility penalty.

Regarding natural vs. artificial threats, there is a credible argument that all natural threats are of substantially low probability. We’re still here. Homonids have been around for at least two million years, despite radically inferior numbers and technology for 99.9% of that time. If our ancestors could survive natural disasters, then we’ll be able to also, with our far superior technology and numbers. Asteroids capable of causing major extinctions only strike the Earth about once every hundred million years or less. In the 600 million or so years that there has been complex multicellular life, there have only been six major extinctions, if you include the present one that humans are causing.

For the central risks that I mentioned, which can also be abbreviated GNR (genetics, nanotechnology, robotics) or GRAIN (genetics, robotics, AI, nanotechnology), I recommend the three S’s: science, standards, and security. Scientific investigation of the risks provides a sound basis for further policy. This takes actual money and work, and won’t occur automatically. Taxpayers should foot the bill. Free market incentives for self-regulation are not enough. Industries have an incentive to downplay the magnitude of risk for short-term gain. I say this as a capitalist and advocate of science and progress. (In our polarized political climate, such disclaimers are unfortunately mandatory.)

After science comes standards. All of an industry, say the nanotechnology industry, or the synthetic biology community, needs to come up with some basic set of safety rules, both for individual workers and for the effects of their industry on the planet and environment as a whole. Examples of industry standards are too numerous to list. How much government involvement should be included in the approach will vary depending on your political philosophy. Too much meddling will cripple an industry and encourage clandestine workarounds, and too little meddling may cause an industry to adopt a “no rules” policy that maximizes profits while ignoring risk. If your libertarian philosophy causes an industry to pursue dangerous practices that increase global risk, then your philosophy has failed to adapt to the dangers of the future. If your interventionist philosophy causes an industry to become frustrated and transfer their operations to another country with no rules, then you’ve failed again. Insofar as it’s possible, discard your context-insensitive political beliefs and adopt context-sensitive, non-partisan approaches to these new challenges. Only then will enough people actually agree with you that the approach is adopted and makes a difference.

After standards comes security. The standards have to actually be enforced, or they are useless. If dangerous genetically engineered microbes are not kept under lock and key, unsavory individuals may get ahold of them and use them to fulfill nefarious ends. Security measures will be bolstered by transparency and increasing surveillance and sousveillance, a natural consequence when you combine human curiosity and cheaper/smaller cameras. Local and global agencies have to cooperate as effectively as possible to ensure that standards are being enforced, both in private and public realms.

Those are my thoughts for today. In summary:

1) if you’re an academic, author, or journalist, write about extinction risks,
2) if you’re otherwise involved in science and technology, help create structures to manage global risk,
3) if you’re someone that makes a decent salary or otherwise has money or resources, consider contributing some of it to efforts to mitigate extinction risk,
4) if you’re anybody else, think carefully about the issue and get informed.

The goal is a world where the annual probability of risk is extremely low, approaching zero. Even if the annual probability is just one in a million, then after a million years we’re likely to destroy ourselves. It would be great if the human race and our descendants and variants lives for a long time, billions of years, colonizing the universe and living happy and fulfilling lives. Our thoughts and actions at this crucial juncture could make the difference.

3 Responses »

“I must say that this is the greatest factor — the way in which the expedition is equipped — the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order — luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time — this is called bad luck.”
– from The South Pole, by Roald Amundsen

Space stations or lunar settlements won’t help mankind avoid numerous types of extinction risks. This is because 1) any colony would remain near-completely dependent on Earth unless very large and in possession of advanced nanotechnology, and 2) the greatest danger, from superintelligence, could easily reach its long arm into space and crush any human colony if it wanted to.

This is not a challenge we can run away from. We have to stay here and fix it. Space will not swoop down and save the day.

Regarding self-replicating threats, it’s likely that a deep underground self-sufficient bunker would be nearly equivalent in its protective value to a space station, not to mention thousands of times cheaper. On Earth, there is air, organic and inorganic building materials, water, radiation shielding, proximity to other humans, and many other amenities. Even if you completely nuked the face of the planet, it would still remain the most habitable neighborhood in the solar system, hands down. This might have something to do with the fact that we descend from a lineage that has lived here and adapted to the environment for billions of years.

When dealing with extinction risks, we have to be practical, not fanciful, with visions of expensive space stations or lunar bases. That’s reality.

Continuing on with the practical viewpoint, we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. All the nukes and guns and electromagnetic pulses in the world won’t save us from something that’s fundamentally smarter than we are. The new species will merely think of everything we could come up with to fight against it and plan far in advance to counteract these threats. By the time we realize we’re under attack, it will be way too late. No non-brain-damaged human would lose a battle of wits with a Homo erectus, and no Neo sapiens or Colossus will lose a battle of wits with humans.

Accepting the threat of superintelligence involves 1) understanding that human intelligence is finite, understandable, and ultimately engineerable, just like the body (surprise!), and 2) humans are not local instantiations of some Turing-complete Godhead that intelligent species lapse into the second they’re smart enough to take over their own planet, but actually close to the dumbest that a species can be and establish a civilization. Incremental evolutionary processes don’t provide huge intelligence boosts, so Homo sapiens is just a minor tweak on what came before us, a minor tweak just good enough to launch us into the civilizational feedback loop of local dominance. A major tweak would put us into an entirely new realm, but most thinkers seem to assume that such a major tweak will just result in more entities essentially the same as us, but with bigger, bald heads, the propensity to speak in calm, authoritative language, and wear shiny silver/purple clothing. But this is just another monkey in a suit, not a new being.

Asserting with idle confidence that superintelligence won’t be here for centuries, or ever, is just another repeat of anthropocentric conceit. This is just over-worshipping intelligence like the phenomenon of heavier-than-air flight was once over-worshipped (”they’re trying to be like angels”), life was over-worshipped (”humans will never be able to create life in a lab”), the Sun was over-worshipped (”mankind will never be able to harness the power that illuminates the Sun”), the division between the heavens and Earth was over-worshipped (”we’ll never fly to the Moon”), and so on. We pretend that mysteriousness is a property of the territory rather than the map, in a (sometimes subconscious) effort to protect the last segments of the natural world from being understood scientifically. Why do you think Star Wars was so popular, even among scientists? The mysterious “Force” trumped the most advanced technology in the Galaxy. In real life, technology wins, not the make-believe psychic force. Luke gets hit by a heat-seeking missile before he’s even near the Death Star. He goes boom.

But yes, let’s keep developing cybernetics, synthetic life, space travel, biotechnologies, and advanced robotics. We humans will always be on top, and when we create superintelligence, the magic of market forces and man-machine interfacing will ensure that it embodies our values. No need to panic, be alarmist, apocalyptic, or deluded. Everything will be just fine.

15 Responses »

Insufficient resources are currently devoted to the mitigation of extinction risks. This is the argument of organizations such as the Lifeboat Foundation, the Singularity Institute, and the Oxford Future of Humanity Institute. In the web edition of the Bulletin of the Atomic Scientists, Sandberg, Matheny, and Cirkovic ask, “How can we reduce the risk of human extinction?” They offer many valuable suggestions, including expansion into space, developing secure bunkers and interdisciplinary research in quantitative risk assessment, probability theory, and technological forecasting. I’m frequently engaged in the latter, most recently by participating in a summer project funded by a grant from SIAI, and in an ongoing basis by encouraging donations to the Lifeboat Foundation.

Fundraising progress has occurred but is largely disappointing. The Future of Humanity Institute does not publish its annual budget, but judging by the staff, I’d guess their budget is about $600,000 a year, funded by James Martin. According to guidestar.org, the Singularity Institute’s revenue in 2006 was about $460,000, and the Lifeboat Foundation’s was far less, about $10,000. All these organizations could use a lot more funding, in the tens of millions.

Why is the situation so poor? My guess is not lack of wealth or will to address the problem, but lack of ideas for low-hanging fruit to pick that helps the cause. If we had better actionable ideas, individuals and foundations would be more inclined to step forward to fund them. Of course, there is also the problem of the “silly factor” regarding extinction risks, and the vagueness of tangibility of such a broad venture.

More reasons: prevention of extinction risk is not a positive goal, it’s a negative goal. People prefer to fund positive goals. Another possible reason is that many philanthropists are on the older side, and inclined to worry more about their own demise than the hypothetical extinction of the human race.

Another challenge is that the most prominent scientists that fear human extinction, Martin Rees and Stephen Hawking, haven’t done enough to address the problem. Hawking encourages development of space, but what about other measures? Little activity there. Same with Sir Rees. After a blast of activity around his recent book, little else follows. An absence of specific initiatives and public proclamations of support. Little of the entrepreneurial spirit we see in business.

My preferred initiative to counteract extinction risk is Friendly AI research, but I know that this idea is not popular to everyone in the risk mitigation community. How about research into self-sufficient closed systems, human intelligence enhancement, building safeguards into gene synthesis equipment, or pursuing other avenues that Sandberg et al present in their recent article? Perhaps a prominent scientist needs to pick one of these ideas and ask their wealthy friends to fund them. After all, it’s only our future.

6 Responses »

On Monday I proposed that a demonstration of extremely destructive self-replicating technology may be necessary before the world takes the risk of it seriously, and starts devoting the necessary attention and money to developing comprehensive safeguards. I proposed such a demonstration very hesitantly, in the context that it may be the only way to get global attention on this very important issue, but still, some said I had “lost it”, and that such talk “polarises opinions on Transhumanism”.

Two points:

1) Prevention of global catastrophic risks is only tangentially related to transhumanism. When I was thinking about the proposal, I had nothing about transhumanism in mind. My blog tagline is “Transhumanism, AI, nanotechnology, and extinction risk”, because these are separate topics — interrelated, maybe, but separate. Sometimes I talk about just transhumanism, sometimes I talk about just nanotechnology, sometimes I talk about the relationship between the two. On Monday I was talking about just extinction risk, and what can be done to lower it.

2) Extinction risk prevention is more important than transhumanism. If we don’t survive the 21st century, then not only will the future lack cyborgs, life extension, space stations, and all that other exciting stuff, but it will also lack such mundane things as smiles, good food, jobs, picnics, writing, and just about everything else uniquely associated with the human species. Based on all the conversations and reading I’ve done, I consider humanity’s risk of wiping itself out in the next few decades to be substantial, and therefore consider it worthwhile to develop outside-the-box solutions to counteract that likelihood. Transhumanism and biological modifications are one of those fun activities you get to do if your species survives.

That absolutely includes proposals to test bioweapons in controlled environments for the purpose of determining their destructive capacity.

Take a look at the situation. Two of the greatest living scientists, Stephen Hawking and Martin Rees, have spoken out on the extreme danger of human extinction, and both the public and academia have failed to listen. If the people won’t listen to them, who else is there? Paris Hilton? Wilford Brimley? Our effort to educate the public on the dangers of advanced technology is failing, and we need solutions fast.

No one could deny the destructive power of the atomic bomb after the test at Trinity. But people have difficulty imagining new, better technologies with even more destructive potential. Discussions surrounding the regulation and control of post-nuclear weapons of mass destruction should be front and center. This should be the first topic that gets brought up at Presidential debates, and all the leading universities should have institutes devoted to studying self-replication in biological and nonbiological systems, to understand how difficult these systems are to build and how to keep them in control in the instance of accidental release.

Instead, the possibility of out-of-control self-replicators is widely seen as a joke, a science fictional plot device. I disagree: the possibility is real, so real that it could threaten human life by the year 2015 or earlier. Attention must be focused on the issue immediately. I am merely brainstorming to come up with some way of doing this.

To temper my suggestion, I’ve realized it could be done in a way more limited fashion than currently envisioned. As John Hunt remarked in the comments thread for Monday’s post, the test could be carried out in a level four containment facility, or feature something more mundane than wiping out all the life on an island, for instance “having little robots cracking all of the eggs on an small island and using their juices for fuel”.

Unfortunately, the point here is to make people afraid. We have every reason to be rationally afraid. Being rationally afraid doesn’t mean panicking, it means taking global catastrophic risk seriously. If people aren’t afraid, then either they disagree with us on technical grounds (if so, we welcome these arguments) or emotional grounds (thinking about human extinction is too unpleasant). Even though representatives of the latter group are very common, I have no sympathy for them — if ignoring the prospect of human extinction is necessary for you to feel pleasant on a daily basis, then ignore it, but tolerate the brainstorming of your more pessimistic peers on how to lower the risk.

17 Responses »

Usually, the abrupt extinction of the human species occurs in the context of a joke or fictional plot device, and is more rarely considered in serious terms. That’s why I’m pleased to see two events on extinction risk occurring this year — a day-long seminar on preventing extinction by the IEET, occurring at the Computer History Museum in Palo Alto, and a Global Catastrophic Risks conference at Oxford. The conference is 17-20 July, the seminar is on the 14th of November. If you plan to attend, I will see you at the Palo Alto seminar.

The jokey way in which most people react to extinction risk is a major obstacle to getting the world to take these possibilities seriously. Risk prevention advocates acknowledge the lacksidasical approach to x-risks in popular culture, in contrast to issues like health care, civil rights, and foreign relations. The latter issues are important, yes — but so is x-risk prevention. Leading scientists like Stephen Hawking have argued that the likelihood of humanity wiping itself out in the next century could be as high as 50%. Even if the figure is as low as 10%, it merits more attention than it gets.

That’s why I’ve come to believe that a controlled demonstration of extremely destructive weapons may ultimately be necessary to convince the world that we need to take extinction risk prevention seriously. This is a very difficult position to come to, considering both the possible PR blowback and the very real dangers inherent in developing and deploying such weapons. But I’ve come to think it may be the only way to really get prevention efforts going, on the same scale as global warming prevention or larger.

Of particular interest are weapons of mass destruction driven by chemical loops, synthetic, natural, or modified organisms, or robotics, which demonstrate an open-ended ability to self-replicate using common hosts, like humans, or materials available in the field, like dirt and sunlight. If a country’s military were able to use such a weapon to destroy all life on, say, a very small quarantined island populated only by plants and insects, then the world’s attitude towards extinction risk would turn around overnight. Poking, prodding, arguing, and intellectualizing can only go so far. At some point, people need to see it with their own eyes.

22 Responses »