Accelerating Future

Say that the world’s foremost microbiology expert announced that he or she was working on a synthetic virus with the ability to wipe out all human life. Or that a space-capable country, say, the United States, decided that its new mission is to locate the largest possible near-Earth asteroid and alter its trajectory so it impacts the Earth. Or that the world’s foremost expert on machine learning decided that he is tired of humanity and wants to create an AGI that shreds us all to pieces, to put us out of our own misery. (Yes, the technology to make these risks real may not exist now, but imagine 10 or more years down the line — the near future.)

I’d want to be able to say, “So what? Go ahead. You’ll never succeed, we have safeguards against that in place, and they’ve been extensively tested.”

But I can’t, and we’re always in danger, and will continue to be until safeguards are in place.

A key difference between me and many others who think about extinction risk is that I wouldn’t be unpleasantly surprised at a Doomsday Announcement by a leading scientist — surprised as in I’d feel sad about the announcement, but I would have accounted for the possibility in advance. I’d say, “yes, it’s sad you have to be this way, but I knew all along that we needed safeguards in place that would account for this eventuality”.

Of course, the available resources at my disposal are far less than is necessary to put such safeguards in place. However, it could potentially be done for lower cost than one might think. By fueling our watchguard systems with superintelligence rather than human intelligence, they might actually work, rather than failing when the guy watching the cameras goes to take a leak (”human error”). Recursively self-improving superintelligence might be accessible merely through a well-funded seed AI or human intelligence enhancement effort. A few million or even a few thousand dollars can go a long way here.

Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it? Say “we need to optimistic in life, we can’t always worry about risks, lol”? The problem with that outlook is that it causes you to die.

13 Responses »

In inbox-land, the place where emails happen, I have received a piece of “electronic mail”. This e-mail comes from Peter Voss, Founder & CEO of A2I2, a company formed in December 2001 with the “express goal of developing and commercializing an effective general intelligence software engine”. I’ve been following the company since it came into existence, because hey, making claims about AGI is a big deal. Here’s the email:

“Dear friends of A2I2,

We are nearing the commercialization phase of our project.

In the past many of you have expressed a desire to be involved in some way, and a few of you have helped us in various ways (thank you).

At this stage we could use assistance in four areas:

1) Help us identify a business that could serve as a pilot site.
2) Help us find a high-powered CEO to help with our commercial division.
3) Help us brainstorm various business issues — i.e., provide seasoned business advice.
4) Help us test our technology — no technical skills required!

If you feel that you are willing and able to help us, then send a short email introducing yourself to mail at adaptiveai dot com.

Please note that we require you to sign an NDA (non-disclosure).

Peter Voss and Tas Dienes”

Well, here’s your chance to get involved in an AGI company. I have no idea what they’re cooking up, or what it will do, but you can find more information on the company at their site. Peter Voss has said previously that AGI might be possible within a very short time, just 5-10 years. I’m skeptical, but Voss is no kook, so it’s worth at least considering what he has to say.

3 Responses »

Convergence 08, our new unconference, will be held at the Computer History Museum in Mountain View, CA on November 15-16. Early bird registration (available until Oct. 20) is cheap, just $70 for students and $145 for general. After Oct. 20, the price goes to $120 for students and $190 for non-students. Buy tickets now, get conversations with smart people later.

Convergence 08 is co-sponsored by humanity+ (formerly called the WTA), the Singularity Institute, Immortality Institute, Foresight Nanotech Institute, Methuselah Foundation, and the Long Now Foundation. Keynote will be given by futurist Paul Saffo. Because it’s an unconference, attendees will get to structure much of the conference program themselves, giving presentations on a variety of topics. People will be giving presentations simultaneously so there’ll be no boredom — you can wander from talk to talk and stick to your favorite. This conference is being marketed as the first and only forum dedicated to NBIC (nano-bio-info-cogno) technologies.

The day before the conference there is also a seminar on extinction risks, led by the Institute for Ethics and Emerging Technologies (IEET) and the Center for Responsible Nanotechnology (CRN). The text for this academic seminar is the recently-released Global Catastrophic Risks.

Help make this conference a success by showing up and impressing us with how awesome and knowledgable you are. Squeeze extra value out by chatting up and taking photos with heavyweights like Ben Goertzel, Aubrey de Grey, Barney Pell, Nick Bostrom, James Hughes, and let’s not forget, that other guy, me.

No Responses »

As I see it, there are three main categories of risk: bio, nano, and AI/robotics. These man-made risks make up the vast majority of the threat magnitude over the coming century and deserve most of the attention.

Threats of low probability include asteroid strikes, supervolcano eruptions, alien invasions, simulation getting shut down, and many others. Though there is disagreement on whether nuclear war, particle accelerator disasters, or runaway climate change deserve to be counted as substantial-probability extinction threats over the coming century, I would say they are not.

A word on focusing on low probability threats alongside higher probability threats. Mentioning low probability threats just for the sake of comprehensiveness is rhetorically damaging. It distracts from the central thrust by introducing superfluous information. Worse, it can damage credibility of the entire message. Whether fair or unfair, we have seen the doom-worriers of the Large Hadron Collider heavily maligned by both scientists and laypeople in print and online. Even if an x-risk mitigator thought there was some probability of planetary doom due to the LHC, say one in a hundred thousand, the credibility sacrifice of pushing the issue is bound to detract from one’s ability to advocate mitigation of other, much higher-probability threats. So it should be avoided. Of course, if the LHC occupies a dominant portion of the risk pie in one’s personal estimate, it would be rational to devote attention to that, despite the credibility penalty.

Regarding natural vs. artificial threats, there is a credible argument that all natural threats are of substantially low probability. We’re still here. Homonids have been around for at least two million years, despite radically inferior numbers and technology for 99.9% of that time. If our ancestors could survive natural disasters, then we’ll be able to also, with our far superior technology and numbers. Asteroids capable of causing major extinctions only strike the Earth about once every hundred million years or less. In the 600 million or so years that there has been complex multicellular life, there have only been six major extinctions, if you include the present one that humans are causing.

For the central risks that I mentioned, which can also be abbreviated GNR (genetics, nanotechnology, robotics) or GRAIN (genetics, robotics, AI, nanotechnology), I recommend the three S’s: science, standards, and security. Scientific investigation of the risks provides a sound basis for further policy. This takes actual money and work, and won’t occur automatically. Taxpayers should foot the bill. Free market incentives for self-regulation are not enough. Industries have an incentive to downplay the magnitude of risk for short-term gain. I say this as a capitalist and advocate of science and progress. (In our polarized political climate, such disclaimers are unfortunately mandatory.)

After science comes standards. All of an industry, say the nanotechnology industry, or the synthetic biology community, needs to come up with some basic set of safety rules, both for individual workers and for the effects of their industry on the planet and environment as a whole. Examples of industry standards are too numerous to list. How much government involvement should be included in the approach will vary depending on your political philosophy. Too much meddling will cripple an industry and encourage clandestine workarounds, and too little meddling may cause an industry to adopt a “no rules” policy that maximizes profits while ignoring risk. If your libertarian philosophy causes an industry to pursue dangerous practices that increase global risk, then your philosophy has failed to adapt to the dangers of the future. If your interventionist philosophy causes an industry to become frustrated and transfer their operations to another country with no rules, then you’ve failed again. Insofar as it’s possible, discard your context-insensitive political beliefs and adopt context-sensitive, non-partisan approaches to these new challenges. Only then will enough people actually agree with you that the approach is adopted and makes a difference.

After standards comes security. The standards have to actually be enforced, or they are useless. If dangerous genetically engineered microbes are not kept under lock and key, unsavory individuals may get ahold of them and use them to fulfill nefarious ends. Security measures will be bolstered by transparency and increasing surveillance and sousveillance, a natural consequence when you combine human curiosity and cheaper/smaller cameras. Local and global agencies have to cooperate as effectively as possible to ensure that standards are being enforced, both in private and public realms.

Those are my thoughts for today. In summary:

1) if you’re an academic, author, or journalist, write about extinction risks,
2) if you’re otherwise involved in science and technology, help create structures to manage global risk,
3) if you’re someone that makes a decent salary or otherwise has money or resources, consider contributing some of it to efforts to mitigate extinction risk,
4) if you’re anybody else, think carefully about the issue and get informed.

The goal is a world where the annual probability of risk is extremely low, approaching zero. Even if the annual probability is just one in a million, then after a million years we’re likely to destroy ourselves. It would be great if the human race and our descendants and variants lives for a long time, billions of years, colonizing the universe and living happy and fulfilling lives. Our thoughts and actions at this crucial juncture could make the difference.

3 Responses »

“I must say that this is the greatest factor — the way in which the expedition is equipped — the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order — luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time — this is called bad luck.”
– from The South Pole, by Roald Amundsen

Space stations or lunar settlements won’t help mankind avoid numerous types of extinction risks. This is because 1) any colony would remain near-completely dependent on Earth unless very large and in possession of advanced nanotechnology, and 2) the greatest danger, from superintelligence, could easily reach its long arm into space and crush any human colony if it wanted to.

This is not a challenge we can run away from. We have to stay here and fix it. Space will not swoop down and save the day.

Regarding self-replicating threats, it’s likely that a deep underground self-sufficient bunker would be nearly equivalent in its protective value to a space station, not to mention thousands of times cheaper. On Earth, there is air, organic and inorganic building materials, water, radiation shielding, proximity to other humans, and many other amenities. Even if you completely nuked the face of the planet, it would still remain the most habitable neighborhood in the solar system, hands down. This might have something to do with the fact that we descend from a lineage that has lived here and adapted to the environment for billions of years.

When dealing with extinction risks, we have to be practical, not fanciful, with visions of expensive space stations or lunar bases. That’s reality.

Continuing on with the practical viewpoint, we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. All the nukes and guns and electromagnetic pulses in the world won’t save us from something that’s fundamentally smarter than we are. The new species will merely think of everything we could come up with to fight against it and plan far in advance to counteract these threats. By the time we realize we’re under attack, it will be way too late. No non-brain-damaged human would lose a battle of wits with a Homo erectus, and no Neo sapiens or Colossus will lose a battle of wits with humans.

Accepting the threat of superintelligence involves 1) understanding that human intelligence is finite, understandable, and ultimately engineerable, just like the body (surprise!), and 2) humans are not local instantiations of some Turing-complete Godhead that intelligent species lapse into the second they’re smart enough to take over their own planet, but actually close to the dumbest that a species can be and establish a civilization. Incremental evolutionary processes don’t provide huge intelligence boosts, so Homo sapiens is just a minor tweak on what came before us, a minor tweak just good enough to launch us into the civilizational feedback loop of local dominance. A major tweak would put us into an entirely new realm, but most thinkers seem to assume that such a major tweak will just result in more entities essentially the same as us, but with bigger, bald heads, the propensity to speak in calm, authoritative language, and wear shiny silver/purple clothing. But this is just another monkey in a suit, not a new being.

Asserting with idle confidence that superintelligence won’t be here for centuries, or ever, is just another repeat of anthropocentric conceit. This is just over-worshipping intelligence like the phenomenon of heavier-than-air flight was once over-worshipped (”they’re trying to be like angels”), life was over-worshipped (”humans will never be able to create life in a lab”), the Sun was over-worshipped (”mankind will never be able to harness the power that illuminates the Sun”), the division between the heavens and Earth was over-worshipped (”we’ll never fly to the Moon”), and so on. We pretend that mysteriousness is a property of the territory rather than the map, in a (sometimes subconscious) effort to protect the last segments of the natural world from being understood scientifically. Why do you think Star Wars was so popular, even among scientists? The mysterious “Force” trumped the most advanced technology in the Galaxy. In real life, technology wins, not the make-believe psychic force. Luke gets hit by a heat-seeking missile before he’s even near the Death Star. He goes boom.

But yes, let’s keep developing cybernetics, synthetic life, space travel, biotechnologies, and advanced robotics. We humans will always be on top, and when we create superintelligence, the magic of market forces and man-machine interfacing will ensure that it embodies our values. No need to panic, be alarmist, apocalyptic, or deluded. Everything will be just fine.

15 Responses »

What happened to “Singularitarian” being defined as someone who follows the Singularitarian Principles? Dr. Jones and others who contributed to the IEEE Special Issue on the Singularity mostly use the term to describe Ray Kurzweil and his fans, but for five years between 2000 and 2005 the word applied to a different, yet overlapping group with radically distinct beliefs:

- no fixed timeline
- no argument that all of history has been predeterministically building up to this point
- no argument that technological progress is slowing down, speeding up, moving sideways, or any other such specific claims
- no particular attention given to pre-transhuman intelligence technologies except insofar as they influence when and how superintelligence is created
- central focus on superintelligence as a distinct technological milestone
- acceptance of the point that deliberately designed AGI may exist before neuromorphic AGI
- acceptance of the fact that we might completely blow ourselves up before the Singularity hits
- acceptance of the fact that the first superintelligence might not give a damn about us, and just decide to rearrange our atoms into something more to its liking (like tiling the universe with instantiations of the deity Yog-Sothoth, or something equally ridiculous)
- no magical rosy scenario where human upgrades and AGI research coincidentally fuse seamlessly in a way that happens to completely benefit mankind
- acknowledgment of the Everest-sized challenge of creating AGI that doesn’t eliminate us outright, rather than hand-waving it over with “maintaining an open free-market system for incremental scientific and technological progress, in which each step is subject to market acceptance, will provide the most constructive environment for technology to embody widespread human values” (The Singularity is Near, pg 420). Yeah, right.
- etc…

“Singularitarian” used to mean making minimal assumptions: that superintelligence is possible, it could have a huge impact on the world, and our actions now may influence the final outcome. Now, it comes with a huge set of baggage that I wouldn’t wish on anybody.

15 Responses »

Insufficient resources are currently devoted to the mitigation of extinction risks. This is the argument of organizations such as the Lifeboat Foundation, the Singularity Institute, and the Oxford Future of Humanity Institute. In the web edition of the Bulletin of the Atomic Scientists, Sandberg, Matheny, and Cirkovic ask, “How can we reduce the risk of human extinction?” They offer many valuable suggestions, including expansion into space, developing secure bunkers and interdisciplinary research in quantitative risk assessment, probability theory, and technological forecasting. I’m frequently engaged in the latter, most recently by participating in a summer project funded by a grant from SIAI, and in an ongoing basis by encouraging donations to the Lifeboat Foundation.

Fundraising progress has occurred but is largely disappointing. The Future of Humanity Institute does not publish its annual budget, but judging by the staff, I’d guess their budget is about $600,000 a year, funded by James Martin. According to guidestar.org, the Singularity Institute’s revenue in 2006 was about $460,000, and the Lifeboat Foundation’s was far less, about $10,000. All these organizations could use a lot more funding, in the tens of millions.

Why is the situation so poor? My guess is not lack of wealth or will to address the problem, but lack of ideas for low-hanging fruit to pick that helps the cause. If we had better actionable ideas, individuals and foundations would be more inclined to step forward to fund them. Of course, there is also the problem of the “silly factor” regarding extinction risks, and the vagueness of tangibility of such a broad venture.

More reasons: prevention of extinction risk is not a positive goal, it’s a negative goal. People prefer to fund positive goals. Another possible reason is that many philanthropists are on the older side, and inclined to worry more about their own demise than the hypothetical extinction of the human race.

Another challenge is that the most prominent scientists that fear human extinction, Martin Rees and Stephen Hawking, haven’t done enough to address the problem. Hawking encourages development of space, but what about other measures? Little activity there. Same with Sir Rees. After a blast of activity around his recent book, little else follows. An absence of specific initiatives and public proclamations of support. Little of the entrepreneurial spirit we see in business.

My preferred initiative to counteract extinction risk is Friendly AI research, but I know that this idea is not popular to everyone in the risk mitigation community. How about research into self-sufficient closed systems, human intelligence enhancement, building safeguards into gene synthesis equipment, or pursuing other avenues that Sandberg et al present in their recent article? Perhaps a prominent scientist needs to pick one of these ideas and ask their wealthy friends to fund them. After all, it’s only our future.

6 Responses »