Accelerating Future

In the secret, back-room Singularitarian mailing lists and discussion venues, we often ask: “More publicity good? Or more publicity bad? How much publicity is optimal?”

There’s no question that our cause (building safe seed AI) has more exposure now than ever. While it can be hard, if not impossible, to distinguish references to Singularity a la Kurzweil from Singularity a la I.J. Good, the two concepts are meshed together and people really do get exposure to both, even if they come away thinking that Singularity means “transhumanism” instead of “recursively self-improving superintelligence”. And the people who are really in the know can actually tell the difference. For instance, Kevin Kelly, founding editor of WIRED, recently wrote about our version of the Singularity at his blog, the Technium. When the Intel CTO mentioned the Singularity coming by 2060, he was talking about Kurzweil’s Singularity, so in my mind that doesn’t really count.

The goal is to get ourselves enough exposure to get the funding and talent we need to implement Friendly AI as quickly and safely as possible, and no more. Any additional exposure is a risk, because it increases the chance that someone with a ton of money says, “AGI, that sounds like a great idea! Good thing Isaac Asimov did all the groundwork on that friendliness issue for us, so we can just plow ahead on the intelligence part!” Then, after a successful brute force implementation, the AI develops self-replicating robotics, creates trillions of dummies that meet the definition of “human” based on its training set, and goes about spending the rest of eternity converting the universe into sock puppets and making certain to obey them. (Which is pretty easy, considering that the AI controls both the dummies and the system doing the obeying.)

The answer to the “more publicity?” question depends greatly on how hard one wagers AGI to be, or more appropriately, what your probability distribution over difficulty levels is. The people who wager that AGI is relatively “easy”, as in, requiring about a dozen brilliant programmer-theorists a la Fellowship of the Ring, along with a good ten or twenty million dollars, won’t want our cause to gain much more publicity or exposure. Those who wager AGI is extremely hard, as in requiring thousands of programmer-theorists and billions of dollars, would obviously want as much exposure as possible, as it would be necessary to reach the finish line. I fall somewhere in the middle.

On Overcoming Bias, Eliezer Yudkowsky recently observed how he thought many people in the field of AGI were simply ordinary. In my worldview, this is great. My personal experience with SIAI employees and interns indicates they are anything but ordinary. That means the “good guys” — those who make a huge deal about AI Friendliness and warn that we could all be exterminated if we mess up AGI programming — are doing better than the “bad guys” — those who just want to create AGI because it sounds like an interesting research project and are anticipating nothing more than obedient robots with IQs of 90.

But, in my view, the “good guys” still don’t have enough resources and talent, so we need more exposure. Not exposure to the general public, but targeted exposure to highly educated audiences. In a certain sense, the meme is self-filtering. Our version of the Singularity can’t be boiled down to soundbites easily. It helps to have detailed background knowledge about things like philosophy of mind, reductionism, rationality, the human tendency towards anthropocentrism, Homo economicus, evolutionary psychology, and more. Average members of the general public may stumble upon blogs like this and try to understand what I’m saying, but based on what I’ve seen, they’re likely to seize on some tiny incidental point I made and ignore the bigger picture, thereby stopping the spread of the meme in its tracks. Insofar as it makes reckless drives towards AGI less probable, that’s a good thing.

In the end, I don’t think that a million dollars a year and a dozen supergeniuses is enough. We need more resources, more talent, because the challenge of AGI is huge. It looks like the probability of success (by anyone) before 2015 is quite low, and the good guys have a significant theoretical head-start. I think we can afford (and in fact require) more exposure, until the necessary philanthropists and supergeniuses step forward. A major software project is not cheap, and taking the planning fallacy into account, things are going to take more work than we suspect. But once we reach that threshold — stop! Don’t keep plugging ahead for exposure like a mindless robot. That’s just what we’re trying to avoid, y’know?

And wait — you said there are smart bloggers out there that actually aren’t writing about this stuff?

22 Responses »

The IEET sometimes replicates blog posts of mine in their articles section. They do this because I said they could. I like it when they do that, because I think what I say is important and should be heard by more people. Otherwise I wouldn’t say it.

You can help me and other IEET fellows by caring about the IEET and what it does. There’s cool stuff there. For instance, IEET executive director James Hughes, who many of you may have heard of, recently did an interview with the Boston Globe on extinction risks. It helps when he and IEET fellows do things like that, because it lowers the chance that we all die. And not dying is awesome.

3 Responses »

Say that the world’s foremost microbiology expert announced that he or she was working on a synthetic virus with the ability to wipe out all human life. Or that a space-capable country, say, the United States, decided that its new mission is to locate the largest possible near-Earth asteroid and alter its trajectory so it impacts the Earth. Or that the world’s foremost expert on machine learning decided that he is tired of humanity and wants to create an AGI that shreds us all to pieces, to put us out of our own misery. (Yes, the technology to make these risks real may not exist now, but imagine 10 or more years down the line — the near future.)

I’d want to be able to say, “So what? Go ahead. You’ll never succeed, we have safeguards against that in place, and they’ve been extensively tested.”

But I can’t, and we’re always in danger, and will continue to be until safeguards are in place.

A key difference between me and many others who think about extinction risk is that I wouldn’t be unpleasantly surprised at a Doomsday Announcement by a leading scientist — surprised as in I’d feel sad about the announcement, but I would have accounted for the possibility in advance. I’d say, “yes, it’s sad you have to be this way, but I knew all along that we needed safeguards in place that would account for this eventuality”.

Of course, the available resources at my disposal are far less than is necessary to put such safeguards in place. However, it could potentially be done for lower cost than one might think. By fueling our watchguard systems with superintelligence rather than human intelligence, they might actually work, rather than failing when the guy watching the cameras goes to take a leak (”human error”). Recursively self-improving superintelligence might be accessible merely through a well-funded seed AI or human intelligence enhancement effort. A few million or even a few thousand dollars can go a long way here.

Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it? Say “we need to optimistic in life, we can’t always worry about risks, lol”? The problem with that outlook is that it causes you to die.

13 Responses »

In inbox-land, the place where emails happen, I have received a piece of “electronic mail”. This e-mail comes from Peter Voss, Founder & CEO of A2I2, a company formed in December 2001 with the “express goal of developing and commercializing an effective general intelligence software engine”. I’ve been following the company since it came into existence, because hey, making claims about AGI is a big deal. Here’s the email:

“Dear friends of A2I2,

We are nearing the commercialization phase of our project.

In the past many of you have expressed a desire to be involved in some way, and a few of you have helped us in various ways (thank you).

At this stage we could use assistance in four areas:

1) Help us identify a business that could serve as a pilot site.
2) Help us find a high-powered CEO to help with our commercial division.
3) Help us brainstorm various business issues — i.e., provide seasoned business advice.
4) Help us test our technology — no technical skills required!

If you feel that you are willing and able to help us, then send a short email introducing yourself to mail at adaptiveai dot com.

Please note that we require you to sign an NDA (non-disclosure).

Peter Voss and Tas Dienes”

Well, here’s your chance to get involved in an AGI company. I have no idea what they’re cooking up, or what it will do, but you can find more information on the company at their site. Peter Voss has said previously that AGI might be possible within a very short time, just 5-10 years. I’m skeptical, but Voss is no kook, so it’s worth at least considering what he has to say.

3 Responses »

Convergence 08, our new unconference, will be held at the Computer History Museum in Mountain View, CA on November 15-16. Early bird registration (available until Oct. 20) is cheap, just $70 for students and $145 for general. After Oct. 20, the price goes to $120 for students and $190 for non-students. Buy tickets now, get conversations with smart people later.

Convergence 08 is co-sponsored by humanity+ (formerly called the WTA), the Singularity Institute, Immortality Institute, Foresight Nanotech Institute, Methuselah Foundation, and the Long Now Foundation. Keynote will be given by futurist Paul Saffo. Because it’s an unconference, attendees will get to structure much of the conference program themselves, giving presentations on a variety of topics. People will be giving presentations simultaneously so there’ll be no boredom — you can wander from talk to talk and stick to your favorite. This conference is being marketed as the first and only forum dedicated to NBIC (nano-bio-info-cogno) technologies.

The day before the conference there is also a seminar on extinction risks, led by the Institute for Ethics and Emerging Technologies (IEET) and the Center for Responsible Nanotechnology (CRN). The text for this academic seminar is the recently-released Global Catastrophic Risks.

Help make this conference a success by showing up and impressing us with how awesome and knowledgable you are. Squeeze extra value out by chatting up and taking photos with heavyweights like Ben Goertzel, Aubrey de Grey, Barney Pell, Nick Bostrom, James Hughes, and let’s not forget, that other guy, me.

No Responses »

As I see it, there are three main categories of risk: bio, nano, and AI/robotics. These man-made risks make up the vast majority of the threat magnitude over the coming century and deserve most of the attention.

Threats of low probability include asteroid strikes, supervolcano eruptions, alien invasions, simulation getting shut down, and many others. Though there is disagreement on whether nuclear war, particle accelerator disasters, or runaway climate change deserve to be counted as substantial-probability extinction threats over the coming century, I would say they are not.

A word on focusing on low probability threats alongside higher probability threats. Mentioning low probability threats just for the sake of comprehensiveness is rhetorically damaging. It distracts from the central thrust by introducing superfluous information. Worse, it can damage credibility of the entire message. Whether fair or unfair, we have seen the doom-worriers of the Large Hadron Collider heavily maligned by both scientists and laypeople in print and online. Even if an x-risk mitigator thought there was some probability of planetary doom due to the LHC, say one in a hundred thousand, the credibility sacrifice of pushing the issue is bound to detract from one’s ability to advocate mitigation of other, much higher-probability threats. So it should be avoided. Of course, if the LHC occupies a dominant portion of the risk pie in one’s personal estimate, it would be rational to devote attention to that, despite the credibility penalty.

Regarding natural vs. artificial threats, there is a credible argument that all natural threats are of substantially low probability. We’re still here. Homonids have been around for at least two million years, despite radically inferior numbers and technology for 99.9% of that time. If our ancestors could survive natural disasters, then we’ll be able to also, with our far superior technology and numbers. Asteroids capable of causing major extinctions only strike the Earth about once every hundred million years or less. In the 600 million or so years that there has been complex multicellular life, there have only been six major extinctions, if you include the present one that humans are causing.

For the central risks that I mentioned, which can also be abbreviated GNR (genetics, nanotechnology, robotics) or GRAIN (genetics, robotics, AI, nanotechnology), I recommend the three S’s: science, standards, and security. Scientific investigation of the risks provides a sound basis for further policy. This takes actual money and work, and won’t occur automatically. Taxpayers should foot the bill. Free market incentives for self-regulation are not enough. Industries have an incentive to downplay the magnitude of risk for short-term gain. I say this as a capitalist and advocate of science and progress. (In our polarized political climate, such disclaimers are unfortunately mandatory.)

After science comes standards. All of an industry, say the nanotechnology industry, or the synthetic biology community, needs to come up with some basic set of safety rules, both for individual workers and for the effects of their industry on the planet and environment as a whole. Examples of industry standards are too numerous to list. How much government involvement should be included in the approach will vary depending on your political philosophy. Too much meddling will cripple an industry and encourage clandestine workarounds, and too little meddling may cause an industry to adopt a “no rules” policy that maximizes profits while ignoring risk. If your libertarian philosophy causes an industry to pursue dangerous practices that increase global risk, then your philosophy has failed to adapt to the dangers of the future. If your interventionist philosophy causes an industry to become frustrated and transfer their operations to another country with no rules, then you’ve failed again. Insofar as it’s possible, discard your context-insensitive political beliefs and adopt context-sensitive, non-partisan approaches to these new challenges. Only then will enough people actually agree with you that the approach is adopted and makes a difference.

After standards comes security. The standards have to actually be enforced, or they are useless. If dangerous genetically engineered microbes are not kept under lock and key, unsavory individuals may get ahold of them and use them to fulfill nefarious ends. Security measures will be bolstered by transparency and increasing surveillance and sousveillance, a natural consequence when you combine human curiosity and cheaper/smaller cameras. Local and global agencies have to cooperate as effectively as possible to ensure that standards are being enforced, both in private and public realms.

Those are my thoughts for today. In summary:

1) if you’re an academic, author, or journalist, write about extinction risks,
2) if you’re otherwise involved in science and technology, help create structures to manage global risk,
3) if you’re someone that makes a decent salary or otherwise has money or resources, consider contributing some of it to efforts to mitigate extinction risk,
4) if you’re anybody else, think carefully about the issue and get informed.

The goal is a world where the annual probability of risk is extremely low, approaching zero. Even if the annual probability is just one in a million, then after a million years we’re likely to destroy ourselves. It would be great if the human race and our descendants and variants lives for a long time, billions of years, colonizing the universe and living happy and fulfilling lives. Our thoughts and actions at this crucial juncture could make the difference.

3 Responses »

“I must say that this is the greatest factor — the way in which the expedition is equipped — the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order — luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time — this is called bad luck.”
– from The South Pole, by Roald Amundsen

Space stations or lunar settlements won’t help mankind avoid numerous types of extinction risks. This is because 1) any colony would remain near-completely dependent on Earth unless very large and in possession of advanced nanotechnology, and 2) the greatest danger, from superintelligence, could easily reach its long arm into space and crush any human colony if it wanted to.

This is not a challenge we can run away from. We have to stay here and fix it. Space will not swoop down and save the day.

Regarding self-replicating threats, it’s likely that a deep underground self-sufficient bunker would be nearly equivalent in its protective value to a space station, not to mention thousands of times cheaper. On Earth, there is air, organic and inorganic building materials, water, radiation shielding, proximity to other humans, and many other amenities. Even if you completely nuked the face of the planet, it would still remain the most habitable neighborhood in the solar system, hands down. This might have something to do with the fact that we descend from a lineage that has lived here and adapted to the environment for billions of years.

When dealing with extinction risks, we have to be practical, not fanciful, with visions of expensive space stations or lunar bases. That’s reality.

Continuing on with the practical viewpoint, we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. All the nukes and guns and electromagnetic pulses in the world won’t save us from something that’s fundamentally smarter than we are. The new species will merely think of everything we could come up with to fight against it and plan far in advance to counteract these threats. By the time we realize we’re under attack, it will be way too late. No non-brain-damaged human would lose a battle of wits with a Homo erectus, and no Neo sapiens or Colossus will lose a battle of wits with humans.

Accepting the threat of superintelligence involves 1) understanding that human intelligence is finite, understandable, and ultimately engineerable, just like the body (surprise!), and 2) humans are not local instantiations of some Turing-complete Godhead that intelligent species lapse into the second they’re smart enough to take over their own planet, but actually close to the dumbest that a species can be and establish a civilization. Incremental evolutionary processes don’t provide huge intelligence boosts, so Homo sapiens is just a minor tweak on what came before us, a minor tweak just good enough to launch us into the civilizational feedback loop of local dominance. A major tweak would put us into an entirely new realm, but most thinkers seem to assume that such a major tweak will just result in more entities essentially the same as us, but with bigger, bald heads, the propensity to speak in calm, authoritative language, and wear shiny silver/purple clothing. But this is just another monkey in a suit, not a new being.

Asserting with idle confidence that superintelligence won’t be here for centuries, or ever, is just another repeat of anthropocentric conceit. This is just over-worshipping intelligence like the phenomenon of heavier-than-air flight was once over-worshipped (”they’re trying to be like angels”), life was over-worshipped (”humans will never be able to create life in a lab”), the Sun was over-worshipped (”mankind will never be able to harness the power that illuminates the Sun”), the division between the heavens and Earth was over-worshipped (”we’ll never fly to the Moon”), and so on. We pretend that mysteriousness is a property of the territory rather than the map, in a (sometimes subconscious) effort to protect the last segments of the natural world from being understood scientifically. Why do you think Star Wars was so popular, even among scientists? The mysterious “Force” trumped the most advanced technology in the Galaxy. In real life, technology wins, not the make-believe psychic force. Luke gets hit by a heat-seeking missile before he’s even near the Death Star. He goes boom.

But yes, let’s keep developing cybernetics, synthetic life, space travel, biotechnologies, and advanced robotics. We humans will always be on top, and when we create superintelligence, the magic of market forces and man-machine interfacing will ensure that it embodies our values. No need to panic, be alarmist, apocalyptic, or deluded. Everything will be just fine.

15 Responses »