Accelerating Future

2 Responses »

Cross-posted from SIAI blog.

Consider two classes of AIs. One class of AIs manipulates external objects to direct the world towards a goal state, the other doesn’t. AIs with the greatest real-world impact fall into the first category. The objects may be virtual as well as physical, although they’re both ultimately the same thing, as reality is harmonious and unified.

Within the first category, there are AIs with motivations that output the open-ended, indefinite manipulation of external objects, and AIs with motivations that cause the manipulations to stop after a critical threshold of utility maximizing (or satisficing) is performed.

A CEV-AI is an example of the latter category. It extrapolates humanity’s volition, creates an optimizing process that embodies it, then shuts itself down. There’s a technical problem here–how to program it in such a way that it doesn’t attempt to turn the planet into a supercomputer to compute humanity’s volition, disintegrating humanity in the process? Some version of interim Friendliness no doubt, but remember, the CEV-AI’s primary job is to output humanity’s collective will, not be nice to humanity on a day-to-day basis. I’ll let the Friendly AI theorists try to figure that one out.

But back to the categories, I would think that most possible AIs fall into the first category: the open-ended, indefinite manipulation of external objects. In fact, most intelligences probably do. If a human life were extended to a quadrillion years, those quadrillion years would likely consist of the manipulation of external objects. Same thing if you extended the life of a chimp, or a badger indefinitely. The results might get boring pretty fast (rest, eat, sex, rest, eat, sex), but that manipulation of external objects would keep on going.

Imagine a sim-world, maybe something like that game Spore, that Will Wright thinks will change the world, with an indefinitely-living couple, be it chimp or human, living in it. Eventually their semi-random walk and offspring would encompass the world, and in the case of the human, they might even learn how to convert the world into billions of O’Neill colonies for maximum usefulness. When minds have an open-ended desire to manipulate the external world, in the long run, things never stay the same.

Because AIs would be running on accelerated substrates, the “long run” for them could be a few minutes or hours. An AI with an open-ended desire to manipulate external objects will eventually pattern over anything not to its liking, like a gardener will eventually pluck all the weeds in a garden if he has the time to do so. That’s why it’s damn important to make sure the first AI considers us, with all our flaws and imperfections, to be in alignment with its goals: if not, we’re toast in the long run, and for the AI, the long run ain’t very long at all.

9 Responses »

It’s that time of the year, when transhumanists, AI guys, and transhumanist AI guys get together under the banner of the Singularity Institute to impress or disappoint us with their knowledge of the Singularity (whatever they think it is), AI, AIs exterminating humanity, transforming Earth into a paradise, and other items on the menu. Yes, it may be only the second Singularity Summit, but it’s bound to be an awesome one, with various famous people on the list, including some new ones.

If I were to give a talk at the summit, it would probably be an explicit presentation about how hard of a takeoff I think it will be between quasi-human and superintelligent AI, i.e., a damn hard one, and maybe, if I’m fortunate, someone like Eliezer will at least hint to this in his presentation. I also like Peter T.’s hard-hitting language in the press release.

Press Release:

The Singularity Summit to address promise and peril of advanced AI to future of humanity

What are the major challenges to achieving advanced AI? What are the benefits and dangers? How far are we from self-improving AI? How should we prepare for this potentially powerful innovation?

These are among the questions that 17 outstanding thinkers will explore and debate at the Singularity Summit, to be held Saturday and Sunday, September 8-9, at the Palace of Fine Arts in San Francisco, California. The summit is organized by the Singularity Institute for Artificial Intelligence, a 501(c)(3) not-for-profit institute in Silicon Valley for the study of safe advanced AI.

“Advanced AI has the potential to impact every aspect of human life. We are in a crucial window of opportunity where we have temporary but powerful leverage to influence the outcome,” said Tyler Emerson, chair of the summit and executive director of the Singularity Institute for Artificial Intelligence. “Only a small group of scientists are aware of the central issues. It is essential to expand discussion of this critical 21st century issue, which is why I have created the summit.”

Tickets can be purchased online for $50 at http://www.singinst.org/summit2007/tickets/.

Peter Thiel, PayPal Cofounder, Clarium Capital President, and Facebook’s initial investor, will MC the summit and present his new ideas on financial markets and the Singularity. “It’s clear that the term ‘AI’ means a lot of different things,” said Thiel. “It’s one of these terms that has been bandied about a great deal, and has been misused a lot. It has been predicted for a long time that AI is right around the corner, and it’s taking longer than many people thought it would, with many disappointments along the way. However, it’s clear that there’s a massive set of issues happening, and people who don’t think there’s something important going on are living in a fantasy, and need to wake up.”

Confirmed Summit speakers include:

* Dr. Rodney Brooks, famous MIT roboticist and founder of iRobot
* Dr. Peter Norvig, director of research at Google
* Paul Saffo, Stanford, leading technology forecaster
* Sam Adams, distinguished engineer within IBM’s Research Division
* Jamais Cascio, cofounder of World Changing and creator of Open the Future
* Dr. Ben Goertzel, director of research at SIAI and founder of Novamente
* Dr. J. Storrs Hall, author of Beyond AI: Creating the Conscience of the Machine
* Dr. Charles L. Harper, Jr., senior VP at John Templeton Foundation
* Dr. James Hughes, executive director of Institute for Ethics and Emerging Technologies
* Neil Jacobstein, prominent AI expert and CEO of Teknowledge
* Dr. Stephen Omohundro, founder of Self-Aware Systems
* Dr. Barney Pell, founder and CEO of Powerset
* Christine Peterson, cofounder of Foresight Nanotech Institute
* Peter Thiel, cofounder of PayPal and founder of Clarium Capital
* Wendell Wallach, author of Machine Morality: From Aristotle to Asimov and Beyond
* Eliezer Yudkowsky, Friendly AI pioneer and cofounder of SIAI
* Peter Voss, founder and CEO of Adaptive Artificial Intelligence

“To any thoughtful person, the Singularity idea, even if it seems wild, raises a gigantic, swirling cloud of profound and vital questions about humanity and the powerful technologies it is producing,” said Douglas R. Hofstadter at last year’s Singularity Summit at Stanford, author of Gödel, Escher, Bach, which won a Pulitzer Prize in 1980. “Given this mysterious and rapidly approaching cloud, there can be no doubt that the time has come for the scientific and technological community to seriously try to figure out what is on humanity’s collective horizon. Not to do so would be hugely irresponsible.”

About the Singularity Institute for Artificial Intelligence (SIAI):

SIAI is a 501(c)(3) not-for-profit institute in Palo Alto, California, with three long-term goals: furthering the nascent science of safe advanced AI (self-improving cognitive systems) through research and development, research fellowships, research grants, and science education; furthering the understanding of its implications to society through educational outreach, such as the annual Singularity Summit; and furthering education among students to foster an interdisciplinary field for the study of safe advanced AI. Learn more by visiting SIAI at http://www.singinst.org.

4 Responses »

So there was a rumor, possibly false (although the linked page builds on a mention from a scientist that was actually attending the event), that a Dr. Pianka wanted to kill 90% of humanity with ebola. His statement is here. The gist of his statement is that he actually just wouldn’t mind if humanity is killed by ebola, but wouldn’t actively pursue it. Still, the point is that there could be evil supergeniuses among us. (Read his statement for more detail.)

This issue reminds me of an acronym that Phillippe Van Nedervelde used in his talk on existential risks and the Lifeboat Foundation at Transvision 2007 — SIMAD — Single Individual, Massively Destructive. He also pointed to the Unabomber, and showed a picture of him when he was a math teacher, looking just like a typical professor. There is a risk from radical, out-of-control nutcases like Al Qaeda, yes, but these people tend to have problems infiltrating truly relevant organizations or acquiring the complex knowledge necessary to do real damage.

In the case of AI and synthetic biology, the biggest risks will come from smart people who have a grudge against society, and even those with noble motives but insufficient caution or sense of professional ethics. After all, if it were possible for humanity to destroy itself, it would have done so a long time ago… right? Wrong. Selection effects ensure that we will always find ourselves in a civilization that hasn’t previously destroyed it.

In the comments section of a blog I was reading yesterday, someone had this to say:

Much of the problem faced by those trying to tell us about existential risks is the fact that we’ve been bitten to hard and too long by wolf-criers for the past six years. As a result, ANYONE who talks about dangers is likely to get the cold shoulder, regardless of whether 1) they are sincere as opposed to jockeying for power, or 2) whether the risk they’re talking about is actually real or not.

This does seem true, and admonitions about global warming may be partially to blame, as well as terrorist fearmongering (some of which may also, in fact, be well-founded). Anthropogenic global warming is a reality, yes, but I don’t think it’s an existential risk, especially not in the next few decades. Bombardment with warnings on anthropogenic climate change, as well as terrorist attacks, is desensitizing the populace to warnings of existential risk. I’m not saying such warnings are a bad thing, just pointing out the fact that they’re desensitizing us. The fact that the most severe risks have to do with technologies just barely beginning to roll off the assembly lines — advanced AI and robotics, and synthetic biology — doesn’t help matters either.

But, as always, you, the reader, can refuse to be a part of the problem. You can take existential risk seriously, and refuse to write off those who discuss these dangers, like Martin Rees and Stephen Hawking, as “Doomsayers”. For most of the past 10,000 years, catastrophic technological risk has been impossible. Even global thermonuclear war would be more likely to kill off 10% or 20% of the population rather than 99% or 100%. And if you care about the long-term future of humankind as a whole, killing a billion and killing everyone makes a hell of a lot of difference.

(Other good posts on this domain: Concept Funneling, Rapture of the Nerds, Not.)

20 Responses »