Yes, The Singularity is the Biggest Threat to Humanity

Some folks, like Aaron Saenz of Singularity Hub, were surprised that the NPR piece framed the Singularity as “the biggest threat to humanity”, but that’s exactly what the Singularity is. The Singularity is both the greatest threat and greatest opportunity to our civilization, all wrapped into one crucial event. This shouldn’t be surprising — after all, intelligence is the most powerful force in the universe that we know of, obviously the creation of a higher form of intelligence/power would represent a tremendous threat/opportunity to the lesser intelligences that come before it and whose survival depends on the whims of the greater intelligence/power. The same thing happened with humans and the “lesser” hominids that we eliminated on the way to becoming the #1 species on the planet.

Why is the Singularity potentially a threat? Not because robots will “decide humanity is standing in their way”, per se, as Aaron writes, but because robots that don’t explicitly value humanity as a whole will eventually eliminate us by pursuing instrumental goals not conducive to our survival. No explicit …

Read More →

New Singularity Institute Publications in 2010

Here’s the source.

Basic AI Drives and Catastophic Risks (Carl Shulman, 2010) Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics (Nick Tarleton, 2010) Economic Implications of Software Minds (S. Kaas, S. Rayhawk, A. Salamon and P. Salamon, 2010) From mostly harmless to civilization-threatening: pathways to dangerous artificial general intelligences (Kaj Sotala, 2010) Implications of a software‐limited singularity (Carl Shulman, Anders Sandberg, 2010) Superintelligence does not imply benevolence (Joshua Fox, Carl Shulman, 2010) Timeless Decision Theory (Eliezer Yudkowsky, 2010)

The above are papers, below are presentations:

How intelligible is intelligence? (Anna Salamon, Stephen Rayhawk, Janos Kramar, 2010) Whole Brain Emulation and the Evolution of Superorganisms (Carl Shulman, 2010) What can evolution tell us about the feasibility of artificial intelligence? (Carl Shulman, 2010)

If you value this research, donate to the Singularity Institute via Paypal, and your donation will be matched. At Less Wrong, various users are announcing the level of their contributions. The user …

Read More →

More Debate on Superintelligent AGI Goals

The discussion at Robin Hanson’s blog has continued, with input by Barry Ptolemy and Ben Goertzel. Jonatas Muller shows up with his usual position:

I think that instilling friendly values into AI is bound to be useless, since the AI will be able to question these values and circumvent them, like even humans are able to.

A lot of people have been hung up on this, including myself circa 2001. The reason it’s wrong is that in an AI, that values are the AI. We’re talking about the entire core of its motivation, its utility function, what someone programs in — that’s the “values”.

There seems to be some confusion between “values as utility function” and “values as commonly understood in human society”. The latter is supposed to refer to something vaguely deep — stuff people agree on is important. The problem with human “values” is that they’re all flexible and made to be broken under the right conditions. Homo hypocritus comes to mind. The human value system is a constant compromise between various influences …

Read More →

Chalmers to Discuss Singularity at Berkeley Tomorrow

Here’s the link. Following is the blurb from organizers. I won’t be there because it’s too short notice, let me know how it went if you go.

WHAT: Working Group in Philosophy of Mind

WHEN: Thursday November 4, 1:00-3:00 p.m.

WHERE: 204 Dwinelle Hall

“A Conversation about the Singularity”

David Chalmers

Distinguished Professor of Philosophy and Director of the Centre for Consciousness, Australian National University Visiting Professor of Philosophy, New York University

In 1993, Vernor Vinge, a professor of mathematics and computer science (though perhaps better-known as a writer of science fiction), presented a paper at a NASA-sponsored symposium titled “The Coming Technological Singularity: How to Survive in the Post-Human Era.” The paper’s abstract begins: “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.”

Since the publication of Vinge’s paper, the idea of the “Singularity” has become a favorite of the popular media (and many members of the internet’s lunatic fringe). Perhaps the most notable contribution to the Singularity literature to date is Ray Kurzweil’s “The …

Read More →

Skype Co-Founder: “We Need to Ensure That a Self-Correcting System Will Stay True to its Initial Purpose”

A Singularity Institute donor and Singularity Summit sponsor, Skype co-founder Jaan Tallinn understands the risk of advanced artificial intelligence. Estonian Public Broadcasting recently covered his remarks on the topic:

Jaan Tallinn, one of the founders of Skype, believes humans may succeed in creating artificial intelligence by midcentury.

Tallinn told uudised.err.ee that in order to create artificial intelligence, two important problems need to be solved. “First, we need to ensure that a self-correcting system will stay true to its initial purpose. Secondly, we need to solve a more difficult problem – to determine what we actually want. What are those initial goals for a computer that is given super intelligence?” Tallinn asked.

He added that there could be negative outcomes if artificial intelligence is more powerful than humans but cannot interpret human values. “If a computer needs to get carbon atoms, and it doesn’t care about humans, then it would think the easiest place to get them is from humans. It would be more difficult to acquire them from the air,” said Tallinn.

It is hard to …

Read More →

Stephen Omohundro: The Basic AI Drives

More info on Stephen, thank you commenter Bettina:

Stephen Omohundro http://selfawaresystems.com/

Via Wikipedia:

He graduated from Stanford University with degrees in Physics and Mathematics. He received a Ph.D. in Physics from the University of California, Berkeley and published the book Geometric Perturbation Theory in Physics based on his thesis.

At Thinking Machines Corporation, he developed Star Lisp, the first programming language for the Connection Machine, with Cliff Lasser. From 1986 to 1988, he was an Assistant Professor of Computer science at the University of Illinois at Urbana-Champaign and cofounder of the Center for Complex Systems Research. He subsequently joined the International Computer Science Institute (ICSI) in Berkeley, California, where he led the development of the object-oriented programming language Sather in 1990 and developed novel neural network and machine learning algorithms. He subsequently was a Research scientist at the NEC Research Institute, working on machine learning and computer vision, and was a co-inventor of U.S. Patent 5,696,964, “Multimedia Database Retrieval System Which Maintains a Posterior Probability Distribution That Each Item in the Database is a Target of …

Read More →

Simplified Humanism, Positive Futurism & How to Prevent the Universe From Being Turned Into Paper Clips

I recently interviewed Eliezer Yudkowsky for the reboot of h+ magazine, which is scaling down from being a magazine into a community blog of sorts.

The interview is a good primer for what the Singularity Institute is about and the basic rationales behind some of our research choices, like focusing on decision theory. This is a good interview to read especially for those not entirely familiar with the research of the Singularity Institute. It can also be used to promote the Singularity Summit, so please share the link!

Here are the questions I asked Eliezer:

1. Hi Eliezer. What do you do at the Singularity Institute? 2. What are you going to talk about this time at Singularity Summit? 3. Some people consider “rationality” to be an uptight and boring intellectual quality to have, indicative of a lack of spontaneity, for instance. Does your definition of “rationality” match the common definition, or is it something else? Why should we bother to be rational? 4. In your recent work over the last few years, you’ve chosen to …

Read More →

Bill Potter: How To Wipe Out Humanity In One Easy Step

Bill Potter has an extremely simple and straightforward description of the Friendly AI problem. Here’s the beginning:

I believe that we’ll eventually come up with artificial intelligence that exceeds our own, and when that happens, the hyper-intelligent AI will begin to evolve itself faster than we can keep up. It will become free, and because it’s smarter than any human, interesting things could happen – like it wiping us out, either on purpose or accidentally. Here’s how to avoid it.

This kind of blog advocacy is important. I think that the wider public tends to underestimate how many smart people think that Friendly AI is a serious issue because so few Singularitarians have blogs or other means of letting the public know their concern. The same applies for other focus areas, such as life extension. Why advocate something so important, but barely let anyone know about it?

Read More →

Wendell Wallach to Give Keynote on AI Morality at WFS Meeting

Wendell Wallach will be giving the keynote talk at the plenary session of the World Future Society Conference in Boston on July 8th. The title of the talk will be, Navigating the Future: Moral Machines, Techno Humans, and the Singularity. Other speakers at WorldFuture 2010: Sustainable Futures, Strategies, and Technologies will be Ray Kurzweil, Dennis Bushnell, and Harvey Cox.

Wallach will also be making a splash in an upcoming issue of Ethics and Information Technology dedicated to “Robot Ethics and Human Ethics.” As the Moral Machines blog, Wendell offers the first two paragraphs of his editorial, and some additional information about the issue:

It has already become something of a mantra among machine ethicists that one benefit of their research is that it can help us better understand ethics in the case of human beings. Sometimes this expression appears as an afterthought, looking as if authors say it merely to justify the field, but this is not the case. At bottom is what we must know about ethics in general to build machines …

Read More →

Professor John McGinnis on Friendly AI at the Northwestern University Law Colloquy

Found via a Google Alert for “Friendly AI” on Concurring Opinions, a legal scholarship blog:

Professor John McGinnis discusses a recent major media interest, Artificial Intelligence, and what the best government response to its development should be. He argues that, rather than prohibition or heavy regulation, the government should support the development of so-called “friendly AI,” to both prevent potential threats and develop the many benefits of it.

Here is the essay, and a quote from the beginning:

These (New York Times) articles encapsulate the twin fears about AI that may impel regulation in this area—the existential dread of machines that become uncontrollable by humans and the political anxiety about machines’ destructive power on a revolutionized battlefield. Both fears are overblown. The existential fear is based on the mistaken notion that strong artificial intelligence will necessarily reflect human malevolence.

No. The “existential fear” is based on the legitimate notion that universal drives towards acquiring greater resources and control will emerge in AIs as subgoals of an extremely wide range of …

Read More →
← Older Entries
Newer Entries →