Read about Kevin Kelly’s flagrant Singularity advocacy in his answer to the most recent Edge.org question, “What Will Change Everything?”:

“It is hard to imagine anything that would “change everything” as much as a cheap, powerful, ubiquitous artificial intelligence—the kind of synthetic mind that learns and improves itself. A very small amount of real intelligence embedded into an existing process would boost its effectiveness to another level. We could apply mindfulness wherever we now apply electricity. The ensuing change would be hundreds of times more disruptive to our lives than even the transforming power of electrification. We’d use artificial intelligence the same way we’ve exploited previous powers—by wasting it on seemingly silly things. Of course we’d plan to apply AI to tough research problems like curing cancer, or solving intractable math problems, but the real disruption will come from inserting wily mindfulness into vending machines, our shoes, books, tax returns, automobiles, email, and pulse meters.

This additional intelligence need not be super-human, or even human-like at all. In fact, the greatest benefit of an artificial intelligence would come from a mind that thought differently than humans, since we already have plenty of those around. The game-changer is neither how smart this AI is, nor its variety, but how ubiquitous it is. Alan Kay quips in that humans perspective is worth 80 IQ points. For an artificial intelligence, ubiquity is worth 80 IQ points. A distributed AI, embedded everywhere that electricity goes, becomes ai—a low-level background intelligence that permeates the technium, and trough this saturation morphs it.”

Kelly’s vision is a combination of watered-down Kurzweilian “everything will go alright, because I have little clue about the fragility of human value!” overoptimism about AI and old-fashioned global brain-becoming-superintelligent, Web 2.0-friendly nonsense.

These sorts of comments are alright because they marginally boost the respectability of our radical hard takeoff superintelligence vision, but they encourage complacency by saying, “don’t worry about AI friendliness, the market will solve that!” So what Kelly advocates is a form of passive singularitarianism — we sit back, the economy will continue improving, AI will be introduced as a Google-like form of software, it will grow only with feedback and dependence on the human world. It implies we can pull the plug at any time.

If not explicitly programmed with a complex and subtle goal system that gives rise to worlds we value, AI will drift into what we could call “failure modes” — runaway optimizing habits that are too simple to contain human-relevant complexity or value. Convergent, economically rational drives will enter into the picture and destroy the initial intentions of the programmers unless these intentions are phrased very specifically.

“The problem with computers is they only do what you tell them, not what you meant to tell them.” – Danielle Needle

We need active singularitarianism — where superintelligence/AI advocates realize that things could go terribly wrong if we’re not careful, and devote serious time and resources to averting that outcome.

See also Helicopt-o-bot.