The Three Singularity Schools, Kurzweil, and Superintelligence
So, there's been some interesting debate lately about Singularity University, which George Dvorsky has kindly summarized for us here. I'm not going to weigh in on that debate, because I think that a list of the academic tracks isn't enough to pass judgement, and that we actually have to wait for the course materials (which, according to David Orban from personal communication with Ray Kurzweil, will be released under a Creative Commons Attribution license) to say anything meaningful. Otherwise, I think that early reactions to the idea are mostly based on one's prior opinion of Kurzweil's stuff rather than reacting to anything genuinely new.
The point of this article is to remind the reader that there are three schools of Singularity thought -- this is so fundamental, but so few people are aware of it. It should be the first thing that people learn when introduced to the concept. As I argued in 2007, the word "Singularity" has lost all meaning, but if we're stuck with it, we should at least pull apart three of the major meanings it tends to have. (Though the number of meanings it has in practice are almost unlimited, generated by Silicon Valley socialite types who are trying to look cool but only know about the Singularity from a few short blurbs on places like CNET.) The three schools are Accelerating Change, Event Horizon, and Intelligence Explosion. The full talk on the subject is available here.
When I was introduced to the Accelerating Change school by reading The Age of Spiritual Machines at age 16, I thought the concept had a lot of explanatory power. I still do. However, Mr. Kurzweil's presentation of the idea gives it an aura of inevitability that is misleading, for instance predicting AGI in precisely the year 2029, and a rupture in the fabric of our understanding in 2045. He addresses the concept of existential risk at length in The Singularity is Near, curtailing the implications of inevitability, but his less thoughtful fans have a tendency to miss this.
When I bought this domain in 2003, I was still very excited about the Accelerating Change school of the Singularity, hence the name "accelerating future". Since then, I've become more moderate in my enthusiasm for the idea. One problem is that the value of different varieties of technological advancement, or even technological advancement in general, are highly subjective. So even if a given technological metric is advancing at a loose exponential, this is not impressive to someone who sees linear practical returns from that particular advancing metric. The highly quantitative nature of the Accelerating Change analysis is also especially likely to provoke and alienate those averse to technological determinism. Still, I think the world is far more technologically deterministic than most humanities types would like to believe.
In the end, I find the Event Horizon and Intelligence Explosion schools of the Singularity more acutely relevant to our future than the Accelerating Change analysis. These other schools point to the unique transformative power of superintelligence as a discrete technological milestone. Is technology speeding up, slowing down, staying still, or moving sideways? Doesn't matter -- the creation of superintelligence would have a huge impact no matter what the rest of technology is doing. To me, the relevance of a given technology to humanity's future is largely determined by whether it contributes to the creation of superintelligence or not, and if so, whether it contributes to the creation of friendly or unfriendly superintelligence. The rest is just decoration.
Take space colonization for example. Does it matter to the future of humanity if we spend billions of dollars on building space stations and missions to the Moon? Only insofar as it influences how and if superintelligence is created, and as far as I can tell, it doesn't. That exclusivity about superintelligence has caused some to question my sanity, but that's the same reaction I would expect if I were an intelligent Homo habilis in an alternate universe ranting about how developments in technology were only relevant insofar as they gave us the ability to produce Homo sapiens or something even smarter. People are so preoccupied by the impact of humans that they fail to realize that the creation of transhumans would sideline much of our ongoing impact in the global sense.
That's the thing about superintelligence that so offends human sensibilities. Its creation would mean that we're no longer the primary force of influence on our world or light cone. Its funny how people then make the non sequitur that our lack of primacy would immediately mean our subjugation or general unhappiness. This comes from thousands of years of cultural experience of tribes constantly killing each other. Fortunately, superintelligence need not have the crude Darwinian psychology of every organism crafted by biological evolution, so such assumptions do not hold in all cases. Of course, superintelligence might be created with just that selfish psychology, in which case we would likely be destroyed before we even knew what happened. Prolonged wars between beings of qualitatively different processing speeds and intelligence levels is science fiction, not reality.
It's interesting that quotes which were originally fielded to back up the Intelligence Explosion school find themselves repurposed in The Singularity is Near to argue for the Accelerating Change school. There is actually somewhat of a merger between the two schools in the book, to the point where one might find it difficult to disentangle them, and condemn one idea for the flaws of the other. For instance, on page 10 of the book is a quote by myself, from a 2003 interview with Phil Bowermaster. The quote goes, "When the first transhuman intelligence is created and launches itself into recursive self-improvement, a fundamental discontinuity is likely to occur, the likes of which I can't even begin to predict." Well, that quote is at odds with Mr. Kurzweil's presentation in the rest of the book, which argues that AI will surpass the human brain around 2029 but the rupture of predictability won't occur until 2045. According to my quote, that rupture would occur almost instantaneously.
A related quote by Eliezer Yudkowsky appears on page 35: "Our sole responsibility is to produce something smarter than we are; any problems beyond that are not ours to solve... [T]here are no hard problems, only problems that are hard to a certain level of intelligence. Move the smallest bit upwards [in level of intelligence], and some problems will suddenly move from "impossible" to "obvious". Move a substantial degree upwards, and all of them become obvious." These quotes are hammering home the point that superintelligence is the main thing, and all that tangential stuff about genetic engineering, nanotechnology, robotics, and so on, are just the inventions of merely human-level thinkers, and in the long run their impact will be measured by their contribution (or lack thereof) to superintelligence.
I hope that our differences in thinking from Mr. Kurzweil have been made clear.
February 10th, 2009 - 12:53
I’m skeptical of prophetic claims of exponential technological growth. Sure, there’s Moore’s Law. And Kurzweil is doubtless right that biotech progress will be exponential for a time as information and other technologies boost it. But there are just as many other examples of technologies that have stagnated because the economic payoff of the next iteration is not worth the cost, they are “good enough.”
I’m very skeptical that “recursive self-improvement” in intelligence is possible (like Deep Thought designing its successor!). What arguments can be made that intelligence level N could predict the requirements of N+? Rather than simply faster, etc?
It would seem to me that leaps forward in intelligence would be made the old-fashioned way, by trial and error and rigorous selection (i.e. Darwinian fitness), which is to say, by evolving.
But I haven’t read the Kurzweil books, etc, so am willing to be convinced!
February 10th, 2009 - 16:04
Singularly epic failure predicted for this venture. Not one of the Great Engineering Challenges will be solved or accelerated thanks to it. Innovation refuses to work that way. The primary outcomes of these “hey, let’s get together and invent the future”-workshops (oh, now it’s a University – how prestigious), are moist hot air, handwaving, a whole lotta carbon dumped in the form of air fuel, thick wads of cash exchanging hands …and the future arrives at exactly the same pace as it would have without the futurefest. And someone else invents it.
In other news, the event horizon of music composing has arrived: Songsmithularity is here. Any songs beyond this point are not ours to compose…
February 10th, 2009 - 16:26
The Long Nose of Innovation
http://www.businessweek.com/innovate/content/jan2008/id2008012_297369.htm
“Any technology that is going to have significant impact over the next 10 years is already at least 10 years old. That doesn’t imply that the 10-year-old technologies we might draw from are mature or that we understand their implications; rather, just the basic concept is known, or knowable to those who care to look.”
Entirely new ideas are very hard to come by; refining old ones to world-changing perfection takes a long time; just look at any Grand Technology, like phones, microchips, operating systems. No doubt, version 7 of Songsmithularity may actually kick some talented musician’s butt.
Of course, some seriously neat tech may take less time. But no less than a decade. It seems 3-5 years is the absolute minimum to get anything worthwhile done. If you’re starting from scratch, be prepared to do your homework first, adding 5-20 years.
February 10th, 2009 - 16:32
“Starting from scratch” meaning creating something entirely new, unprecedented tech. Say, someone claims an AGI leading to the Singularity is possible in less than a decade. You can’t take that seriously unless that someone’s been working on it for 10-30 years and already has some proof of concept.
February 10th, 2009 - 16:39
“Seriously neat tech” meaning recursively self-improving or somehow evolving and optimizing, kicking humans out of the loop. An example of minor seriously neat tech is an operating system that learns what you do, and does everything you’d rather not; a servant, not just a tool.
February 10th, 2009 - 17:20
Moving intelligence to a digital substrate introduces additional exponentials (a.k.a. positive feedback loops) into the accelerating change curve. Kurzweil does not take that into account, so his timeline is wrong.
The additional exponentials are driven by intelligence speedup (moves to a faster substrate) and intelligence enhancement (qualitative improvements to the mind architecture).
I think it’s still useful to look at it from an accelerating change point of view, if you plan to be enhanced rather than left behind. If you are left behind, it will look instantaneous, as you say.
February 10th, 2009 - 18:55
Excuse me but I must point out to all the singularity worshippers that the emperor is 100% nude. All this talk of self-recursive intelligence explosions is religious nonsense; our machines don’t have the intelligence of an insect, let alone a rodent or an Einstein. If someone can provide some evidence for exponential artificial intelligence growth, some theory, design or technology that gives you reason to believe it is imminent, I’ll start to take the idea seriously. Until then, Singularitarianism seems like the idle fantasizing of techno-geeks who are desperate for a religion they can believe in. No big deal, but like all religions it could be dangerous if people start taking it too seriously.
February 10th, 2009 - 20:05
@Guy Mac:
“What arguments can be made that intelligence level N could predict the requirements of N+?”
Well Einstein’s secretary might have predicted that Einstein required a cup of coffee every morning. Just because someone is smarter than you doesn’t mean they don’t need food, sustenance, human companionship etc.
Similarly with superhuman AI. It’ll need an energy source, access to pre-existing knowledge, and (if we do OUR jobs right) the friendship of human beings.
“It would seem to me that leaps forward in intelligence would be made the old-fashioned way, by trial and error and rigorous selection (i.e. Darwinian fitness), which is to say, by evolving.”
Biological evolution is a good analogy for the way technological ideas develop. An excellent book on the subject of evolution in technology, economies, and business is “The Origin of Wealth” by Eric Beinhocker. Well worth a read.
February 10th, 2009 - 21:00
Songsmith-Guy,
Yeah, I actually pretty much agree with the Long Nose argument. There may be a few technologies that pop into existence kinda quickly, though… for instance, synthesizing viruses from scratch was considered impossible a decade ago, but scientists achieved it recently, and it’s given us interesting new knowledge. And what about releases of powerful software that’s only been in the works for a few years, like new types of simulation or communication software? Or the invention of the atomic bomb? Or the Moon Race, which took under a decade? I will concede that most inventions are the result of long-term tinkering, but a famous (and significant) proportion are more sudden.
Still, I think that’s just an interesting aside. I don’t understand how Singularity advocates or Kurzweil make claims that technology needs to pop out of researchers’ minds in under a decade. Do they?
Sean,
Well, then I’ll talk to you about this again when a theory, design, or technology emerges to your satisfaction.
The validness of an idea usually doesn’t go from “meaningless” to “important” suddenly due to some discrete development. It’s either “a good idea all along”, or “a bad idea all along”, and history will prove it one way or the other.
February 11th, 2009 - 05:53
Here’s what I think each of the schools of thought:
1. The accelerating change school of the singularity is the one I find most compelling. This is because it is both logically plausible and reflects the experience humans have had of changing technologies in the past. Technologies like electronics combine with digital computer theories to develop fast computers that go on to have a major effect on other areas of development. I think the accelerating change argument is the most coherent and reasonable depiction of a technological singularity
2. The event horizon school is flakier. First, I have issues with the idea that greater-than-human intelligences are necessarily unpredictable, second, I don’t believe that raw intellectual or cognitive ability is the primary driver for technological progress, and thirdly we have seen that it is already impossible to accurately predict all the outcomes of any technological development, let alone strong AI/posthuman superintelligence so the “even horizon” idea is trivial and meaningless.
3. The intelligence explosion school is flakier still. It is based on the assumption that a sufficiently powerful general intelligence would necessarily be able to comprehend how it’s own mind works and know how to improve it. I do believe that as knowledge of the workings of the brain increases it will lead to real gains in various intellectual capacities, through nootropics, brain augmentation, or through brain simulation on faster substrates. Gaining additional knowledge about the brain doesn’t require us to be “smarter.”
With reference to the last point: the knowledge of how the brain works will be gained through trial-and-error scientific experimentation and ongoing technological development of brain-scanning technology (itself developed by trial-and-error technological tinkering), surgery (surgery itself was again also developed through the inductive tinkering of the barber surgeons), and neural interface technology (which is being tinkered with as I write).
February 11th, 2009 - 06:09
I’m not saying they’re saying “under a decade” (except for one fairly credible researcher http://ieet.org/index.php/IEET/more/goertzel20080823/ – credible because he seems to at least have the necessary time put into his work) but the tone of their communication is that they’re going to solve some major, grand, large scale, high impact problems, just by putting a bunch of smart people in the same room and letting the CO2 levels rise. And do that in a relative hurry. You may ask “Where does it say so?” Don’t know if it does explicitly, but it doesn’t say: “Going to bear fruit 20-30 years from now”. You can’t sell that. There always has to be this feeling of immediacy, communicated between the lines, that the future is just around the corner to get people excited and open up their wallets.
Whereas, in the real world, what actually Actually happens, anything worthwhile usually takes a whole Lifetime to come to grips with, let alone do something about it that hasn’t been done before.
The easy stuff is already done. Problems we’re faced with solving today are Deep. 10 weeks doesn’t have that quality. Not even a year. A decade is a good start.
What good is it to put a group of relative strangers in a room? Cross-pollination? Synergy? For that time they only manage to become a bunch of useless specialists, deprived of their research tools and teams. You can rattle off a number of such “innovation acceleration” terms and tout the blessings of group effort, but it won’t change the fact that history is replete with stories of epic, world-changing breakthroughs that are the result of an SDI – Single Dedicated Individual – or a small team led by one. Groups of disparate people that get together for 10 weeks to discuss? Yeah, I guess there are a few examples of those too, but key individuals played a crucial part. Lose them, and the success of the project goes down the tubes. And they surely weren’t accelerated by anything like SU.
These kinds of schemes have a built-in fail. They’re architecturally incapable of achieving what they claim. So what’s effective? Use the funds wasted on this non-starter to fund individual researchers/teams of important and promising tech. Massively fund massively important and promising tech. Perhaps that’s what they actually do with the money.
The only good outcome I expect is that it makes the issues more known and that may increase funding.
Perhaps, if all the smart, key people who actually have a potential to make a difference attend the next 20 sessions, you *might* see something come out of it, but until then all we’re likely to see is an ever increasing corpus of futurebabble.
February 11th, 2009 - 07:36
Despite my apparent pessimism and negativity, I eagerly await its launch and hope for the best. There’s room for at least one Singularity University on planet Earth. Perhaps even two. If it doesn’t solve anything, who cares. It’s not exactly the most unworthwhile things humans engage in. I hope to watch videos of the proceedings; a get-together of smart people never fails to entertain.
February 12th, 2009 - 01:52
“Does it matter to the future of humanity if we spend billions of dollars on building space stations and missions to the Moon? Only insofar as it influences how and if superintelligence is created, and as far as I can tell, it doesn’t.”
I would be hesitant to look at the value of all technological advancements through the lens of contributing or not contributing directly to super intelligence. We shouldn’t assume such a thing is just around the corner and in the meantime we need to be concerned with existential risk and the preservation of our species.
Space travel, for instance, then becomes of paramount importance in ensuring the long term survival of humanity in the face of localized catastrophe so that one day we do indeed produce a super intelligence.
February 14th, 2009 - 11:36
“3. The intelligence explosion school is flakier still. It is based on the assumption that a sufficiently powerful general intelligence would necessarily be able to comprehend how it’s own mind works and know how to improve it. I do believe that as knowledge of the workings of the brain increases it will lead to real gains in various intellectual capacities, through nootropics, brain augmentation, or through brain simulation on faster substrates. Gaining additional knowledge about the brain doesn’t require us to be “smarter.†”
One place that accelerating computation is applicable for augmentation is with genetic algorithms and genetic programming. Not that a mind will simply emerge out of the combinatorial universe, but if there is a basis for initial design, then the evolutionary processes should be able to find ‘better’ models. Things that might currently be just out of read using these methodologies may become acquirable with every 10 fold increase in computational power.
Note that massive paralization, as the industry is presuing now, is directly applicable to the use of genetic algorithms and programming, and has a one to one match in terms of increasing performance/ decreasing time.
The big issue though is that its hard to ensure that any such ‘evolved’ solutions remain benevolent.
February 14th, 2009 - 11:38
Apologies:
This sentence:
“Things that might currently be just out of read using these methodologies may become acquirable with every 10 fold increase in computational power.”
should be
Things that might currently be just out of reach using these methodologies may become acquirable with every 10 fold increase in computational power.
March 31st, 2009 - 23:59
“3. The intelligence explosion school is flakier still. It is based on the assumption that a sufficiently powerful general intelligence would necessarily be able to comprehend how it’s own mind works and know how to improve it. [...]â€
There is a relatively straightforward way to use additional computational power to enhance economic and scientific output once you have uploads. You just run additional copies of relevant researchers. If you have 1000x increase in computation power, you run 1000 times faster, 1000 more copies or a combination thereof. It lets you leverage exponential advances in computing power to gain exponential increases in output.
You may quibble that this is not enhancing a single individual’s intelligence. However, groups are already smarter than individuals (if properly organized), so it seems to me that this would be functionally indistinguishable from our point of view.