Vernor Vinge has an interesting and somewhat unique take on the Singularity, ironic because all the spinoffs are based on his original definition. However, I regularly disagree with some of his points.

One of the points he frequently makes is that a hard takeoff (superintelligence nearly overnight) would necessarily be bad. I disagree — there are likely to be bad hard takeoffs, and good hard takeoffs. If the superintelligence in question actually cares about human beings, then surely its “hard takeoff” could be orchestrated in such a manner that everyone benefits and no one has their life “flip turned upside down”. On the other side of the coin, if the superintelligence didn’t give a damn about human beings, then we’d likely have our constituent atoms rearranged into something it considers more “interesting”, like a cosmic whiteboard for its beloved mathematical equations.

Favoring a hard or soft takeoff is not like picking between chocolate and vanilla ice cream. Instead of being based on a matter of human preference, it’s likely that objective facts about the structure of cognition will dictate how quickly an AI or intelligence-enhanced human would be capable of improving its own intelligence and directing it towards the achievement of real-world goals. These facts include: how smart humans are relative to what’s possible, how easy it is to use an abstract theory of intelligence to implement concrete improvements, what sorts of knowledge are necessary to implement these improvements, and so on. Though a soft takeoff may be possible, I tend to focus on the hard takeoff possibility, because it’s the primary scenario you can benefit by preparing for in advance. Given a slow takeoff, there is a longer window of opportunity to guide circumstances towards beneficial ends.

So, check out this table I threw together:


If there’s a soft takeoff, preparation was probably less crucial all along, though it is still very likely to be helpful. If there’s a hard takeoff, preparation was probably necessary, and if you didn’t put in the necessary effort (say, because there wasn’t any immediate monetary payoff), then you and the rest of mankind could be terminally screwed. By “preparation” here I mean setting the initial conditions of the intelligence explosion directly, either by picking who to test out the intelligence-enhancement machine on or by programming the AI that actually grows up to be the first superintelligence. Anything else, like stockpiling canned goods in your basement, is pretty useless.

Another problem I have with Vinge in this video is that he initially implies that it’s impossible to prepare in advance if the Singularity is a hard takeoff. Well, no. The long-term behavior of a superintelligence could very well depend on its initial conditions. Superintelligence derived from an AI programmed just to pick stocks might be less sympathetic to our human plight than a superintelligence derived from an AI programmed specifically with philosophical and moral issues in mind. Though he claims early in the video that it would be useless to prepare for a hard takeoff, near the end he brings up the analogy of children and says that if we are wise in the way we build smarter-than-human intelligence, we might be doing ourselves a favor. This is a welcome chance of emphasis in his positions, as in past years he has largely neglected the possibility that humans might be able to nudge the Singularity in more pleasant directions by manipulating the initial conditions.

I get a weird feeling from all this Singularity coverage by IEEE. Did they cover the topic because they think it might actually happen, or because it’s just the hip new thing that all the intellectuals are talking about? Probably the latter, but I can’t be sure.

H/t to Bob Mottram for the video.