Vernor Vinge’s Latest Take on the Singularity
Vernor Vinge has an interesting and somewhat unique take on the Singularity, ironic because all the spinoffs are based on his original definition. However, I regularly disagree with some of his points.
One of the points he frequently makes is that a hard takeoff (superintelligence nearly overnight) would necessarily be bad. I disagree -- there are likely to be bad hard takeoffs, and good hard takeoffs. If the superintelligence in question actually cares about human beings, then surely its "hard takeoff" could be orchestrated in such a manner that everyone benefits and no one has their life "flip turned upside down". On the other side of the coin, if the superintelligence didn't give a damn about human beings, then we'd likely have our constituent atoms rearranged into something it considers more "interesting", like a cosmic whiteboard for its beloved mathematical equations.
Favoring a hard or soft takeoff is not like picking between chocolate and vanilla ice cream. Instead of being based on a matter of human preference, it's likely that objective facts about the structure of cognition will dictate how quickly an AI or intelligence-enhanced human would be capable of improving its own intelligence and directing it towards the achievement of real-world goals. These facts include: how smart humans are relative to what's possible, how easy it is to use an abstract theory of intelligence to implement concrete improvements, what sorts of knowledge are necessary to implement these improvements, and so on. Though a soft takeoff may be possible, I tend to focus on the hard takeoff possibility, because it's the primary scenario you can benefit by preparing for in advance. Given a slow takeoff, there is a longer window of opportunity to guide circumstances towards beneficial ends.
So, check out this table I threw together:
If there's a soft takeoff, preparation was probably less crucial all along, though it is still very likely to be helpful. If there's a hard takeoff, preparation was probably necessary, and if you didn't put in the necessary effort (say, because there wasn't any immediate monetary payoff), then you and the rest of mankind could be terminally screwed. By "preparation" here I mean setting the initial conditions of the intelligence explosion directly, either by picking who to test out the intelligence-enhancement machine on or by programming the AI that actually grows up to be the first superintelligence. Anything else, like stockpiling canned goods in your basement, is pretty useless.
Another problem I have with Vinge in this video is that he initially implies that it's impossible to prepare in advance if the Singularity is a hard takeoff. Well, no. The long-term behavior of a superintelligence could very well depend on its initial conditions. Superintelligence derived from an AI programmed just to pick stocks might be less sympathetic to our human plight than a superintelligence derived from an AI programmed specifically with philosophical and moral issues in mind. Though he claims early in the video that it would be useless to prepare for a hard takeoff, near the end he brings up the analogy of children and says that if we are wise in the way we build smarter-than-human intelligence, we might be doing ourselves a favor. This is a welcome chance of emphasis in his positions, as in past years he has largely neglected the possibility that humans might be able to nudge the Singularity in more pleasant directions by manipulating the initial conditions.
I get a weird feeling from all this Singularity coverage by IEEE. Did they cover the topic because they think it might actually happen, or because it's just the hip new thing that all the intellectuals are talking about? Probably the latter, but I can't be sure.
H/t to Bob Mottram for the video.
August 7th, 2008 - 16:02
Preparations/initial conditions – these could be tested and fined-tuned “off-line” with a “Singularity Simulator” – a hypothetical, self-modifying program designed to model the beginning stages of a potential Singularity.
If the Simulator (its variants and sets of initial conditions) tosses out FAI code consistently, we know there might be a problem ahead. On the other hand, if it enhances initial FAI functionality, we have reason to be assured that an actual Singularity implementation, based on the modeled design, will also be FAI, past the range of simulations.
August 7th, 2008 - 16:48
Yes, this is a good idea in principle (something similar is called “wisdom tournaments” in CFAI), but unfortunately, a perfect simulator would be a Singularity itself, and if it messed up, we’d all die.
August 7th, 2008 - 17:43
Agreed, a bunch of “caged” Singularity Simulator test runs can only provide assurance without certainty. We’d have to go on faith that FAI would be preserved in all future iterations of an actual Singularity system, an uncomfortable notion to engineers who like to understand the response of a system to all possible external forces and control signals.
August 7th, 2008 - 19:58
“SI Jail”: http://www.sl4.org/archive/0106/1647.html
August 7th, 2008 - 21:51
“Hard Takeoff: The Singularity scenario in which a mind makes the transition from prehuman or human-equivalent intelligence to strong transhumanity or superintelligence over the course of days or hours.” Yes, I’m sure that we will figure out many possible ways in which we could convince a newborn vast alien intelligence with atomically precise control over matter expanding at the speed of a chemical explosion to not convert us all into fuel and diamondoid circuitry in the window of somewhere between 10 minutes and say, 6 days. The odds of a hard takeoff being healthy for anyone is about the same as the odds of randomly shooting a shotgun into a crowd of people turning out to be a good thing because you shot their life threatening tumour out. Now if you will excuse me, I have an overly optimistic blog to unsubscribe from.
August 7th, 2008 - 22:01
How about an AI that only absorbs computing resources from deep underground and in limited quantities, because it actually cares about what happens to humans? It wouldn’t take much to become vastly more intelligent and capable than all six billion humans.
Intelligence enhanced humans could potentially spark hard takeoffs too. What would you do if you were the mind at the center of a hard takeoff? Kill everyone? Let yourself become someone who doesn’t care about others?
Re: unsubscribe, if you really unsubscribed then why did you come back and read this? Is it really “unsubscribing” if you come back to check out the comments? Note how it is impossible for me to lose here, because if you’re reading this, I’ve already won. ;D
August 7th, 2008 - 23:26
I said I was unsubscribing, I didn’t say I wasn’t going to wait to see if you responded, so no, you actually didn’t win. In fact, you’ve basically pointed out exactly how lacking your critical thinking skills are. Nothing says “top notch futurist thinker” like a badly constructed “gotcha”. Oh, and if you delete these comments, I win. Gotcha.
August 8th, 2008 - 02:45
I’ve got to agree with binary-number-guy. It’s more likely for smart people to hold less intelligent people in contempt than otherwise.
This is my justification for not being vegetarian: I look at two different cows and compare their lives. Are they really distinguishable? They might be on physically different farms, but to what extent is there new and varied life in their separate existences? From my perspective there is none: a cow’s life is a cow’s life, they’re all equal.
To a super AI will all human lives look the same? Will there really be substantial and interesting differences meriting our continued existence? Or would 6 billion humans simply seem redundant?
August 8th, 2008 - 03:48
I never claimed to be a top notch futurist thinker. I am whatever you want me to be. As long as you keep giving me attention, your “unsubscription” means little. I only delete comments that are really poorly thought out. All you’ve demonstrated so far is that you’re capable of blurting out a position without bothering to defend it. And that your IP indicates you’re from Canada.
But yeah, your model of a hard takeoff doesn’t take into account the mind’s preferences. It just assumes it blindly expands outward without regard for others, which implies the physical absence of cognitive structures associated with concern for humans. If these structures were present, the outcome would be different.
Jordan, binary-number-guy didn’t even say his reason for thinking that hard takeoffs kill humans. He might have a completely different reason than the one you present.
August 8th, 2008 - 04:43
Jordan: This is my justification for not being vegetarian: I look at two different cows and compare their lives. Are they really distinguishable? They might be on physically different farms, but to what extent is there new and varied life in their separate existences? From my perspective there is none: a cow’s life is a cow’s life, they’re all equal.
To a super AI will all human lives look the same? Will there really be substantial and interesting differences meriting our continued existence? Or would 6 billion humans simply seem redundant?
– I suspect that six billion human lives would seem rather highly redundant, and I don’t think it takes superintelligence to realize this. But the solution to this may not be to delete most of the humans. The solution may involve letting the humans transition into posthumans. Six billion posthumans may not be redundant, because posthumans would be able to occupy a wider variety of complex states of existence.
A lot of people make the comment that a very smart, powerful agent (e.g. an AI computer program) would delete humanity and re-use our atoms for something else, the justification being that we would be very simple, boring creatures.
But the more I think about it, the less certain I am that this is true. For if we are extremely simple compared to an AI, we will not take very much computer time to keep alive in a simulated world. Why delete information which might be useful and is extremely cheap to keep?
I am aware that people will accuse me of arguing in favor of this outcome because I have a personal stake in it, and I am doing what I can to minimize that bias.
August 8th, 2008 - 05:21
The Singularity is happening right now. The Singularity is YouTube, Wikipedia, Google, etc.
August 8th, 2008 - 09:18
Ok, let me put it this way. When we as a society develop the ability to consistently create people that actually care about what happens to humans, then maybe we can start thinking about trying to pass that wisdom on to our machine friends. The point is the psychology of any superhuman AI is ultimately unknowable until we experience it for ourselves. They need to be kept in the bottle, virtual spaces, without the ability to affect or observe the physical world. It doesn’t even require any nanotech whatsoever. If the simple ascii text that I’ve posted here is enough to manipulate you into acting as immature as you did in your first response just imagine what a superhuman intelligence could make you do with something as simple as access to the internet or an outside phone line. Everything we have done as a society is the result of a very simple instruction set that consists of, essentially; Persist, Consume, Reproduce. There is no way for us to predict the psychology of something smarter than us, but yeah, I’m sure that by controlling the starting conditions we can guarantee that whatever results from it will want to give us hugs and devote the large portion of it’s thought to whether or not we want it to build us jet-packs and space elevators. That isn’t naive at all.
August 8th, 2008 - 09:22
To sum up, the only superhuman intelligences that we should ever allow access to the physical world are the ones that we ourselves must become. Even that has no guarantees, but at least we would be starting from a position that can be related to.
August 12th, 2008 - 23:47
Be the vital part of the take off process, and you are safe!
It is also vital, that no autonomous center, independent of us, will emerge anyhow.
The hunt down for such centers should have the highest priority in the first place.
We need something like unbreakable Constitution which will always be preserved. This way we will miss something, we will be just unable to have everything, doe to those Constitution’s limitations.
But who cares?
August 12th, 2008 - 23:54
the only superhuman intelligences that we should ever allow access to the physical world are the ones that we ourselves must become.
Not us. Our Constitution, I called it The Protocol, long time ago, is the only way through what this Universe is to be changed.
You can do everything you want INSIDE the Protocol. You are not punished, but prevented to do something illegal.
Now, we only need a descent Protocol to live by.