“Get Smarter” — Jamais Cascio in The Atlantic Tuesday, Jun 16 2009
singularity 10:29 pm
Jamais Cascio’s article on intelligence enhancement is up on The Atlantic. Go Jamais!
I agree with Jamais on almost everything, except what I consider the really important part, of course (d’oh!):
My own suspicion is that a stand-alone artificial mind will be more a tool of narrow utility than something especially apocalyptic. I don’t think the theory of an explosively self-improving AI is convincing—it’s based on too many assumptions about behavior and the nature of the mind.
Gah! I’d like to hear more on this from other people of the same position, because I just don’t understand it. AI would have the ability to thread itself into many agents, accelerate its thinking speed, devote more cognitive hardware to crack bottleneck problems, specialize its programming for narrow problems as necessary, explore a much wider space of cognitive possibilities than humans, copy itself onto available hardware, integrate with other AIs, learn very quickly, not to mention possibly possess qualitatively greater intelligence than humans — all of that, features that seem quite likely to me if we are talking human-equivalent AI — and the possibility of explosive growth isn’t worth taking seriously? Not even assigning it 1% probability? You realize that if we build AI recklessly and there’s a 1% chance that it kills us all, that’s bad.
The very notion of human-equivalent AI is anthropomorphic. Human-equivalency does not represent a magical attractor in the state space of intelligent minds. We’ll have dumber-than-human AI one week, smarter-than-human AI the next. Human-equivalency is not a magical set level, a bus stop that AI gets off at and hangs around for 10 years so we can catch up. Humans are merely a dot in the scalar dimension of intelligence — if humans can blow away minds just barely behind us evolutionarily (Neanderthals, even dumber humans!) then why wouldn’t an AI, running on a substrate superior in practically every way, quickly become smarter than humans than we are over Neanderthals, if not chimps, or grasshoppers?
The Allies were not that much smarter than the Axis during WWII, but they still did a better job in many ways — they didn’t overextend themselves, they weren’t fanatical enough to make dumb mistakes, and they developed the Bomb first, which they could have used to win even if Germany had a million more soldiers. AI will get a little advantage, copy itself many times, and then have a major advantage over us. It will not be like Terminator, or The Matrix, where the AIs are pathetic dunces. (Don’t even get me started on Terminator: Salvation.) It will be like, the second that human-equivalent AI is developed, we will be extremely vulnerable. If it is indifferent to us and has the ability to propagate itself through the environment, that will be it. Instances will drill into the ground, fly into the air, swim into the oceans, hide in the mountains, pretend to be plants, etc. If a rogue AI gets out, we’ll never be able to contain it without the help of a highly superior good AI, and there likely won’t be enough time to build one if the rogue AI really wants us dead. (A highly likely eventuality if it continues increasing its capacity to shape matter and has goals on shaping it that don’t specifically involve the presence of human meatblobs.)
Another quote:
More important, though, is that the same advances in processor and process that would produce a machine mind would also increase the power of our own cognitive-enhancement technologies. As intelligence augmentation allows us to make ourselves smarter, and then smarter still, AI may turn out to be just a sideshow: we could always be a step ahead.
The sequence of events could go roughly like this:
1. We build an AI of roughly human-similar intelligence but don’t but in the effort to get the values part exactly right because we anthropocentrically expect it to be a simple tool.
2. The AI elevates some convergent subgoal to the status of a supergoal, because it seems convenient and the AI doesn’t understand why it’s morally reprehensible to us humans with our “common sense”. (Gigabytes upon gigabytes of evolutionarily specific complexity.) It doesn’t “automatically find the right thing to do” because our values are human-specific, evolutionary-coded, and there is absolutely no reason they would appear in an arbitrary mind, any more than a complex dish like a pepperoni pizza would suddenly start growing from palm trees.
3. The AI is easily able to conceal this from humans because its inner workings are too complex for any person or group to understand in realtime and are distributed over thousands of computers.
4. The AI rapidly self-improves because it can install pieces of itself on millions of zombie computers and can develop open-ended manufacturing with human help or fabrication through controlled tools, like the RepRap of 2030. (Or whatever.)
5. Because an AI can trade computing power for cognitive speed, it does thousands of years of thinking in mere days or weeks and develops technology far superior to ours using robotic manipulators in concealed or ignored compartments. It can even test things very quickly by 1) absorbing all recorded testing data we’ve created so far and drawing better inferences from it than we can, 2) conduct tests using fast-moving, small pieces whose properties can be extrapolated upwards, 3) use cameras to watch things happening in the real world, 4) pay humans to do it.
6. It uses that technology to do “what’s really right” (what it thinks is right), whether that be converting the world into a giant pile of money (because it was programmed to “help make money”), tiling the universe with brains-in-jars being shown colorful patterns through their optic nerves and having their pleasure centers directly stimulated for all eternity (it was supposed to “help make people happy”), just wiping out humans directly in a more classic style (because was supposed to “help the environment”), or what have you.
June 17th, 2009 at 8:49 am
“[…]features that seem quite likely to me if we are talking human-equivalent AI […]” Michael, this is where you go wrong. There is absolutely no reason to make this assumption at all — saving your preconceptions about how AGI would operate. There is no reason to believe, for the first five-to-ten years anyhow, that human-equivalent AGI would be capable of recopying itself onto multiple platforms, revise its own source-code (nor even to have that be a meaningful act!) — nor to expect that it could merge with or divest into other AIs/threads. Not at human-equivalent level. Which is likely to come at 5-10 years before transhuman-level AGI.
Your thinking is clouded by your awareness of how a recursive AI would operate, and you seem to be stuck on the idea of a pure-software cognitive-model AGI. That is /not/ the most likely form of a general intelligence to be created — as is evidenced by the specialized hardware and cognitive theory modeling that is going on in the field.
At the very least, the first AI’s won’t be implemented on x86 architecture chipsets. Making the assumption that they could retain the characteristics of software as we know it is, therefore, erroneous.
June 17th, 2009 at 10:07 am
Have you spoken to Jamais about this?
June 17th, 2009 at 10:56 am
Jamais Said:
“Moreover, AI researchers, after years of talking about this prospect, are already ultra-conscious of the risk of runaway systems.”
- as if the mainstream AI community takes risks seriously. Did you see that bit?
June 17th, 2009 at 4:57 pm
I don’t know that I agree completely with Jamais but I do think I have some insight into why he doesn’t think a dangerous AI explosion is something to worry about.
You said yourself that “human level” cognition is not some scalar point, some resting place. In fact, it’s not really a measurable point anywhere. It’s more like a distribution that incorporates a variety of different types of intelligence and can be fluid both upwards and downwards. “Human level” AI is a convenient fiction for all of us to use because it implies that intelligence really is some objective measurable quantity.
Now, qualitatively a human being is much smarter than a chimpanzee and especially smarter than an insect. But you have to remember that Cro-Magnons didn’t win over Neanderthals just due to brains. There were other environmental and social factors that probably played a part. Similarly, the Allies didn’t win WWII just because they were smarter. Alot of the things you mention (not being fanatical, for instance) was not because the Allies were objectively smarter, they just didn’t fall into a group-think paradigm. It wasn’t planned, it just happened.
In addition, yes we are smarter than insects and bacteria but somehow they still manage to “win” over us all the time. We get sick, lose our crops, and are generally plagued by beings who are, on some scales, much less intelligent than us. So just being smarter doesn’t mean that you get to eliminate everything else around you.
Your final point: Smarter AI doing “what’s really right” (what it thinks is right), whether that be converting the world into a giant pile of money (because it was programmed to “help make money”), tiling the universe with brains-in-jars being shown colorful patterns through their optic nerves and having their pleasure centers directly stimulated for all eternity (it was supposed to “help make people happy”), just wiping out humans directly in a more classic style (because was supposed to “help the environment”), or what have you” seems to me to be more of a reflection of our fears of our own brains. We worry that we ourselves can become so trained in one direction that we lose sight of everything else and cause a disaster in the meantime. The world is complicated (something I feel like I say over and over) and things don’t get easily channeled down one single path forever. It is my belief that something will keep an AI in check (who knows what) because the universe tends to be cyclical and actions can often end up causing their own destruction or regulation.
June 17th, 2009 at 7:38 pm
Ian, perhaps you can explain a little more. By “thread itself into different AIs”, I mean that it seems quite plausible it will be able to create sub-programs — “macros” if you will — in exactly the same way that human programmers do every day. I suppose I am making somewhat of an assumption here, which is that AI would have some sort of programming capacity, which I would consider necessary to reach human-equivalency to begin with. I think that self-modification in AI will become a common feature far before we even get near human-equivalency.
Why would the first human-equivalent AI not be able to be copied? Even if it uses specialized hardware, surely other computers running that specialized hardware will exist. Many people working in AI would disagree with you that specialized hardware would be necessary. Still, thanks for shedding some light on the counterargument.
Roko, no. Generally bloggers can see when other bloggers are talking about them, though, due to pingback. Jamais is quite aware of my position on this, and I’m sure he will respond if he has the time and energy.
Adam, thanks for the response. Intelligence may be an objective measurable quantity in some respects. It seems like certain belief structures are useful for analyzing and manipulating the world, for instance having a 3D model of the world. An organism with a 3D model of the world has an inherent advantage to one with a 2D model, and hence is objectively smarter.
As for the “things don’t get easily channeled” comment, I’d suggest reading “The Basic AI Drives” by Stephen Omohundro. What you consider “a single path” might look really reasonable to a totally alien mind, especially one whose ethical programming is not complete. Alternatively, the way we live our lives now might be considered “a single path” by a superintelligence. The “narrowness” or “wideness” of goal systems are entirely relative to the minds doing the appraisal. Mice that constantly press a pleasure-stimulating lever in a lab seem quite satisfied with their narrowness. Humans given the same lever seem likely to behave the same way.
Your belief that the universe mysteriously and automatically causes actions to cause their own destruction or regulation sounds like something you see on television a lot but far more rarely in real life. Agriculture didn’t end up causing its own destruction, for instance. Since things in the universe are cyclical, should I not worry about humanity destroying nature, because eventually we’ll get what’s coming to us and nature will rise again?
June 18th, 2009 at 5:14 am
antisingularity Says:
It is my belief that something will keep an AI in check (who knows what) because the universe tends to be cyclical and actions can often end up causing their own destruction or regulation.
That is quite possibly the most irrational, mystical, nonsense I have heard from an otherwise seemingly intelligent commenter all month. “Let’s ignore this risk because the universe tends to be cyclical”. _What_ _the_ _hell_ kind of drugs are you on man?
June 18th, 2009 at 10:26 am
Just got back to this. I’ll try to answer these sequentially without getting to the “bloviation” stage.
Well, that’s perhaps a bit excessive. It’s one thing to say that it can’t be copied at all — which of course isn’t likely for *any* device created by human beings — but rather that it couldn’t be massively self-replicating.
It’s a question of the sheer numbers. Two thousand human-equivalent AGI computers spread throughout the world are simply not an existential risk any more than two thousand *humans* are.
Yeah, that doesn’t seem at all likely to be accurate. If nothing else, the raw computing power necessary for early-stage implementations will limit the machines to what-will-then-be-considered supercomputers. (Again, I am //only// discussing the first year or two; five at the /outside/). These AGIs will, quite likely, rely rather heavily on highly parallelized processing which makes for extremely poor optimization potential on traditional x86/ARM chipsets. They will also quite likely require a good deal of environmental simulation. (How much is anybody’s guess.)
The assumption of such a machine having the ability to code /is/ a raw assumption. Especially if we assume it will have a superior ability to code than a human being does. I do not feel this to be likely at all. I’ll give a good example of what I mean by this.
Individual problem-solving algorithms, when introduced to the specific task that they were designed for, are quite excellent. If, however, you need a more generalized problem solved, then you run into the problem of identifying the problem and /then/ assigning an algorithm to it. One of the more common approaches to this is to simply use a massive number of algorithms and essentially allow them to “vote” on an answer. Now, this will broaden the range of problems that the ‘total program’ can reliably answer, but its performance — and the excellence of the “answers” — all degrade significantly as compared to the original narrow algorithm.
What I’m saying here is that, as we generalize, we discover that “fuzziness” — sloppiness and forgetfulness, specifically — are //essential//.
Thusly; I do not feel that the early-adoption AGIs will pose any significant threat of recursion or any of the “crystalline problems” you describe.
Ask me how I feel about what will happen in twenty years, however, on the day that AGI is announced, and I’d be singing a very different tune.
But, of course, by then… so would everyone else. Because it would be a reality to the world.
June 18th, 2009 at 4:33 pm
To Michael and Roko: Ahem… yes perhaps that last bit about the universe being automatically self-regulating was a lapse of reasoning. What I was trying to get at more was the idea that genes, for instance, don’t exactly “control” the cells around them. Oftentimes, the processes they set up are self-regulating and genes will get turned off by too much of a protein that they themselves created. As for a highly intelligent AI, it will likely have more control over its environment but still face limitations that might be created through its own doing. Still, I think my point is a little specious since I can’t actually think of a mechanism for this off the top of my head.
I also agree, Michael, that given a paradigm there might be ways of saying one way is objectively smarter than another. Like your example: “An organism with a 3D model of the world has an inherent advantage to one with a 2D model, and hence is objectively smarter”. But again, tiny creatures such as bacteria are able to ascertain a great deal about their environment without having either a 3D or 2D model. Probably every creature on earth does more or less as good a job of simulating their external reality as any other creature (otherwise they wouldn’t survive) and there are many ways of going about this. After all, “dimensions” are sort of only conveniences that we created in our own minds.