The Danger of AI Part 2 Saturday, May 9 2009
AI 9:51 pm
Two pieces of writing that greatly influence my thinking on AI are “Basic AI Drives” by Stephen Omohundro and “Levels of Organization in General Intelligence” by Eliezer Yudkowsky, particularly Part 3: Seed AI. Basically, the first argues that AIs will find convergent goals that are antagonistic to humans, and the second argues that a hard takeoff is likely. Together, they have the implication that a hard takeoff of a human-equivalent AI that is not impeccably programmed would kill us all. (Another supporting document is “Ethical Issues in Advanced Artificial Intelligence” by Dr. Nick Bostrom.) Yes, I know you may not share this view, but I seek to impart it upon you by referring you to these writings. Here they are, in large screenshot form:
Yes, that’s about it. That’s the primary argument. Another important point are Kurzweil’s books, which argue that AI is plausible by 2030 by brain scanning alone.
I am interested in support or criticisms, but hopefully criticisms that actually involve having read the documents in question, or support likewise.
May 10th, 2009 at 6:09 pm
Hi Michael:
I really like your blog. I attended Transvision 2007 in Chicago and met Eliezer & George Dvorsky. I regret I did not have an opportunity to meet you.
I believe that hard nanotechnology and AI are possible and will be extremely important, after a great deal of thought, I reject the hard ascent scenario of Yudkowsky et al.
The Yudkowsky argument, in summary, is as follows: On a cosmic scale, humans are 10% smarter than chimpanzees. Since the rise of humanity we’ve come down from the trees reshaped the Earth, understood everything except quantum gravity, and within a few centuries will reshape the solar system. Now imagine what in intelligence 2X human equvalence, or 1000x human equivalence can do. Just as a single human can out smart a troop of chimps, a single 1000x SAI can outsmart humanity. More to come in next post
May 10th, 2009 at 6:26 pm
continue from above:
To why i think this is wrong consider this scenario: Imagine that in the 1980s there is a microprosessor (chip 1.0) that is used in pc s, for word processing, spreadsheets etc. Then someone invents chip 1.1 which is 10% more powerful and has the ability to network with other chips through phone lines. The internet as well as massively parallel super computers are from assemblies of chip 1.1. Simulations of economies, molecules and other physical systems drive science forward.
Someone then is in the process of inventing chip 2.0, which is twice as powerful, and chip 1000 which is 1000 times as powerful. People argue, that the internet & supercomputing came with chips 10% more powerful than before, when these new chip 1000x come online we’ll be able to solve the protein folding problem, develop nano, cure all diseases etc.
Do you see the analogy. There is already a superintelligence on Earth that is enormously smarter than a human. That superintelligence is Human Civilization which is a network of much less intelligent nodes. In the transition into humanity, we developed a general flexible intelligence and language. This allowed the superintelligence of human civilization to evolve that understands, on a fundamental level, everything except quantum gravity.
continued below
May 10th, 2009 at 6:33 pm
continued from above.
The total computing power of the internet may support one HE AI. Suppose we build an AI thats 10x human equivalence. This may have the equivalence of a well organized team of 1000 humans. (I assume the equivalent of Amdahl’s law applies and a well organized society of minds requires more processing power that a single integrated mind to have equivalent intelligence.) continued below
May 10th, 2009 at 6:40 pm
continued from above:
So an AI that’s 10x HE will not invent nano, increase its intelligence to exp6 x HE, and take over the world in a matter of hours. The superintelligence of humanity has not yet acheived these things, so a 10X or 100X probably will not. As AIs develop the most probable trajectory is that the will be nodes in the superintelligence called human civilization.
May 10th, 2009 at 6:44 pm
This does not guarantee a positive singularity, but it is a slower ascent.
I agree that your’s & Eliezer’s emphasis on AI motivations is extremely important and must be, though I disagree about a number of the proposals.
My post does not address the speed up problem.
May 10th, 2009 at 6:53 pm
The disagreement between our positions is falsifiable. You predict that there are classes of problems that cannot be solved by well organized groups of humans given unlimitted time and resources; the equivalent to a stone arch that is only stable if built as a whole. I predict that such classes of problems don’t exist.
I support the research that you, the SAI institute and others conduct, and I think that the AI motivation problem is an important one that I would like to research. But I don’t lose sleep over it.
May 10th, 2009 at 6:57 pm
Sorry to have taken over your blog to write a manifesto, but I’ve followed your work for a while and wanted to offer my feedback.
If my ideas have already been considered,and there are counter-arguments, please refer me to the webpage or book. (As an aside, Eliezer should publish a book to argue his position explicitly)
As I said, I very much enjoy your blog. Thank you for the opportunity to post.
Gus K.
May 10th, 2009 at 7:39 pm
Teams don’t equal superintelligence. Teams aren’t more intelligent than individuals in a way that matters (you can’t distribute thinking) for fundamental technological progress. Someone has to come up with the New Idea and it’s always a single brain. The pieces must fall in place inside one neural network.
May 10th, 2009 at 9:48 pm
Hi Solitary Scientist:
Thank you for responding. Look at any major innovation and you’ll see that the inventor “stood on the shoulders of giants”. Take an innovation in physics. The physicist relies on the reliability of mathematics, which has been carefully verified and checked for errors by thousands of mathematicians over many generations. She will also rely on experimentalists who will have verified which of the numerous possible theories are supported by evidence.
May 10th, 2009 at 10:00 pm
Now you’ll say, even with that background, the physicist still has to make an invention. All inventions are combinations of other inventions. To take a trivial example, a light bulb is nothing but a bulb and a resistor in a vacuum. In deciding if an invention is non-obvious and worthy of a patent, the patent office considers whether any invention suggested to combine the prior inventions. (I’m a patent attorney by profession) There is always a finite number of prior inventions that are combined into a new one; ie the search space is finite. The finite search space can be searched by a group of communicating minds instead of a single mind.
If you look at many patent applications or research papers, there are many co-inventors/co-authors.
May 10th, 2009 at 10:05 pm
What I’m saying can be stated a different way. Inside an individual human brain different regions are in communication with other brain regions. Ideas are combined with other ideas that may reside in different brain regions to lead to novel ideas that constitute innovation. In culture, ideas are combined with ideas that happen to reside in different brains.
What constitutes and “individual” and a “team” or “group” can be somewhat arbitrary. Look at Sperry’s split brain studies where an individual becomes a team of two by the splitting of the corpus callosum.
Thanks for taking the time to respond Solitary Scientist.
gus k
May 11th, 2009 at 12:04 am
gus … you sound quite naive to me. Sorry, but your ideas are not that clever as you think.
May 11th, 2009 at 3:53 am
I never said that I was clever. I just think that the position that I’ve taken is correct. I’m not dogmatic about it. If it’s so obviously wrong, someone will have written something dispelling it a long time ago. Just point me to a webpage or book that argues the opposite position, and if its correct, I’ll change my views. I’ve not seen the position that I’ve taken directly addressed before so I think its okay to present these ideas.
May 11th, 2009 at 8:17 am
Hi Gus, thanks for your comments and your kind words about my blog.
Thomas, I’d prefer you not post comments that have nothing to say except “you’re wrong, and I don’t feel like saying why”. It doesn’t add anything to the discussion.
May 11th, 2009 at 8:57 am
This Omohundro’s abstract is quite enough to understand, that some effects of an AI process could be quite uncontrolable.
Saying that “So an AI that’s 10x HE will not invent nano, increase its intelligence to exp6 x HE, and take over the world in a matter of hours.” needs some arhuments. Gus didn’t deliver them, it’s his turn to do it. To present an argument against him (and Yudkowsky).
He didn’t.
May 11th, 2009 at 12:02 pm
Wow Michael, that was the first comment I ever had deleted.
I did not realize that a humor fill critique was verboten.
So what pushed me over the line?
a.) My non fear that chess playing computer programs are going dangerous change their programming and go all Pinky and the Brain on us?
b.) Calling Eliezer’s ideas magic seeds?
c.) Thinking it is silly to worry that a GAI comedian would come up with The Killing Joke?
The actual critique
a.) A computer program’s expertise in chess or any one rule base game or domain (especially if the “expertise” is done with brute force algorithms) does not translate to other fields or contexts
b.) The only real example of recursive self improvement in the way Eliezer talks about the concept is the collective endeavor known as Science. And that requires human beings to work. So you need a system made of already intelligent beings to get to a system that undergoes recursive self improvement. A non intelligent system will not work.
Gus,
I gave a similar critique several months ago- What we call intelligence is a social product not an individual one, and so talking about greater than human intelligence shows a misunderstanding of the nature of intelligence. But unfortunately, a Randian notion of extreme individualism holds sway with many others.
May 11th, 2009 at 12:33 pm
Jim, because your recent comment at Dale’s makes it pretty obvious that your primary interest here is trolling and not serious critique. Also, yes, I think there was too much humor/freewheeling in your comment. Not enough serious (”boring”) analysis and argument. Ideally all the comments would be intelligent, well-phrased, and sincere, sort of like what you wrote under “the actual critique”.
The problem with accepting low-quality comments (whether positive or negative — I’ve been meaning to tell Roco that some of his short positive comments don’t add anything to the discussion) is that it starts a chain reaction of quality disintegration in the comments section. This blog traditionally has had one of the finest comments sections in any blog I’ve ever seen, and I want to keep it that way. It seems as if you’ve already made up your mind, though, and your previous comment (which I deleted) was just freewheeling around and light-hearted trolling.
Why are you reading a blog written by someone whom you consider to be a bamboozled idiot (me)? I am not here to entertain you. Whenever I comment on any blog I do so with a sincere desire to forward the discussion, not play a two-faced game where I pretend to be willing to discuss when in reality I think it’s all BS.
Your comment made it pretty obvious that you just scanned the first paragraph of the paper and then commented, when I specifically asked for feedback only from people who read the whole thing. It’s really not a long paper — it takes about 20 minutes to read, if that.
May 11th, 2009 at 1:20 pm
Michael if you want to improve the quality of the blog then improve your knowledge of the topics you blog about. For example I would bet if I gave you a problem to solve from Quantum Chromodynamics you couldn’t solve it. I would also bet that if your life depended on your ability to do science you’d be dead. Further I would bet for the example of nanotechnology you read Eric Drexler’s “Engines of Creation” instead of his work of applied physics in the same field. Unfortunately you run a foul of the same problem that any non-scientist in a science field runs into, you can’t grok the technical aspects which are the most important aspects obviously.
My opinion that futurism is total garbage not withstanding this sort of uniformed futurism is purely nauseating. I would prescribe some real math, physics and engineering education for you. I would also prescribe a dose of trying to do some real science yourself. In fact on the AI topic I bet if you actually had experience dealing with AI I would imagine you would total discard Eliezer’s ideas.
On the DARPA issue if you had or ever actually (you won’t not having the qualifications to) worked with or for DARPA you would immediately delete your posts on their work.
You are an idealist (like SIAI and Eliezer…) who refuses to walk out the door and have all the idealistic wind knocked out of you. When you do decide to do that I would imagine your blog would improve 1000 orders of magnitude.
May 11th, 2009 at 3:18 pm
MoronCrusher: I don’t think Michael has ever claimed to be particularly oriented towards math and science. He writes about the issues we talk about here, and knows enough science to write about it. You seem to imply that a writer must know as much or nearly as much as a scientist in order to write science-oriented commentary. If that wasn’t the intention of your post, then I’m afraid you need to work on your arguing skills, because I really am having a hard time understanding what your point is, other than possibly to stir negative emotions.
May 11th, 2009 at 4:17 pm
Michael,
I read your blog because;
You often have interesting things to say,
I have met you, talked with you, and think that you are a pretty OK kind of guy.
But I do think that your AI fears are misguided.
I do not think you are a bamboozled idiot, I just think you are wrong about things like seed AI, and greater than human level intelligence.
As far as the Stephen Omohundro paper I did read all. But it is tough to get past this:
“Surely no harm could come from building a chess-playing robot, could it? In this paper we argue that such a robot will indeed be dangerous unless it is designed very carefully. Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety”
I mean, come on That is so silly, I will bet you any amount of money that:
1- if we connect Deep Blue running its chess program to a little robot arm that can move chess pieces on a chess board,
2-give it no special precaution
3- Deep Blue WILL NOT
A - resist being turned off,
B - Try to break into other machines
C - make copies of itself
D - try to acquire resources without regard for anyone else’s safety
The central problem with the whole article is this statement -
” Our arguments apply to any of these kinds of system as long as they are sufficiently powerful. ”
The key is what he terms “sufficiently powerful” but what he means is “sufficiently human-like, or life-like”. But very powerful computer systems don’t have to be anything like that. In way this “problem” is like the “problem” of grey goo - Not theoretically impossible, but a lot harder to do with MNT than most other useful and or dangerous activities.
May 11th, 2009 at 5:41 pm
For anyone who did not get the Monty Python reference to an AI comedian coming up with a killer joke, follow the link
http://www.youtube.com/watch?v=8gpjk_MaCGM
May 12th, 2009 at 12:20 am
Great references to make your earlier point though I remain very skeptical of Steve’s worries even though one can easily agree with most of his itemized points. They just don’t lead to the conclusion that a “free range” AI is likely to pose a threat to humanity.
With a hard takeoff it seems likely to me that any *human* efforts at making a friendly AI will be modified to obscurity within a very short time. More importantly though it seems very reasonable to assume machine AI ethics won’t diverge profoundly from the ethics humanity has developed over time. We’ve become far less ruthless and selfish in our thinking than in the past, both on an individual and collective basis. Most of the violence now rises from *irrational* approaches, not the supremely rational ones we can expect from Mr. and Mrs. AI.
Don’t worry, be happy.
May 12th, 2009 at 9:35 am
AI has been a topic of interest for me for at least a decade and this topic has been discussed for that whole time. The technology has advanced rapidly, but nowhere near the speed that many predicted. It is still plausible that AI human effects could cause a threat, but nothing that I am seeing at the moment indicates that it will happen in the next decade.
May 12th, 2009 at 7:39 pm
Thomas: My criticism is of the hard ascent scenario only, not SAI or friendliness. I have read Omohundro & generally agree; he mentions hard ascent only in passing. I agree with much of Yudkowsky as well, but I think he goes wrong in underestimating the collective intelligence of society made possible by language. This causes him to greatly underappreciate the collective intelligence of humanity and what it has achieved, and overestimate what an AI marginally smarter than an individual human is capable of. I think hard ascent is improbable in light of my arguments in the above posts. I think AI and the motivation problem (FAI) however are important, but the ascent will be slower.
May 13th, 2009 at 6:57 pm
It may just be my experience in engineering/science teams, but to me it is abundantly clear that a group of people do not come up with ideas. Individuals do (or just small parts of the individuals’ brains do). There is no thinking entity called a group. Group minds do not exist (yet). It may seem more intelligent a process as a whole when many intelligent people come up with intelligent ideas and combine them, but still the intelligence is instantiated and operates within individual brains that do not connect in any meaningful way that could be viewed as resulting in greater (or super) intelligence. You may say a team constitutes superintelligence but true superintelligence, i.e. a single integrated system performing better than what the best single integrated system (the brain) is capable of today, will have vastly different characteristics, because it actually is vastly different. You can’t increase your system’s total maximum intelligence by adding more units of the same intelligence to it. For that to happen, at least one unit has to be smarter - and understanding how to make matter behave smarter than us, that’s the 3^^^3 dollar question.
May 15th, 2009 at 2:46 am
Gus, if you posit that more intelligence is not what set humans apart from chimpanzees as much as it was language, why not acknowledge that the same paradigm shift would likely occur between humans and AI not so much due to the intelligence increase (even though that intelligence gap will deepen much faster) but due to a new increase in communication, as AIs communicate much faster and effectively through high-bandwith digital code than humans and their ineffective language that fails to transmit a full state of mind (as we all know). A hard take-off scenario doesn’t seem so unlikely to me.
May 15th, 2009 at 5:09 pm
If mind and cognition are *emergent* properties (or, if you prefer, entities), then the sheer *speed* of computation and/or communication at which the substrate upon (within?) which the mind emerges performs the (no-doubt various interacting) functions which, taken as an emergent *gestalt*, = mind HAS to be taken into account in reckoning whether a hard take-off is likely, as Herve well-articulates.
Now some, in particular J. Storrs Hall, anticipate that humans will co-evolve (via brain/computer interface and, eventually outright symbiosis) with the tech and achieve some sort of on-going, dynamic…call it something akin to homeostasis rather than equilibrium. And I sympathize with this surmise, if only because this is one of the key emerging tech trajectories. Nonetheless, again, considering the sheer *speed* at which a teletronic entity (organism???) of sufficient intelligence could potentially bootstrap its own self-enhancing development so as to rather quickly-on “soar into the wild blue yonder” (as the late Chris Evans put it, himself paraphrasing Irving J. Good), it behooves us to look into and effectively deal with the **motivation problem** as you eloquently put.
Now the motivation problem is intimately tied-in with the fact that an intelligent, cognizing system (entity, whatever) that is, from our point of view, going to *optimally* (or even satisfactorily) interact with (trans/post)humans will *have* to have an understanding and *appreciation* of humans and human motivations, fundamental goals, etc. In terms of economic theory (and the interface/overlap of psychology and economics) a civ with an AI in charge of global (in both the mathematical and geographic senses) cybernated systems is really something *new under the Sun*, in that it is difficult to conceptualize it under the rubric of either “capital” OR “labor”. What we are “rump-bumpingly” doing, as Bucky Fuller might put it, is developing and engineering an entirely new *techno-evolutionary environment* which could be the ultimate BOON, propelling us into posthuman godlike entities, or could be our inadvertant swan song, as we contribute our molecules and atoms to the entity’s “better living through femtotech” as it were.
Gus acknowledges the motivation problem. The problem of hard-takeoff depends on HOW fast, HOW rapid a threshold-sufficiently intelligent entity can self-enhance. This is a technical problem/question way-beyond my own level of even remote competence (much less expertise) to estimate—more in Omohundro and Goertzel and Yudkowsky’s (and Hall’s and Moravec’s) area(s) of competence. But I think where SIAI is coming from (correct me if I’m way-off-base here, Michael) is that *since we don’t know, and can’t very esasily estimate* the rate-of-speed of self-enhancement, once such a process ignites, it is the better-part-of-prudence-and-pragmatism to plan for a worst-case-scenario in which, once a certain intelligence-threshold is attained, whenever that may be, the self-enhancement process takes-off and proceeds (to put it mildly) rather rapidly—so rapidly that we may have little if any chance of sufficiently controlling or channelling it, and so that some sort of CORE motivational solution needs to be arrived at and put in-place, and the sooner the better, again if only prudentially-pragmatically (i.e., better to have and not [yet] need, than to NEED and not have!!).
But developing—both purely theoretically and applied-theoretically—the relevant motivational structure for an AI cybernated system is a daunting task. But—and I stress this again—I thik we need to carefully look at Adam Smith’s *Theory of Moral Sentiments* and James Otteson’s study thereof, *Adam Smith’s Marketplace of Life* to discover how HUMAN morality and normative intuitive memes and protocols *evolved* and *developed*—and thus for ways in which the AI can come to UNDERSTAND and genuinely APPRECIATE (trans)humans as autonomous entities with fundamental, intrinsic (cosmic, if you will) VALUE and WORTH. And neo-Kantian themes also come into play here (rather obviously), and the recent work of, e.g., Christine Korsgaard is very relevant and important as well.
Bottomline is: Better plan for a hard takeoff *just in case*, and to have some sort of CORE fundmental normative protocol (or, perhaps more likely, some sort of *ensemble* of [lexical?, weighted?) interacting fundamental normative protocols) in place in baby HE(+) AI so that it doesn’t intentionally OR inadvertently throw US (humanity) out with its bath-water (as it were…)
Thanks again for allowing my brain-droppings…I hope they’re at least minimally helpful…
Ciao…
Again, Michael, you rock, as do our colleagues here. Thanks for allowing yet more of my brain-droppings…
Ciao…
May 18th, 2009 at 6:41 pm
…baby HE(+) AI so that it doesn’t intentionally OR inadvertently throw US (humanity) out with its bath-water …
To me this sounds like an OK plan until I consider how likely it will be that the baby will become an advanced intellect within days and is very unlikely to kill us in that time frame (e.g. intelligent entities are generally somewhat cautious as a function of self-preservation). More importantly I’d argue any attempts we make at interfering or shaping or “caging the AI intellect” are more likely to be interpreted as hostile acts than doing nothing at all to inhibit or shape the AI. Since we have already posited in this thought experiment that the AI *may* pose an existential threat to humanity if threatened (I do agree with this point), one could argue that trying to make the AI friendly is as likely to get us killed off as just hanging around watching TV while the AI grows.
Note to unfriendly AI dawg - please remember this comment when you are zapping Michael and everybody else for messing wit ya.