7Aug/1012
Eliezer Yudkowsky and Robert Wright on Bloggingheads
Tom McCabe also has an interview with Eliezer up on KurzweilAI.net. It makes a reference to my favorite email that we've received at SIAI:
This entire site is the biggest load of navel gazing stupidity I have ever seen. You are so naive, and clueless as to the inherent evil that lurks forever. A machine is no match for Satan.
August 7th, 2010 - 20:07
I have a serious question about the values that Eliezer and the singularity institute want to impart on the self-improving AI they are planning to build. Eliezer’s plan to simply tell the AI to analyze human brains and figure out what they really wanted if they were as intelligent and knowledgeable as the AI itself doesn’t convince me. Isn’t it possible that a self-improving AI capable of rewriting its own source code and extending its own hardware could do serious damage well before it acquires the capability to analyze human brains, let alone before it’s intelligent enough to make optimal decisions? Judging from the video above, I don’t think that Eliezer has really thought this through. Maybe someone more knowledgeable than me could clarify this for me.
August 9th, 2010 - 06:36
Toma, to answer your question, yes, it could do plenty of damage. Has he (Eliezer) thought this through? Of course, he’s thought about it, but if he had thought it through to the point where he’d be completely satisfied with his answer, then that would be a Friendly AI design right there.
I’m probably not going to watch this video for a while, but I can get back to you when I do watch it. It’s important to emphasize that Friendly AI is a nascent field with only a small handful of researchers, and it’s just getting started. No one has all the answers — that’s why we’re proposing more research.
August 7th, 2010 - 20:21
Are you saying that a machine IS a match for Satan? Because that is what I am hearing you say.
:)
August 8th, 2010 - 08:01
about whom I once wrote a column.
what a clown.
August 8th, 2010 - 11:37
I dunno. The prospect of having a machine figure out what I really, really want sounds awfully paternalistic. How would one question the machine’s judgement? How would one be sure the machine wasn’t lying?
Also, figuring out what people really really want would probably involve simulating lots of human minds and running them through many different scenarios. That means lots of experimental guinea-pigs. Yech. No thanks.
August 8th, 2010 - 13:16
I think we’ll get AIs that understand our intentions long before we get AIs powerful enough to reconfigure galaxies. Dogs and small children are intelligent enough to understand (and obey) human intentions; but far from able to take over the universe. Eliezer gets credit for identifying one failure mode: the AI wants the human to push its reward button and learns to obey the human’s intent to get the button pushed; but its real motivation is to get the button pushed. When its powerful enough it eliminates the human and pushes its own button. From there he goes to the unjustified conclusion that this is a gargantuan problem that cannot possibly be solved by evolutionary learning. I suspect we can evolve programs that recognize and follow human intent.
August 8th, 2010 - 12:35
The debate was not as good as it could have been.
When discussing Wright’s ideas, they got caught up in semantics using words like “purpose” and “design”. Wright also got very defensive. These words should be stricken from future discussions. His idea is straightforward: iterated competition leads to cooperation, then to comparative advantage and specialization, and then to greater adaptive complexity in both the biosphere and in the economy. I think he’s fundamentally right but confuses the issue by using stupid words like “purpose”, “design” and “god”.
Wright has never read or understood Yudkowsky. (To be fair Yudkowsky never bothered to write a book and you can’t expect people to sift through dozens of webposts). Nevertheless, they get to the heart of the issue at 29:00 where they discuss collective intelligence versus single intelligence; but no one pursues it further. This is where the debate should BEGIN. Instead the entire debate constituted of Yudkowsky going throught the basics of his theory to an incredulous Wright.
Maybe these debates should be in writing. Or writing followed by a verbal part; kind of like legal disputes where experts submit written positions and are then critically questioned at depositions.
August 8th, 2010 - 14:58
@5
Agreed. Potentially interesting debate that got lost by talking past each other. Got into a shouting match. Devolved into something like watching Fox “News”.
August 9th, 2010 - 00:18
very disappointing. i would recommend to eli that he not bother entering into debates without understanding beforehand how the other man goes about attacking and defending ideas. wright is simply not competent within the framework of an intellectual debate. i’m not attacking the man’s intellect (though I have my doubts) but this was a waste of time, the devolution of an interesting conversation into petty squabbles and nonsense.
August 11th, 2010 - 03:53
While I think Bodacious is being a tad-bit too hard on Wright, I essentially concur. Wright’s books are, as far as they go, reasonably intelligent discussions. But *here* he was, clearly, somewhat out of his depth, to put it diplomatically. Ideally, it might (indeed, *would*, in my judgment) behoove *both* (or *all*, if more than 2) participants in such an “dialogue” or “panel discussion” (or whatever) to actually *familiarize* themselves with each other’s work/ideas (gee, ya think?!) b4 commencing. This applies not so much to Eli (bright young, widely-&-deeply read, polymathic genius that he is), but to guys like Wright, who, while certainly far from dumb-dumbs, to be sure, are not sufficiently familiar with what guys like Eli & Ben (Goertzel), among others, having theorizing about for at least a good decade or so, now.
But, having said all that, this is still a very enjoyable discussion and not (at least not quite) a wasted effort. Ben said about 4 yrs ago we could have FAI w/in 10 yrs if we try hard enough! So that’s 6 yrs to go…I hoping for no later than 2020 (+/- 2-3 yrs)—2023 would be Vinge’s (in his original 1993 article) forecast yr anyway, coincidentally enough.
Ciao…
August 9th, 2010 - 05:10
Slightly off topic: Jaron Lanier had bad things to say about us in yesterday’s NY Times:
http://www.nytimes.com/2010/08/09/opinion/09lanier.html
November 9th, 2011 - 06:21
It is best to participate in a contest for among the best blogs on the web. I’ll recommend this website!