Roko: “It is probably not too much of an exaggeration to say that building a superintelligent computer program is like building a god” Monday, Dec 7 2009
intelligence 7:35 pm
Roko on artificial intelligence in his latest Good article:
Computers that can behave intelligently and rewrite their own code could be dangerous in ways that are not at all obvious to their creators. The codebreaker scenario outlined above is an example of a hard takeoff event—where a smarter-than-human AI quickly (in a matter of hours or months) takes control of the entire world and utilizes all the resources in the world to achieve whatever goal it has been programmed with. Whether such a hard-takeoff scenario is likely has been disputed; however, the ability of a smarter-than-human intelligence to surprise us shouldn’t be underestimated.
In the next Good piece we will be positive for a change — I will summarize the potential benefits of successful friendly AI.
OK but what is the solution?
Is there really one?
I mean on the one hand there are all the risks involved in building strong AI and on the other hand we can’t (or we will not) really stop the future.
Do we become Samuel Butler’s disciples and go the Unibomber way or do we become singularitarians?!
I’d rather do the latter but am not sure it is the “right” way…
I used to think the answer might be that there isn’t one, but this page convinced me that a satisfactory solution is probably possible, along with the rest of SIAI’s Friendly AI work.
I can completely empathize with your dilemma! There is a pathology in today’s society where when people see a problem presented to them in a narrative, they automatically expect to then be presented with a solution (like salesmen do all the time), but sometimes the universe just doesn’t care about us and there is no solution in sight. The universe has no special obligation to provide us with solutions, and there is no cosmic destiny to guide us. (Contrary to what Kurzweil implies.)
“It is probably not too much of an exaggeration to say that building a superintelligent computer program is like building a god”
Precisely a point I made in a paper last year. It will be interesting (supposing the superintelligence emerges) to see how religious groups hail it – as an anti-Christ character, as their incarnate god, or proof against their god (after all, in their minds, why would he allow himself to be supplanted by silicon).
It is really the only question, when will this deus ex machina be launched.
A trivial scenario which begins with 1000000 lines of Java code and continues shortly with exploding Earth is already very plausible if you ask me. Well, you don’t, but never the less.
I agree that the universe doesn’t owe us any solutions or even survival.
I also agree that, contrary to Kurzweil, there is no destiny.
Yet, when on the crossroads of a dilemma we can’t just sit there — we have to make a choice, take action and bear the consequences of those.
Unfortunately, at the time of making a choice and taking action we rarely have anything more than an imperfect quality of information and grasp thereof. Thus there is always some kind of a jump or “a leap of faith” that things will eventually work out for the better. But then again, when the stakes are that high and the issue is that complex it is hard to see if any leaps of faith will help us or kill us…