Skype Co-Founder: “We Need to Ensure That a Self-Correcting System Will Stay True to its Initial Purpose”
A Singularity Institute donor and Singularity Summit sponsor, Skype co-founder Jaan Tallinn understands the risk of advanced artificial intelligence. Estonian Public Broadcasting recently covered his remarks on the topic:
Jaan Tallinn, one of the founders of Skype, believes humans may succeed in creating artificial intelligence by midcentury.
Tallinn told uudised.err.ee that in order to create artificial intelligence, two important problems need to be solved. "First, we need to ensure that a self-correcting system will stay true to its initial purpose. Secondly, we need to solve a more difficult problem – to determine what we actually want. What are those initial goals for a computer that is given super intelligence?" Tallinn asked.
He added that there could be negative outcomes if artificial intelligence is more powerful than humans but cannot interpret human values. "If a computer needs to get carbon atoms, and it doesn’t care about humans, then it would think the easiest place to get them is from humans. It would be more difficult to acquire them from the air," said Tallinn.
It is hard to say what qualifies as artificial intelligence, said Enn Tõugu, senior researcher of the Cybernetics Institute at the Tallinn University of Technology. "I can’t really even tell you what exactly is intelligence, intellect, reason or knowledge," he said.
"I tend to think that we can talk about intelligence as a human quality […] that computers can possibly attain. To some degree, it already is so. For example, I see such beginnings in Google," Tõugu said.
To modify the above slightly, to achieve artificial intelligence, those two problems should be solved -- they won't necessarily be. Creating artificial intelligence without making a self-modifying system stable or accurately specifying what we want could be a species-ending disaster, but entirely possible. In fact, economic pressures may make it more likely than the alternative -- Friendly AI.
October 25th, 2010 - 00:34
He is almost certainly familiar with the SIAI AI agenda; even the atom-as-resource-for-UFAI example is a SIAI staple. If he’s not familiar I am extremely impressed and surprised at once.
October 25th, 2010 - 04:32
This blog post started by stating that he is a “Singularity Institute donor and Singularity Summit sponsor”.
October 25th, 2010 - 13:40
Why can’t we just tell the AI to do everything we want and nothing we don’t? Why would it do anything else, ever? Give it a list of stuff you want and let it do the research and achieve those things. Then keep it in a waiting loop or, more usefully, in an info acquisition loop, sort of a learning-coma, until asked for new things. How could it start doing things it’s not asked for, especially if doing anything not on our wish list is explicitly forbidden?
I know you can answer: it can do anything and we can’t control it. But really, don’t you think that we could build a smart-enough system for achieving our purposes that can be controlled like regular software?
October 25th, 2010 - 15:23
What would you ask it to do? What else would it need to do to accomplish that? How would it weigh /different/ ways of accomplishing that?
October 25th, 2010 - 21:20
http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
October 26th, 2010 - 00:52
It might be able to be controlled like software if we can specify our wish accurately (see Nick’s link).