Bob Mottram Objections to 2010 Singularity Research Challenge and Response
Bob Mottram isn't impressed by the Singularity Institute's grant proposals for our $100,000 Singularity Research Challenge:
It's kind of sad how SIAI seems to have become obsessed with "AI risks" and human extinction. Perhaps they always were from the beginning, but it's just my perception of them that was at fault. There's certainly a place for some group, existing independently from academia, who actively promote AI related R&D in a direction which has positive value to society and addresses problems which are highly relevant. This applies especially to the work which is less glamorous, more ambitious stuff which requires an expenditure of effort on a longer time scale than a typical PhD thesis or DARPA/X-prize contest.
The list of grant proposals for the Singularity Research Challenge seems incredibly disappointing, and focused on spurious notions of risk which, in my opinion, would have no beneficial impact on AI even if it were to be funded in its entirety.
To clarify what is happening, what Bob Mottram considers "spurious notions of risk", we consider "deadly serious notions of risk", so this is the main source of disagreement. Here was my response:
Our Uncertain Future project is pioneering probabilistic futurism in AI and WBE studies, and has received thumbs-up from several academics including Bela Nagy, who manages the Santa Fe Institute Performance Curve Database.
A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider "nice" or "common sense" is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values. See the Fun Theory sequence on Less Wrong, for instance.SIAI believes that AGI is an extremely difficult endeavor and deserves far more theory-level work than programming in the dark or working towards narrow AI tasks that drain away our attention at the expense of the Singularity itself.
Basically, if you consider an intelligence explosion plausible, SIAI's activities make sense, and if you don't, they don't. It's not a matter of marketing, just disagreement on which tasks are the most important for humanity to face right now. We consider clarifying decision theory and creating a reflective decision theory to be a major priority, for instance, and spend time on that accordingly.
To clarify further, in 2009 SIAI grew large enough to break into several loose divisions. This is excellent, because the Singularity Institute is one of the most important organizations on the planet and is one of the only barriers standing between humanity and extinction from unFriendly AI. However, it makes the task of explaining what we do all the more complicated. It so happens that I am paid to explain it, but sometimes I get discouraged because I discuss the organization constantly on this blog, occasionally several posts per day, and there is still a great amount of confusion about what our organization does and believes. Perhaps I ought to run an SIAI Video Q&A in the vein of Eliezer's recent Less Wrong Q&A.
What happened in the last year is that Anna Salamon and Steve Rayhawk joined us and created the Visiting Fellows program, under Anna's leadership. (Anna, Steve, and myself were only recently added to our staff page.) This entity is only peripherally related to SIAI's central AI project, which was more or less put on hold for two years while Eliezer Yudkowsky wrote the Less Wrong sequences. As our 2009 accomplishments document states, Eliezer worked with Marcello Herreshoff (his profile can be found here) on Friendly AI over the Summer.
So, think of SIAI as having three branches -- 1. Administrative/PR, which consists of President Michael Vassar, Media Director Michael Anissimov (aka me), and Chief Compliance Officer Amy Willey, 2. AGI research that constitutes serious progress towards seed AI, which makes up years of Eliezer's past work, Eliezer's future work after he finishes putting together his rationality book, Marcello Herreshoff's intermittent work, and contributions from Peter de Blanc, Nick Hay, and others (since 2006), including Anna Salamon and Steve Rayhawk, and 3. the Visiting Fellows Program, which including Visiting Fellows and various volunteers.
The goal of the Visiting Fellows Program is to put together extremely smart people concerned about reducing existential risk and have them pursue academic projects that make the best use of their respective strengths. Branch #3 also serves as a filtering mechanism for 2. The thing is, starting a true AGI project would be very expensive, not so much in money but in terms of intelligence, philosophy, computer science, and math knowledge required. Consolidating the necessary personnel will not be easy.
Why are we concerned about "AI risks" and human extinction? Well, this is why, among other reasons. SIAI is not about pursuing intermediate AI commercial benefits -- our organization only exists to pursue the Singularity and minimize AI risk. Writing illustrating this point has been produced in substantial quantities since our founding in 2000. SIAI is mostly a bunch of utilitarians.
Would readers be interested in a Lulu book putting together a lot of information about the Singularity Institute in one place? Only about 0.5% of my blog readers ever comment, so I feel like I'm talking to a vast sea of silent lurkers all the time. Seriously, it's weird.
In general, our approach turns off people like Bob Mottram, but inspires praise from people like Alan Darwst. In particular, Mr. Darwst writes:
Among the utilitarians I've met over the years, a sizable fraction have come to the conclusion that the optimal destination for utilitarian funding is organizations that research speculative futuristic scenarios and the philosophical / scientific / methodological questions that such research requires. In particular, many of these utilitarians have named the Singularity Institute for Artificial Intelligence (SIAI) as a good example of such an organization, so I'll focus on it here, but the discussion can apply more broadly.
The way the Singularity goes is a matter of life and death for humanity. Unfriendly AI programmed to value anything besides a very specific set of Homo sapiens-characteristic values will probably overlook our material preservation. From the perspective of most possible minds, humans are just another arrangement of atoms. We don't have any inherent moral value. "Moral value" is an "imaginary" thing that only exists among the tiny space of minds-in-general with explicit moral philosophies.
If we had the ability to build AGI today, our planet would not last the year, because we haven't solved Friendly AI. If we could build a seed AI now, we wouldn't know how to specify its goals in a way that doesn't eliminate us all completely. We are clueless. We can't create a utility function that is consistent under reflection and preserves individual humans when a tremendous amount of optimization pressure is applied to fulfilling it. We need a mathematical model of value that leaves us alive even when the unimaginable power of superintelligence is channeled into it. I think that Coherent Extrapolated Volition is a good enough solution that it would work, but it needs to be specified in much more detail. That's exactly what one of our grant proposals is about.
These grant proposals deserve funding now. We are about to walk into a minefield and we don't even have a map. We need to throw everything at the problem -- people, money, attention, everything.
January 9th, 2010 - 21:29
Dear Michael,
I understand you’re trying to maximise the return on every dollar without looking like a cult (mass produced uniforms for everyone :) )
having said that, What is it about the work that Singularity Institute is doing that requires you to be in Palo Alto, which except for Switzerland and Luxemborg is probably the most expensive real estate in the world?
Couldn’t you guys cut costs by moving to a cheaper place in the US or somewhere else in the world and get your academic connections via the internet?
What technological advances would you think have to become ubiquitous before you could contemplate moving?
Please please do not interpret this as a hostile question. I believe in your cause. A true friendly hard-takeoff is one of the VERY FEW things I can visualise that will resolve the present problems of coordination in a status seeking and special interest ridden society.
I intend to donate whether or not you remain in silicon valley. Its just that everytime i seek to donate, the Dollar to indian Rupee ratio really pinches me mentally.
regards,
Prakash
January 9th, 2010 - 22:17
“having said that, What is it about the work that Singularity Institute is doing that requires you to be in Palo Alto, which except for Switzerland and Luxemborg is probably the most expensive real estate in the world?”
Real estate in the Bay Area is indeed expensive, but it’s nothing like Moscow, London, New York City, Paris, etc. (http://www.globalpropertyguide.com/press-relations/Most-expensive-real-estate-markets-in-2009).
“Couldn’t you guys cut costs by moving to a cheaper place in the US or somewhere else in the world and get your academic connections via the internet?”
The Bay Area has been a center for futurists/technophiles for a long time now, and it’s very difficult to just pick up an entire community and more it somewhere else, especially on a minimal budget. If the Singularity Institute is much larger in 5-10 years, I think it’s quite possible we’ll have one center in the Bay Area, and another somewhere else to save on living expenses.
January 9th, 2010 - 22:50
“the Singularity Institute is one of the most important organizations on the planet and is one of the only barriers standing between humanity and extinction from unFriendly AI.”
No offense, but you sound totally delusional.
January 10th, 2010 - 01:24
Prakash,
If you’d like to talk about it please send me an email. myfirstname.mylastname@post.harvard.edu
January 10th, 2010 - 02:04
Re: “A hard takeoff from a human-indifferent AI is not a fallacious risk. It is quite real. Because human moral values are complex, creating a machine that does what we would consider “nice†or “common sense†is much more difficult than creating a machine with human-level intelligence but insufficiently complex and specific values.”
Much the same could have been said about operating systems. Computers are already much more powerful than humans in a wide range of domains – and computers are so deeply embedded in the world’s infrastructure that – unless the world’s computers are “nice” to humans – then all hell would break loose.
In fact, computers are usually “nice” to humans – due to market forces and the law. The solution adopted was to let the humans tell the computers what to do. That’s essentially the same solution that Asimov proposed to this problem, long ago, with his 2nd law.
As for a “hard takeoff” – it is a fallacy – as far as I can see. Machines compete with a symbiosis of the last generation of machines and humans. The idea that there will be a sudden increase in the rate of progress when machine intelligence reaches human level is thus a fantasy. Machines don’t just compete with humans, they compete with augmented humans and companies – which are gradually moving targets, which are already self-improving. Automation is a gradual process which is happening now. Progress may well get faster – but not in the way that “singularity” supporters often claim it will.
January 10th, 2010 - 03:42
Bob Mottram’s original comments can be found at:
http://streebgreebling.blogspot.com/2009/12/singularity-research-challenge-funding.html
January 10th, 2010 - 15:33
Tim:
I’m generally in agreement with your views regarding the improbability a hard takeoff.
Even with a slow takeoff, FAI remains an important problem.
There are two levels to solving the FAI problem.
1. The conceptual level of describing in words what the motivation of the AI should be.
2. The technical implementation level of effectuating #1 in low level code, math or architecture.
Prof. William Hibbard has proposed the following FAI solution:
1. Conceptual: the AI should do whatever humans want it to do.
2. Technical implementation: train reinforcement learning on human smiles.
Eliezer has done a good job of criticizing #2, persuasively arguing that an AI with an inflexible goal of maximizing smiling faces might force all of us to constantly smile or tile the solar system with smiling mannequins. Furthermore, there is a danger that the AI will act friendly until the moment when it greatly exceeds human intelligence, and then eliminate us. Eliezer makes a good suggestion that the AI should have a flexible supergoal. These are all good ideas that Eliezer and his colleagues deserve credit for.
In my opinion, Eliezer then goes in the wrong direction by saying that reinforcement learning cannot work for FAI and that, effectively, we must hand code the AI in a high level symbolic language. A corollary of this is that FAI enthusiasts are in a mad race against Moore’s law. He then proposes the following:
1. Conceptual: CEV, the AI should do what humanity as a whole would want it to do, if humanity knew better.
I have a number of minor criticisms of CEV (the AI will do what we would do “if we were more the people we wished to be†[Ghandi and Ghengis Khan would have different views of the ideal person]) and one major one: CEV is FAI-complete. To make a CEV-AI that follows the general will of humanity, you must first know how to make an AI that follows the will of an individual human mind. If you can do that, THEN YOU’VE SOLVED THE FAI PROBLEM. On a conceptual level, all he has done is to restate the problem and add some complications.
2. Technical implementation: Nothing. We’re working on it. We don’t just lack the code or mathematical formula. We can’t even describe in English words what the code/equation is supposed to look like and do.
My view is that Hibbard was on the right track and that Eliezer, to his credit, has added a necessary modification. One possible solution that I propose is:
1. Conceptual: The AI’s supergoals should be the following: Recognizing a human mind and following its volition:
2. Technical implementation:
a. Recognizing a human mind and following its volition: Reinforcement learning can be used for this and it can be tested as the AI slowly increases in intelligence. Animals and small children can generally understand an adult’s volition.
b. There is still a danger that the AI can have a human indifferent supergoal and do (a) only as a subgoal. To prevent this we can:
i. Insist that the AI never lie to its human.
ii. Insist that the AI seek permission from its human to act prior to any significant action.
iii. We can subject the code to relentless optimization. An AI that has a human-indifferent supergoal but only acts friendly and somehow finds a sneaky way to take action without lying its human, would have enormously more Kolomogorov complexity than an AI that was simply friendly.
These ideas are not perfect, and they are not my only ones. Perhaps they will fail under scrutiny. But at least I’ve actually made a proposal.
On Least Wrong, when Eliezer was asked who could take over FAI research if he died, the answer he gave was:
“No oneâ€.
C’mon now. If we want to be treated as a serious think tank, then let’s act like one.
January 10th, 2010 - 15:58
CEV isn’t, and doesn’t claim to be, a solution to the problem of creating systems that stably pursue a particular goal, the technical FAI problem, the hard problem. It’s a general approach (computing the output of an idealized collective human deliberative process) to deal with the further (and easier) problem of selecting a desirable goal AFTER one has the technical knowhow to instill specified goals.
January 10th, 2010 - 17:32
Carl:
Thank you for responding. I would agree with your definition of the hard technical FAI problem, with minor modification, as “creating systems that stably pursue a particular goal” [being the will/volition of a human mind].
I agree that CEV is not the the solution to the technical FAI problem. I don’t understand the reason why it is needed once we’ve solved the technical FAI problem. The FAI(s) can just communicate with individual humans via an interface like a Facebook page and ask us our preferences. When people have conflicting preferences, a legal system can be created (or we can use the legal system that we have) to deal to resolve the conflicts.
If CEV is not a solution to the technical FAI problem,and there are many society-wide systems that we can employ after technical FAI is created, why is CEV heralded as such a big accomplishment?
January 10th, 2010 - 19:59
I myself am impressed by CEV, because if I myself had to approach the FAI problem, my first inclination would be to create a normative altruist. On reflection, I think CEV has a better chance of working than normative altruism. The CEV means you don’t even have to understand complex human altruism yourself on an information-theoretic level — all you need to do is build something that accurately infers and extrapolates human preferences into a “volition”.
January 10th, 2010 - 20:06
I radically disagree with the statement
”
If we had the ability to build AGI today, our planet would not last the year, because we haven’t solved Friendly AI. If we could build a seed AI now, we wouldn’t know how to specify its goals in a way that doesn’t eliminate us all completely. We are clueless.
”
The idea here seems to be that understanding of how to create beneficial AGI is all-or-nothing: either
1) a wholly rigorous, ironclad theory
or
2) cluelessness
But I have seen no convincing argument that understanding of how to create beneficial AGI is binary in this way.
I agree that research into AGI ethics and motivational and goal systems is very valuable.
However, I tend to think that this research is going to progress a lot faster once it’s both experimental and theoretical, rather than purely theoretical.
I envision a phase AFTER the initial creation of near-human-level AGI systems, and BEFORE the creation of superhuman AGI systems, in which there is a lot of experimentation with a lot of things, including AGI ethics and motivation.
I agree it’s important to point out the potential risks of AGI; but, I think it’s also important not to exaggerate the risks. And a statement like “If we had the ability to build AGI today, our planet would not last the year” strikes me as a highly overconfident statement….
Creating an AGI with a roughly humanlike motivational system, but with violent and aggressive tendencies removed … and teaching this AGI to respect others and demonstrate compassion, etc. … this is not a guarantee of a beneficial AGI; but nor is it a guarantee of an annihilated planet.
I understand the argument that an AGI built as a simplistic reward-maximizer would plausibly kill everyone so as to minimize the risk of them interfering with its reward-maximization mathematical-masturbation process…. But human-like motivational systems are NOT simplistic reward-maximizers to this argument seems irrelevant to the more likely cases of AGI development…. Reward maximization is only part of the self-organizing dynamics of complex human-like minds.
I’ll say more about all this in my forthcoming book “Building Better Minds” (which is mainly a technical AI book, but also touches on these issues).
January 10th, 2010 - 20:11
Michael:
Michael:
I agree with you to a point. The AI(s) should follow the volitions of humans. Where I think that we differ is that I think that the AI(s) should query the volition of individual humans, perhaps in an application like a cell phone or Facebook page. In time the AI(s) wouldn’t have to query very often because they’d be ably to accurately predict. Some kind of rules would have to be enacted by humans to deal with conflicts between humans. CEV seems to interact with only a handfull of humans and then try to extrapolate the preferences of the entire species, instead of just asking individual humans.
January 10th, 2010 - 20:17
”
On Less Wrong, when Eliezer was asked who could take over FAI research if he died, the answer he gave was: “No oneâ€.
”
Perhaps he meant not that no one else could ever solve the FAI problem, but that no one else could continue his own FAI research program, because not enough of his intuitions and ideas are well-documented yet.
I felt that way about my own AGI work until 1-2 years ago. But now my ideas are fleshed out and documented well enough that others could plausibly complete my work building AGI without me. It’s a good feeling.
January 11th, 2010 - 06:41
If solving FAI is so important, why does Eliezer have time to take such a long break from it and “put together his rationality book”. I think more people would be willing to fund the Singularity Institute if we saw more documented progress in creating an FAI (or at least the theory for one) as opposed to a series of tangentially related blog-posts that seem to be more like Eliezer’s hobby.
January 11th, 2010 - 09:46
Ben:
Thank you for responding to my post. You are a researcher who’s ideas I respect. I very much enjoyed your critique/analysis of Jeff Hawkins’ On Intelligence; especially since many of your ideas are so similar. Your books are on my reading list.
January 11th, 2010 - 10:15
Re: “The FAI(s) can just communicate with individual humans via an interface like a Facebook page and ask us our preferences. When people have conflicting preferences, a legal system can be created (or we can use the legal system that we have) to deal to resolve the conflicts.”
1. That allows natural selection between machine-augmented humans. If that is permitted, the humans with the biggest and best machine augmentations are liable to come to dominate. Some people question the desirability of that outcome.
2. The machines have to understand a lot about human preferences to understand that “peel me an apple” also implies: “do not tread on my foot” and “do not wave the knife too near my eyes”. If you can program those preferences in, why not the other ones too?
January 11th, 2010 - 14:37
“If solving FAI is so important, why does Eliezer have time to take such a long break from it and ‘put together his rationality book’.”
This has already been answered before (on this blog as well as over on Less Wrong, if I remember correctly); Eliezer judges the value of getting a few more qualified people into FAI research to be worth the time investment.
January 11th, 2010 - 18:34
“Perhaps he meant not that no one else could ever solve the FAI problem, but that no one else could continue his own FAI research program, because not enough of his intuitions and ideas are well-documented yet.”
This is right.
January 12th, 2010 - 08:28
“This has already been answered before (on this blog as well as over on Less Wrong, if I remember correctly); Eliezer judges the value of getting a few more qualified people into FAI research to be worth the time investment.”
I agree that getting more qualified people into FAI research would be worth it – but is Less Wrong really doing that? That’s not even the blog’s purported purpose (“refining the art of human rationality”), at least not transparently. If he wants to get more smart people in FAI, he should start showing some progress, get some more funding, and then pay the aforementioned people lots of it.
Instead, we just get a lot of condescending talk about how most humans aren’t rational and a bunch of incomprehensible philosophy, which to the outside observer (i.e. the kind doing the funding) looks like killing time because he’s hit a dead-end in his other work.
January 12th, 2010 - 10:44
“… but is Less Wrong really doing that? That’s not even the blog’s purported purpose (‘refining the art of human rationality’), at least not transparently.”
It seems necessary to do quite a bit of “refining the art of human rationality” to get a significant number of intelligent people to realize FAI is a worthwhile goal. This is only anecdotal, but I can personally say that the sequences on Overcoming Bias / Less Wrong were vital to me making that realization.
January 12th, 2010 - 12:39
It just seems implausible at this point that Eliezer will ever finish his work before someone else makes an AGI. He should 1)devote himself to actually doing the FAI work, and 2)work on developing goal architectures (CEV) that are abstract enough to work on other AGI platforms (e.g., Dr. Goertzel’s).
If SIAI needs to raise awareness of the needs for FAI, they have people like Mr. Anissimov publishing lovely articles like “Hungry Optimizers with Low Complexity Values”. If that doesn’t convince you we need AI, no amount of Bayesian math will.
January 13th, 2010 - 01:34
Timelines and funding are problems – but CEV itself is another. It’s a goal wishlist -apparently compiled without consideration of technical issues – a “wouldn’t it be nice if” proposal. Unfortunately, it looks as though it would take about a million years to implement. Projects which unthinkingly brute-force goals into their machines by the carrot-and-stick method look to me as though they will probably get there first – and that methodology puts constraints on what can realistically be implemented. Trying to implement CEV seems likely to immediately put your project out of the picture – ensuring that others get there first.
January 13th, 2010 - 06:02
Yes Tim, I actually agree. Intelligent machines will not be “hungry optimizers”, as they will understand intent (even without extrapolated intent).
Intelligence inherently values other intelligence, and thus moral values. No other creature on earth has these, because no other creature is intelligent. Likewise, when our achievable intelligence was lower (i.e., before language), our species did not have them either.
January 13th, 2010 - 06:46
http://lesswrong.com/lw/1lo/high_status_and_stupidity_why/
Eliezer’s latest post. Clearly putting FAI research dollars to good use.
January 13th, 2010 - 21:50
Tim: I agree with you that CEV is merely a wish list and not a technical strategy. I also share your concerns about AI(s) being programmed to obey only an elite few who’ll then dominate the rest. I also agree with you that the group that waits to hand code an absolutely perfect AI will lose the race.
Dave: ” Intelligent machines will not be “hungry optimizersâ€, as they will understand intent (even without extrapolated intent).” I agree that AIs can understand human intent. The challenge of FAI is to make it their motivation to follow human intent.
I respectfully disagree (& agree with Eliezer + SIAI) with the notion that altruism is inherent to intelligence. Our altruism predates our intelligence. When we were chimp-like creatures we recognized and loved the minds that were our children and our nest mates. When we achieved general intelligence, this eveloved into a general altruism to all minds. A different evolutionary history could create an mind of high intelligence without altruism. Having said that, I think that programming a motivation of altruism towards other minds is probably enough for FAI. Human morality is particular and evolutionarily path dependent, but much of human morality is bullshit. Jonathan Haidt has identified 5 moral drives. Altruism and fairness (the real drivers of morality) are only 2 of them. The rest are just evolutionary garbage that can be discarded.
Dave: I agree about Eliezer. If his intention is to get intelligent people (ie professors) into FAI, then perhaps he should write an academic book and academic papers on the subject. I have given him due credit in my above posts for his contributions. If that’s it, and he just wants to spend his time blogging and writing sf, that’s fine with me. I just hope that an academic community arises that is interested in the friendliness component of AI.
January 14th, 2010 - 09:23
Gus – Thanks for the thoughtful reply. I tend to think of our apparent altruism towards our own children back in the neolithic era more as the exception than the rule – i.e., we were naturally savage, and it was only because we cared about ourselves and our own progeny that we were nice to anyone else. Education and increased intelligence seemed to cure this – the same way everyone in the North reading “Uncle Tom’s Cabin” made people opposed to slavery.
You may be right in that we need an altruism “seed” for the intelligence to build from, but maybe instead we just need to leave out the competitive, selfish, savage aspects of man.