Michael Wilson on AGI Funding Wednesday, Oct 25 2006
AI 7:24 am
On AGIRI’s general mailing list, Michael Wilson of Bitphase AI, Ltd., responds to the question, “how can you tell when an AGI project is worth investing in?”:
There have been many, many well funded AGI projects in the past, public and private. Most of them didn’t produce anything useful at all. A few managed some narrow AI spinoffs. Most of the directors of those projects were just as confident about success as Ben (Goertzel) and Peter (Voss) are. All of them were wrong. No-one on this list has produced any evidence (publically) that they can succeed where all previous attempts failed other than cute powerpoint slides - which all the previous projects had too. All you can do judge architecture by the vague descriptions given, and the history of AI strongly suggests that even when full details are available, even so-called experts completely suck at judging what will work and what won’t. The chances of arbitrary donors correctly ascertaining what approaches will work are effectively zero. The usual strategy is to judge by hot buzzword count and apparent project credibility (number of PhDs, papers published by leader, how cool the website and offices are, number of glowing writeups in specialist press; remember Thinking Machines Corp?). Needless to say, this doesn’t have a good track record either.
As far as I can see, there are only two good reasons to throw funding at a specific AGI project you’re not actually involved in (ignoring the critical FAI problem for a moment); hard evidence that the software in question can produce intelligent behaviour significantly in advance of the state of the art, or a genuinely novel attack on the problem - not just a new mix of AI concepts in the architecture, everyone vaguely credible has that, a genuinely new methodology. Both of those have an expiry date after a few years with no further progress. I’d say the SIAI had a genuinely new methodology with the whole provable-FAI idea and to a lesser extent some of the nonpublished Bayesian AGI stuff that immediately followed LOGI, but I admit that they may well be past the ‘no useful further results’ expiry date for continued support from strangers.
Setting up a structure that can handle the funding is a secondary issue. It’s nontrivial, but it’s clearly within the range of what reasonably competent and experienced people can do. The primary issue is evidence that raises the probability that any one project is going to buck the very high prior for failure, and neither hand-waving, buzzwords or powerpoint (should) cut it. Even detailed descriptions of the architecture with associated functional case studies, while interesting to read and perhaps convincing for other experts, historically won’t help non-expert donors make the right choice. Radically novel projects like the SIAI may be an exception (in a good or bad way), but for relatively conventional groups like AGIRI and AAII insist on seeing some of this supposedly already-amazing software before choosing which project to back.
Personally if I had to back an AGI project other than our research approach at Bitphase, and I wasn’t so dubious about his Friendliness strategy, I’d go with James Rogers’ project, but I’d still estimate a less-than-5% chance of success even with indefinite funding. Ben would be a little way behind that with the proviso that I know his Friendliness strategy sucks, but he has been improving both that and his architecture so it’s conceivable (though alas unlikely) that he’ll fix it in time. AAII would be some way back behind that, with the minor benefit that if their architecture ever made it to AGI it’s probably too opaque to undergo early take-off, but with the huge downside that when it finally does enter an accelerating recursive self-improvement phase what I know of the structure strongly suggests that the results will be effectively arbitrary (i.e. really bad). As noted, hard demonstrations of both capability and scaling (from anyone) will rapidly increase those probability estimates. I understand why many researchers are so careful about disclosure, but frankly without it I think it’s unrealistic verging on dishonest to expect significant donated funding (ignoring the question of why the hell companies would be fishing for donations instead of investment).
There are a few really good things about Michael Wilson. First, he is Bayesian. This means, in general, that he represents his belief confidence in terms of probabilities, takes prior probabilities fully into account, and comprehends the relationship between conditional and prior probabilities. Second, he understands AI and its consequences. This means that he doesn’t regard human-equivalency as a stable-state optima for intelligences capable of recursive self-improvement, has a nonanthropocentric model of the space of minds in general, and fosters a pragmatic design attitude despite a well-fleshed-out understanding of normative reasoning. Third, he has real experience both as a programmer and entrepreneur. Not to say that such experience is a necessary condition for success in AGI - but it doesn’t hurt, especially when coupled with the first two traits.
The ultimate message: there is no “AGI evidence test” in the way that a positive mammography is a good test for breast cancer. The prior probability of success is quite low - lower than one divided by all AGI projects up to this point - so 1/100 or so at greatest. To make up for this disparity, both the AGI evidence indicator must be rare, and the presence of the indicator given an imminently successful project must be close to unity.
For example, say that Bayesianity is the AGI indicator. If the prior probability of success for any AGI project is 1/100, and only 1/10 AGI projects display Bayesianity, and we know that any successful project must display Bayesianity, then according to Bayes’ theorem, the probability of success given Bayesianity is 1/10. This is because the team is question is competing against the other 9 Bayesian teams out of 100 AGI projects for the single probabilistically-likely success.
Unfortunately for AGI researchers, the probability of success for any given AI project before 2020 without a discontinuous breakthrough is could be considerably lower than 1/100 - probably more like 1/10,000. The problem with determining indicators, like Bayesianity, is that we can know little about the necessary characteristics of a successful project, given that there has never been a successful project instance before, ever. The first step to increasing the probability that your project succeeds is to take on the necessary characteristics of a winning project. For example, listening to Enya is almost certainly not a necessary characteristic for a winning AGI project. Therefore, it doesn’t matter whether you listen to Enya or not. However, having over three million dollars and having read over 10,000 cumulative pages on inductive inference are likely characteristics of a winning project, so any projects who want to boost their probability of success will need to fulfill those characteristics, whether they like it or not.
Understand?
October 25th, 2006 at 12:44 pm
Since no AGI projects have yet been successful in terms of producing human level intelligence its hard to say in advance what the necessary characteristics will be.
A simple tally of papers read is unlikely to be a good measure of potential success. Actually the reverse could be the case. The more stuff you’ve read about previous attempts the more contaminated your thinking may become with old failed paradigms.
October 25th, 2006 at 2:04 pm
How very much I agree with Bob here, you can’t imagine!
True.
October 25th, 2006 at 6:02 pm
Just because you’ve read the failed paradigms, doesn’t mean you will use them. Rather, since you *know* they are failed you’ll try other ways.
October 25th, 2006 at 6:44 pm
Also, inductive inference isn’t really “AI” per se - it’s more like the mathematics that must underlie any inductive system for it to work.
Anyone who wants to be successful with AI will need to know a lot about inductive inference, regardless of which approach they ultimately pick.
October 26th, 2006 at 11:35 pm
The term “intelligence” is too broad, too contaminated with unprecise meanings, that should be scraped altogether.
Instead, I want a bunch of algorithms, which do the job. Well, which job?
To predict increasingly good on increasing amount of data. That is all we want, that covers everything.
Even that old Turing test is passable, if the prediction of human’s emotions as reactions to written text is accurate.
There is nothing more about intelligence than that.
The safety measures like Friendliness are a separate problem.
October 26th, 2006 at 11:44 pm
And, yes. If that BOA (bunch of algorithms) predicts well what will be the result of a calculation, the calculation is optimized. Predicting is enough for selfimproving also.
The recursive predicting - what we will predict? - clears the murky waters of the predicting business also. It is selfexplanatory, just as intelligence has to be.
SAI will come out from playing with some special algorithms.
October 27th, 2006 at 7:47 am
“I’d say the SIAI had a genuinely new methodology with the whole provable-FAI idea and to a lesser extent some of the nonpublished Bayesian AGI stuff that immediately followed LOGI, but I admit that they may well be past the ‘no useful further results’ expiry date for continued support from strangers.”
Does this mean SIAI has hit a dead end?
October 27th, 2006 at 8:18 am
“Does this mean SIAI has hit a dead end?”
This simply means SIAI has not shown evidence of useful results to the degree that it would make sense to invest in their approach to AGI over any other approach.
October 27th, 2006 at 8:20 am
Note that this doesn’t necessarily mean that SIAI is neither producing useful results it simply does not publish nor that their approach to AGI doesn’t make sense.
October 27th, 2006 at 8:55 am
I’m not sure calling him a Bayesian makes sense. He may approximate a Bayesian, but I don’t think this is a term you can equivocate on to allow him to have that name.
October 27th, 2006 at 4:21 pm
Saying “Bayesian wannabe” is awkward, and everyone here knows what I actually mean. Here I’m using Bayesian in the sense of “student of Bayes” rather than “person whose neurons are warped into a pattern capable of using true Bayesian inference”.
October 28th, 2006 at 3:21 pm
“To make up for this disparity, both the AGI evidence indicator must be rare, and the presence of the indicator given an imminently successful project must be close to unity.”
The second part is definitely false. You can have very strong indicators of success that are not likely to be exhibited by any successful project, so long as they are even less likely to be exhibited by an unsuccessful project.
Also, just because the evidence indicator is rare doesn’t necessarily help you. This whole argument has the logical structure, “If we start out with a valid indicator of AGI success, it must be rare relative to all the projects that have failed”, not, “If we have an indicator that is rare in failed projects, we have an indicator of success.” This point deserves extra emphasis because so much popular literature inverts it. Even if no previous AGI project has used bananas, it doesn’t mean you need to use bananas. Maybe the reason nobody uses bananas is that bananas don’t help.
Finally, you should be aware that Michael Wilson is not up-to-date (to put it mildly) on current SIAI ideas or work. Current in-house SIAI work is being carried out between myself and Marcello Herreshoff. Wilson is essentially a guy with his own AI project, in the UK, who sees the problem of Friendliness but is taking a very different theoretical approach. In fact, I don’t know what his theoretical approach is, at this point; and he certainly doesn’t know mine.
October 28th, 2006 at 4:33 pm
Good to see you posting here, Eliezer! Personally—not that I’m within several hundred parsecs of being a peer—I find your and Ben’s (Goertzler’s) approaches most plausible & promising. Besides the research on (meta)friendliness is, I think, fruitful in itself. I’ve also always been rather perplexed that such personages as, e.g., Hans Moravec, seem blithely unconcerned about friendliness being a problem…And, of course, your point in your 3rd paragraph superbly spot-on…
October 29th, 2006 at 2:07 am
> Maybe the reason nobody uses bananas is that bananas don’t help.
Maybe. OTOH, bananas may be required.
October 30th, 2006 at 6:28 pm
If somebody takes “base rates” into account, he is not a Bayesian. Bayesians take *prior probabilities* into account, which are not exactly the same thing.
People who talk about “base rates” are essentially giving a non-Bayesian interpretation of Bayesianism.
October 31st, 2006 at 8:24 am
The indicator that Wilson is looking for is: A prototype. We don’t expect the prototype to be perfect, but it should show some sparks of something resembling intelligence.
If such a prototype can’t be churned out by a team of two or three caffiene-fueled hackers in three months, then the project does not have solid ideas underlying it.
I know that this is probably impossible today, and carries unfriendliness-risk, but this is how ALL successful R&D software projects work.
November 1st, 2006 at 9:09 am
I took out the phrase “base rates”… having not really studied statistics in great detail, I thought that base rates and prior probabilities were the same thing.
November 1st, 2006 at 6:01 pm
‘The more stuff you’ve read about previous attempts the more contaminated your thinking may become with old failed paradigms.’ - In my experience this viewpoint is held mostly by cranks. In fact the spectrum of viewpoints goes something like this, ranked roughly from least competent to most competent;
* Ignore all that past work, it’s worse than useless and will only contaminate your thinking, we must brainstorm radical new solutions (cranks, mostly).
* It’s obvious that only predicate logic/neural networks/Bayes/whatever will produce AGI. Papers on those other approaches may be interesting, and there may even be salvagable techniques, but they’re essentially a dead end (most AI academics).
* Many past AI approaches have had local successes, but failed to generalise. They all have strengths and weaknesses. We must learn about as many approaches as possible, promote collaboration and attempt to combine them all into one diverse mega-architecture that will have all of their strengths and none of their weaknesses (some of the more daring AI academics, collaboration junkies, some of the more clueful beginners).
* Past approaches have had only limited success, despite seeming reasonable to their designers (who were often at least as bright as me). We need to understand why they (locally) succeeded, why they (ultimately) failed and why they looked like a good idea at the time despite being broken. You should read as much past work as possible until you can reliably answer these questions, as this will greatly improve your chances of designing an AGI system that can incorporate the core discoveries of past work while hopefully avoiding past misconceptions and misfeatures (the researchers I most admire either hold this view or are getting close to it).
November 1st, 2006 at 6:12 pm
“Does this mean SIAI has hit a dead end?” - the SIAI is a special case because it’s primarily researching FAI theory, and is not (yet) an AGI project. There are many AGI projects clamouring for support and a long track record of failure. To my knowledge the SIAI is the only organisation dedicated to developing FAI theory, a field which has no real track record. As such I’d advocate giving the SIAI more leeway than an AGI project. That said, even if the SIAI is doing good theory work it’s a lot less likely to be useful if it’s never published or shared (because the SIAI is no more immune to the very high AGI project failure prior than anyone else), so there is still a time-out after which the group isn’t worth supporting if there are no visible results. How long that time-out is depends on your personal priorities and general assessment of the situation.
No human uses normative Bayesian inference for more than a tiny fraction of their reasoning (and even that is hard work). Sucks, but there you go. I think it’s reasonable to call people who appreciate that probabilistic logic is the correct way to deal with uncertainty (given adequate cognitive resources) and try to apply it as much as possible (in particular, try to avoid the standard, pervasive human reasoning flaws when dealing with uncertainty) ‘Bayesians’. I don’t think anyone (with a clue) is going to confuse a human with a genuine probability-logic reasoner, and Bayes rule is only one theorem in probabilistic logic anyway.
November 1st, 2006 at 6:29 pm
Oh and I’d basically support Eliezer’s comments on being careful about reasoning here; don’t get your causality reversed. The question of what project to back as a non-expert with no privilidged information who wants to see AGI developed, is different from the question of what project to back if you’re a multimillionaire investor who everyone will fall over themselves to give private demos to, which is different from the question of evaluating competing projects if you’re already an AGI expert, which is different from the question of evaluating a project that you’re personally working on and have complete knowledge of. The problem of not knowing what the best indicators are in advance of someone actually solving the problem is compounded by the fact that many of the best indicators are probably hidden from public view.
No I don’t know what Eliezer is doing at present, he says he can’t explain it, which I understand (explaining AGI and FAI is /really hard/) but doesn’t let him off the hook when he doesn’t even issue quarterly progress summaries. For reasons mentioned above the SIAI should be permitted more lenient evidence-of-progress requirements than an AGI project (again, ignoring the fact that AGI-without-Friendliness will most probably /destroy the world/), but there should still be /some/ disclosure requirement for support-from-strangers.
I am generally happy to summarise what I’m doing, but right now I don’t think that many people would be interested; it’s both strange and out of fashion (i.e. insufficiently emergent or connectionist). That said I’ve nearly finished the initial material for the Bitphase site, so a reasonably complete version should be up by the end of the week. It’s taken a while simply because doing demos for potential clients takes priority over putting up PR material that right now only a few people who know me from AI forums would read.
November 2nd, 2006 at 4:49 am
> * Ignore all that past work, it’s worse than useless and will only contaminate your thinking, we must brainstorm radical new solutions (cranks, mostly).
Every single step further is a cranky one. Or there is nothing new. More cranky - less probable - more brilliant, if proven right.
Less cranky - more probable - more of the same old stuff, if not proven wrong.
Seldom, you can and must forget all the theories, how malaria is spreading with a bad air, to notice the mosquito role in the process.
Usually the old theories are good enough. Need nothing new. And get nothing new.
Most cranks are also getting nowhere. It takes a million or so damn cranks to get somewhere. Most are absolute losers, of course, but you don’t remember them.
Several (academic mostly) cranks are remembered. Only that their cranckhood has been renamed a posteriori, to brilliance or something else in fashion.
Just do your best!
November 2nd, 2006 at 5:06 am
The irony is that if you don’t even bother to read past work, you will almost certainly ‘brainstorm’ something completely unoriginal. All of the ‘I pay no attention to past work’ cranks I’ve seen have actually come up with fairly naive and superficial designs that closely resemble those of many previous cranks (and occasionally competent researchers).
If someone seriously has to ‘forget’ existing theories to avoid being constrained by them, I’d have to say that they probably don’t have the mental flexibility and ability to handle complexity required for AGI design. There are too many consecutive design steps, too many sucessive pitfalls and it’s too hard to recognise incremental success for anyone who doesn’t know what they’re doing to have a nontrivial chance of success. The later stages of the minefield (currently) have to be traversed from first principles, but a thorough study of existing work helps enormously with traversing the early stages.
November 2nd, 2006 at 5:46 am
It is a matter of a degree. How well you will acquainte yourself with everything already known. At least every important past idea is already in your bag, or you are not intelligent enough, to have a chance with your cranckhood, to become a mainstream one day.
But you must hate old theories a bit. Enough to not bother too much with them. They don’t work, do they?
November 2nd, 2006 at 6:12 am
It is possible of course, that one of those old ideas is the right one.
In that case, every implementation was poor, and all you have to do, is a good implementation.
This is what I think, BTW.
November 3rd, 2006 at 1:19 am
Excellent list of views on AGI by Michael Wilson… and yes Michael, it would be great if you would add more (correctly spellchecked!) material to your Bitphase site.
I’m not sure about what Eliezer is saying, I’ll have to examine it more closely.
Fantastic discussion btw…
August 24th, 2008 at 3:25 pm
9uThank’s for greate post.3u I compleatly disagree with last post . lbu
ламинат купить 6h