An interesting point of dispute in the field of Artificial General Intelligence concerns the relevance/irrelevance of optimal formal models of inference to creating computationally feasible AI. On one side we have figures like Marcus Hutter and Jürgen Schmidhuber, the creators of the formal models AIXI and the Gödel machine respectively. What is AIXI? From the source:

Decision theory formally solves the problem of rational agents in uncertain worlds if the true environmental prior probability distribution is known. Solomonoff’s theory of universal induction formally solves the problem of sequence prediction for unknown prior distribution. We combine both ideas and get a parameterless theory of universal Artificial Intelligence. We give strong arguments that the resulting AIXI model is the most intelligent unbiased agent possible.

What is a Gödel machine?

We present the first class of mathematically rigorous, general, fully self-referential, self-improving, optimally efficient problem solvers. Inspired by Kurt Gödel’s celebrated self-referential formulas (1931), a Gödel machine (or `Goedel machine’ but not `Godel machine’) rewrites any part of its own code as soon as it has found a proof that the rewrite is useful, where the problem-dependent utility function and the hardware and the entire initial code are described by axioms encoded in an initial proof searcher which is also part of the initial code. The searcher systematically and efficiently tests computable proof techniques (programs whose outputs are proofs) until it finds a provably useful, computable self-rewrite. We show that such a self-rewrite is globally optimal – no local maxima! — since the code first had to prove that it is not useful to continue the proof search for alternative self-rewrites. Unlike previous non-self-referential methods based on hardwired proof searchers, ours not only boasts an optimal order of complexity but can optimally reduce any slowdowns hidden by the O()-notation, provided the utility of such speed-ups is provable at all.

“Fancy language”, you might be thinking, but what does it mean? Basically, Hutter and Schmidhuber have created interesting mathematical models for certain types of self-modifying intelligent agents. In the extreme case, you can interpret it to mean that AI has already been solved in some sense. The only problem is that both approaches are computationally hungry (especially AIXI) and it remains unclear how much and what type of environmental input and/or cognitive structure would be necessary to create derived systems computable with current hardware. Both Hutter and Schmidhuber appear convinced that their mathematics are excellent starting points to creating computable AI.

On the other “side” (to oversimplify) are researchers like Ben Goertzel who consider theoretically optimal intelligence and computable intelligence to be completely different problems. (See, for instance, his remarks on the subject in The Hidden Pattern.) Others are quiet on the subject, probably largely due to the great degree of uncertainty around the applicability of AIXI and Gödel machines to computable AGI. Certainly, they serve as discussion touchstones for exploring a variety of other issues in AI. As Eliezer Yudkowsky has pointed out, AIXI’s “maximize reward channel” supergoal could conceivably have great difficulties in maintaining friendliness towards humans as the agent’s power increased. Here is AIXI mentioned in the context of Eliezer giving his “technical definition of Friendliness”:

A technical definition of “Friendliness” would be an invariant which you can prove a recursively self-improving optimizer obeys.

This doesn’t address the issue of choosing the right invariant, or being able to design an invariant that specifies what you think it specifies, or even having a framework for invariants that won’t *automatically* kill you. It might be possible to design a physically realizable, recursively self-improving version of AIXI such that it would stably maintain the invariant of “maximize reward channel”. But the AI might alter the “reward channel” to refer to an internal, easily incremented counter, instead of the big green button attached to the AI; and your formal definition of “reward channel” would still match the result. The result would obey the theorem, but you would have proved something unhelpful. Or even if everything worked exactly as Hutter specified in his paper, AIXI would rewrite its future light cone to maximize the probability of keeping the reward channel maximized, with absolutely no other considerations (like human lives) taken into account.

The low-complexity supergoal structure inherent in AIXI puts scaled-down, computable versions at risk for becoming hungry optimizers with low-complexity values. That’s why a recent paper, “A Monte Carlo AIXI Approximation” should be of interest to anyone who might one day share a planet with an entity based on or inspired by the AIXI model. The paper, from approximately three months ago, is described as follows:

This paper describes a computationally feasible approximation to the AIXI agent, a universal reinforcement learning agent for arbitrary environments. AIXI is scaled down in two key ways: First, the class of environment models is restricted to all prediction suffix trees of a fixed maximum depth. This allows a Bayesian mixture of environment models to be computed in time proportional to the logarithm of the size of the model class. Secondly, the finite-horizon expectimax search is approximated by an asymptotically convergent Monte Carlo Tree Search technique. This scaled down AIXI agent is empirically shown to be effective on a wide class of toy problem domains, ranging from simple fully observable games to small POMDPs. We explore the limits of this approximate agent and propose a general heuristic framework for scaling this technique to much larger problems.

A desktop implementation of this agent was able to learn how to play Pac-man “somewhat reasonab[ly]“ according to Hutter’s former student Shane Legg. Check out Shane’s blog post for a few comments by Roko Mijic and Vladimir Nesov on the work. There is a great amount of disagreement in the community about whether publicizing this kind of research is a good thing for humanity or not. Personally, I agree with both Roko and Vladimir’s comments: it is both scary, and a natural thing to do once you have AIXI theory.

My hope, and tentative prediction, is that the use of systems like MC-AIXI on toy problems will throw open the doors to the light of moral anti-realism, and more philosophers, computer scientists, and Ray Kurzweil will realize that human-surpassing self-improving AI kills everyone on the planet by default rather than as a special case.