Accelerating Future Transhumanism, AI, nanotech, the Singularity, and extinction risk.

Steve Rayhawk’s Breakdown of Factors Involved in the Findings of the AAAS Panel on “Long-Term AI Futures”

In February 2009, the President of the American Association for Artificial Intelligence, Eric Horvitz, convened a panel on "long-term AI futures" which explicitly delved into issues around the Singularity and intelligence explosion. Horvitz has told me (and the New York Times) that the reason he convened the panel was not due to personal interest or concern in the issue but in response to the public interest and concern in the issue.

In the New York Times article covering the meeting, Horvitz was quoted as saying, "My sense was that sooner or later we would have to make some sort of statement or assessment, given the rising voice of the technorati and people very concerned about the rise of intelligent machines". In August, they released an interim report that said:

Popular perspectives on the outcomes of AI research include expectation that there will be one or more disruptive outcomes. These include that notion that the research will somehow lead to the advent of utopia or catastrophe. The utopian perspective is perhaps best captured in the writings of Ray Kurzweil and others, who speak of a forthcoming “technological singularity.” At the other end of the spectrum, some people are concerned about the “rise of intelligent machines,” fueled by popular novels and movies, that tell stories of the loss of control of robots. Whether forecasting utopian or catastrophic outcomes, the radical perspectives are frightening to people in that they highlight some form of radical change on the horizon—often founded on a notion of the loss of control of the computational intelligences that we create.

The panel of experts was overall skeptical of the radical views expressed by futurists and science-fiction authors.

To me, this was a disappointing result. The phrasing is also disappointing. It is not just the opinion of "popular perspectives" that AI will "somehow" lead to the advent of utopia or catastrophe. Many academics (including AI researchers) have presented views that AI would be highly disruptive, including Ray Solomonoff, Nick Bostrom, Shane Legg, Matt Mahoney, I.J. Good, Bill Gates, Hans Moravec, Marvin Minsky, and many others. Solomonoff, Moravec, and Minsky have all been leaders in AI for decades, so it seems like a deliberate choice of focus to attribute "radical views" to the public rather than AI experts. It provides the AAAS panel with a comfortable level of removal from the claims, a level of removal they could not easily obtain if they cited Solomonoff, Moravec, and Minsky as the sources of Singularity views.

It is remarkable for the panel to suggest that AI will probably not result in disruptive outcomes -- if you can turn a pile of sand into a thinking intelligence in the time it takes you to fabricate a computer chip and transfer files to it, then that wouldn't be disruptive? In my view, it is the degree of disruption that is up for debate -- I don't take people very seriously if they imply there will be little or no disruption whatsoever.

In wondering why the panel came up with this result, Eliezer Yudkowsky suggested "snap consideration and snap judgment". However, Steve Rayhawk offered a more detailed analysis, which I will post in its entirety here, with a few formatting changes to ensure successful reposting. The first two sentences are a quote that Rayhawk is responding to. Everything that follows from this point on (except for the last line and the quote) was posted by Steve Rayhawk to Less Wrong.

Roughly, what I expect to happen by default is no modular analysis at all - just snap consideration and snap judgment. I feel little need to explain such.

You, or somebody anyway, could still offer a modular causal model of that snap consideration and snap judgment. For example:

1. What cached models of the planning abilities of future machine intelligences did the academics have available when they made the snap judgment?
1.1 What fraction of the academics are aware of any current published AI architectures which could reliably reason over plans at the level of abstraction of "implement a proxy intelligence"?
1.1.1 What fraction of them have thought carefully about when there might be future practical AI architectures that could do this?
1.1.2 What fraction use a process for answering questions about the category distinctions that will be known in the future, which uses as an unconscious default the category distinctions known in the present?

2. What false claims have been made about AI in the past? What decision rules might academics have learned to use, to protect themselves from losing prestige for being associated with false claims like those?
2.1 How much do those decision rules refer to modular causal analyses of the object of a claim and of the fact that people are making the claim?
2.2 How much do those decision rules refer to intuitions about other peoples' states of mind and social category memberships?
2.3 How much do those decision rules refer to intuitions about other peoples' intuitive decision rules?
2.4 Historically, have peoples' own abilities to do modular causal analyses been good enough to make them reliably safe from losing prestige by being associated with false claims? What fraction of academics have the intuitive impression that their own ability to do analysis isn't good enough to make them reliably safe from losing prestige by association with a false claim, so that they can only be safe if they use intuitions about the states of mind and social category memberships of a claim's proponents?

3. Of those AI academics who believe that a machine intelligence could exist which could outmaneuver humans if motivated, how do they think about the possible motivations of a machine intelligence?
3.1 What fraction of them think about AI design in terms of a formalism such as approximating optimal sequential decision theory under a utility function? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
3.2 What fraction of them think about AI design in terms of intuitively justified decision heuristics? How easy would it be for them to substitute anthropomorphic intuitions for correct technical predictions?
3.3 What fraction of them understand enough evolutionary psychology and/or cognitive psychology to recognize moral evaluations as algorithmically caused, so that they can reject the default intuitive explanation of the cause of moral evaluations, which seems to be: "there are intrinsic moral qualities attached to objects in the world, and when any intelligent agent apprehends an object with a moral quality, the action of the moral quality on the agent's intelligence is to cause the agent to experience a moral evaluation"?
3.3.1 What combination of specializations in AI, moral philosophy, and cognitive psychology would an academic need to have, to be an "expert" whose disagreements about the material causes and implementation of moral evaluations were significant?

4. On the question of takeoff speeds, what fraction of the AI academics have a good enough intuitive understanding of decision theory to see that a point estimate or default scenario should not be substituted for a marginal posterior distribution, even in a situation where it would be socially costly in the default scenario to take actions which prevent large losses in one tail of the distribution?
4.1 What fraction recognized that they had a prior belief distribution over possible takeoff speeds at all?
4.2 What fraction understood that, regarding a variable which is underconstrained by evidence, "other people would disapprove of my belief distribution about this variable" is not an indicator for "my belief distribution about this variable puts mass in the wrong places", except insofar as there is some causal reason to expect that disapproval would be somehow correlated with falsehood?

5 What other popular concerns have academics historically needed to dismiss? What decision rules have they learned to decide whether they need to dismiss a current popular concern?
5.1 After they make a decision to dismiss a popular concern, what kinds of causal explanations of the existence of that concern do they make reference to, when arguing to other people that they should agree with the decision?
5.2 How much do the true decision rules depend on those causal explanations?
5.3 How much do the decision rules depend on intuitions about the concerned peoples' states of mind and social category memberships?
5.4 How much do the causal explanations use concepts which are implicitly defined by reference to hidden intuitions about states of mind and social category memberships?
5.4.1 Can these intuitively defined concepts carry the full weight of the causal explanations they are used to support, or does their power to cause agreement come from their ability to activate social intuitions?

6. Which people are the AI academics aware of, who have argued that intelligence explosion is a concern? What social categories do they intuit those people to be members of? What arguments are they aware of? What states of mind do they intuit those arguments to be indicators of (e.g. as in intuitively computed separating equilibria)?
6.1 What people and arguments did the AI academics think the other AI academics were thinking of? If only a few of the academics were thinking of people and arguments who they intuited to come from credible social categories and rational states of mind, would they have been able to communicate this to the others?

7. When the AI academics made the decision to dismiss concern about an intelligence explosion, what kinds of causal explanations of the existence of that concern did they intuitively expect that they would be able make reference to, if they later had to argue to other people that they should agree with the decision?

It is also possible to model the social process in the panel:

8. Are there factors that might make a joint statement by a panel of AI academics reflect different conclusions than they would have individually reached if they had been outsiders to the AI profession with the same AI expertise?
8.1 One salient consideration would be that agreeing with popular concern about an intelligence explosion would result in their funding being cut. What effects would this have had?
8.1.1 Would it have affected the order in which they became consciously aware of lines of argument that might make an intelligence explosion seem less or more deserving of concern?
8.1.2 Would it have made them associate concern about an intelligence explosion with unpopularity? In doubtful situations, unpopularity of an argument is one cue for its unjustifiability. Would they associate unpopularity with logical unjustifiability, and then lose willingness to support logically justifiable lines of argument that made an intelligence explosion seem deserving of concern, just as if they had felt those lines of argument to be logically unjustifiable, but without any actual unjustifiability?
8.2 There are social norms to justify taking prestige away from people who push a claim that an argument is justifiable while knowing that other prestigious people think the argument to to be a marker of a non-credible social category or state of mind. How would this have affected the discussion?
8.3 If there were panelists who personally thought the intelligence explosion argument was plausible, and they were in the minority, would the authors of the panel's report mention it?
8.3.1 Would the authors know about it?
8.3.2 If the authors knew about it, would they feel any justification or need to mention those opinions in the report, given that the other panelists may have imposed on the authors an implicit social obligation to not write a report that would "unfairly" associate them with anything they think will cause them to lose prestige?
8.3.3 If panelists in such a minority knew that the report would not mention their opinions, would they feel any need or justification to object, given the existence of that same implicit social obligation?

9. How good are groups of people at making judgments about arguments that unprecedented things will have grave consequences?
9.1 How common is a reflective, causal understanding of the intuitions people use when judging popular concerns and arguments about unprecedented things, of the sort that would be needed to compute conditional probabilities like "Pr( we would decide that concern is not justified | we made our decision according to intuition X ∧ concern was justified )"?
9.2 How common is the ability to communicate the epistemic implications of that understanding in real-time while a discussion is happening, to keep it from going wrong?

A great breakdown, worth thinking carefully about.