For the last couple weeks, I have been debating Berkeley professor of rhetoric Dale Carrico more or less non-stop. This morning, responding to his criticisms of the Singularity Institute (SIAI), I wrote a summary of reasons to support the organization:

~~~

Dale,

SIAI works towards Friendly (through whatever means works, something other than mathematical-deductive if necessary) seed AGI because the people in the organization see it as a high moral priority. This is humanity’s first experience of stepping beyond the Gaussian curve of ordinary human intelligence distributions. If it is not facilitated by AGI, it will be by enhancing humans: whether through psychopharmacology, neuroengineering, brain-computer interfaces, gene therapy, etc.

The question is not “if” intelligence enhancement technologies will be available, but “when”. When they are, it will become possible to “construct intelligence” actively rather than be limited to human generational cycles, birth-rates, and education. Now there’s nothing at all wrong with these conventional human patterns, but we have to note that the introduction of enhancement technology is bound to throw the existing order out of whack.

It shouldn’t be hard to imagine that enhanced humans or AGIs will get to the point of being substantially smarter than the smartest given humans. After all, the hardware differences, in terms of basic components, between a human and a chimp isn’t actually all that large. But a machine could process thoughts at greater speeds and with more flexibility than any member of our species.

If intelligence enhancement tech really does produce a superintelligence, then we have a moral duty to maximize the probability that said superintelligence cares about humanity as a whole, not itself or any narrow group of humans. Otherwise the outcome could be grim. A few thousand Europeans enslaved native populations of millions with “only” somewhat more advanced technology — here we are talking about fundamental differences in substrate and cognitive architecture. To assume that we could keep intelligence-enhanced people or AGIs under our control is foolish.

So, the idea is to “get them while they’re young”: create superintelligences with altruistic goal systems. SIAI is the only organization pursuing this goal in a structured manner.

Michael

~~~

Dale responded here, in the post “Debating Singularitarians”. (I suggest you go read it before reading what’s below.) Being a bit frustrated by his disrespectful tone, I gave the following response:

~~~

Dale,

I have to say right off that the somewhat disrespectful way you engage in discussion makes me less motivated to spend time on it. For instance, “Cue the music”, “Gosh, that’s big of you”, “True Believers always feel about their Pet Raptures”, etc etc etc., shows you aren’t really taking my opinion or statements very seriously at all. In your responses, your tone doesn’t even address me directly, it sounds more like an attempt to mock me in front of some sympathetic third party audience. Such a disrespectful way of interacting would be frowned upon at a round table meeting or in a classroom context. At a cocktail party, it would cause someone to simply walk away.

I am interested in your criticisms of Singularitarianism because I believe they reflect the concerns of a wider group of people who are silent. But, I find it difficult to engage with your venomous and sarcastic tone. I wish we could talk at least under the pretense of mutual respect. (I have respect for your ideas but the inverse clearly does not apply.)

I don’t have a clear idea of what goes down when we create human-equivalent AI or enhanced human thinkers. I wish you wouldn’t put words in my mouth and claim that I do have an idea, because I don’t. There are a range of possible outcomes, but it’s useless to delve into them if one doesn’t even believe the underlying premise: that significant intelligence enhancement is technologically possible.

I’m barely even into SF. I don’t watch any television or many movies. I watch anime that is mostly fantasy, not sci-fi. I got into transhumanism by reading non-fiction books. Most fiction deals with AI in a really anthropomorphic way, so it doesn’t factor into my thinking about the future of AI in the real world. I dislike much sci-fi and often give it negative reviews, like the negative review of Accelerando I wrote about a year ago.

I don’t think you have a lack of vision or imagination. I disagree with Roko’s critique of you in his recent post.

I don’t think that the risk of rogue AI is only 5% , but substantially greater than that. Like James Hughes and many other transhumanist philosophers, I believe human-level AI is likely to be developed in the first half of the coming century.

Problems are not “reductively solvable through the implementation of instrumental rationality”, but using a variety of techniques such as communication, charisma, creativity, research, brainstorming, and experiments. Any AI of any use would need to possess all these characteristics, or it wouldn’t truly be human-equivalent. If it does possess these characteristics, then it could certainly help with human problems.

I am sympathetic for many of the causes you list, but think that implementing many of them would be nearly impossible. For instance, the USA is never going to put itself in a position where its politicians or generals are subject to punishment by international courts, no matter how much we try to institute such a structure. Making war unprofitable is incredibly difficult, and I’d advocate a technological solution — clean, abundant energy through solar power. The US military is never going to reveal its precise budget, that is a fantasy. And military funding is not going to be appreciably decreased, because since 9-11 people are on edge. Russia is threatening us, Iran is threatening us, China is highly militarized, etc. I have opinions on all the causes you mention but I think that politics as usual is not the way to go about it.

Technological solutions, such as increasing transparency, will help circumvent impasses that have held since the beginning of civilization. Humans have had basically the same motivations since ancient times, but technology changes. Technology such as air travel has fundamentally changed the way people worldwide interact. I think that transparency (facilitated by technology) coupled with demands for government accountability (facilitated by activism) will reduce militarization, but the technological component is critical. If it were so easy to get the militaries of the world to put down their arms, it would have been accomplished a long time ago. There is too much international tension. An eventual world government or strengthening of the UN could help in this regard.

To realize many of your political goals would require that the Republicans in this country magically disappear, which isn’t going to happen. I advocate many centrist political positions because I see them as a possible equilibrium to demands from opposing sides of the political spectrum. Many of your socialist ideas could never be successful, they merely aggravate the conservative crowd and cause the pendulum to swing in the other direction. I am a liberal but I am also practical.

Michael

~~~

And that’s it for now. Dale’s extremely disrespectful and negative attitude totally stresses me out, but I keep arguing for some reason.

As a bonus, an insightful commenter, “Utilitarian”, had this to say in response to another commenter:

~~~

James,

My argument is that if AI is plausible then there will be some appropriate strategies to pursue, most of them very different from SIAI’s donor-funded secretive algorithmic research approach.

If AI is first constructed through imitating brain processes, then beneficial outcomes would hinge more on processes of education and understanding messy emotions rather than formally represented goal systems. Then the SIAI approach would be irrelevant (except as something to be taken up by the humanlike AI) but the personality and motives of resulting AI would remain extremely important.

This is so even without a ‘hard takeoff’: if it takes time to develop and then educate an AI the first ones created can occupy whole labor markets by rapid self-replication. This would create massive populations with extremely similar motivations, a powerful constituency that would shape the future and provide the base for modification for superior intelligence (even if that process takes several years). Creating the initial AI and allowing it some freedom to communicate and replicate would be practically and politically irrevocable (as AI could quickly spread across borders and make itself economically and militarily indispensable) without extraordinarily rapid recursive self-improvement.

The shared motivations of those AIs would matter enormously: shared tendencies towards sociopathy and contempt for humans (harmless in positions of political weakness) could be disastrous even to the point of human genocide, while benevolence at or beyond the upper limits of the human range could make for a tremendously better democratic society. And in either case the susceptibility of digital minds to copying, intelligence enhancement without many of the complications of using biotech on brains, and running on faster hardware would mean tremendous increases in economic and technological growth.

If hostile or unfriendly AI is not a danger at all, then billions of dollars should be put into public funding for basic research in AI, allocated through normal processes of peer review (plus potentially better ones like tiered prizes). The ratio of cost to expected benefit would be very favorable relative to funding for nuclear fusion research (with its or much of our biomedical research funding (most of which will fail to produce anything of value). [Incidentally, I think both fusion power and cancer research are very worth funding, despite their track records of hype and failure.]

If hostile AI is a danger, and we have unfriendly humans and the fact that power is a convergent goal for lots of different nonhuman motivations to support the idea that it is, then other general measures could be appropriate. Theoretical examination of those dangers that can be conceived of now, conditions on government funding or regulation to ensure that precautions are taken, and possibly an increase in research funding to increase the likelihood that AI is developed under the safety framework rather than outside it come to mind.

Regardless, there are many things to try to do to produce beneficial AI other than secretive research aimed at a provably safe algorithmic AI, funded by private donations at an organization like SIAI with staff you dislike for various reasons (some of them very important ones). There are many possibilities under which AI research and AI safety are of critical importance other than the SIAI hard takeoff scenario, and alternative paths to solutions. I would like to see a supplement to Dale’s critique of Superlative discussion of AI with positive alternatives, rather than throwing out the baby with the bathwater. [And I don’t take the advice to join the ACLU as really serious engagement on positive alternatives, given the extent of resources already dedicated to its causes and diminishing marginal returns. Discussion of activism to disclose secret weapons research by DARPA would better suggest engagement rather than just a convenient club for dismissal. I also haven’t seen a response to the argument that if AI safety is nothing to worry about then we should just try to increase public funding for basic research into AI by quite a lot.]