In the secret, back-room Singularitarian mailing lists and discussion venues, we often ask: “More publicity good? Or more publicity bad? How much publicity is optimal?”

There’s no question that our cause (building safe seed AI) has more exposure now than ever. While it can be hard, if not impossible, to distinguish references to Singularity a la Kurzweil from Singularity a la I.J. Good, the two concepts are meshed together and people really do get exposure to both, even if they come away thinking that Singularity means “transhumanism” instead of “recursively self-improving superintelligence”. And the people who are really in the know can actually tell the difference. For instance, Kevin Kelly, founding editor of WIRED, recently wrote about our version of the Singularity at his blog, the Technium. When the Intel CTO mentioned the Singularity coming by 2060, he was talking about Kurzweil’s Singularity, so in my mind that doesn’t really count.

The goal is to get ourselves enough exposure to get the funding and talent we need to implement Friendly AI as quickly and safely as possible, and no more. Any additional exposure is a risk, because it increases the chance that someone with a ton of money says, “AGI, that sounds like a great idea! Good thing Isaac Asimov did all the groundwork on that friendliness issue for us, so we can just plow ahead on the intelligence part!” Then, after a successful brute force implementation, the AI develops self-replicating robotics, creates trillions of dummies that meet the definition of “human” based on its training set, and goes about spending the rest of eternity converting the universe into sock puppets and making certain to obey them. (Which is pretty easy, considering that the AI controls both the dummies and the system doing the obeying.)

The answer to the “more publicity?” question depends greatly on how hard one wagers AGI to be, or more appropriately, what your probability distribution over difficulty levels is. The people who wager that AGI is relatively “easy”, as in, requiring about a dozen brilliant programmer-theorists a la Fellowship of the Ring, along with a good ten or twenty million dollars, won’t want our cause to gain much more publicity or exposure. Those who wager AGI is extremely hard, as in requiring thousands of programmer-theorists and billions of dollars, would obviously want as much exposure as possible, as it would be necessary to reach the finish line. I fall somewhere in the middle.

On Overcoming Bias, Eliezer Yudkowsky recently observed how he thought many people in the field of AGI were simply ordinary. In my worldview, this is great. My personal experience with SIAI employees and interns indicates they are anything but ordinary. That means the “good guys” — those who make a huge deal about AI Friendliness and warn that we could all be exterminated if we mess up AGI programming — are doing better than the “bad guys” — those who just want to create AGI because it sounds like an interesting research project and are anticipating nothing more than obedient robots with IQs of 90.

But, in my view, the “good guys” still don’t have enough resources and talent, so we need more exposure. Not exposure to the general public, but targeted exposure to highly educated audiences. In a certain sense, the meme is self-filtering. Our version of the Singularity can’t be boiled down to soundbites easily. It helps to have detailed background knowledge about things like philosophy of mind, reductionism, rationality, the human tendency towards anthropocentrism, Homo economicus, evolutionary psychology, and more. Average members of the general public may stumble upon blogs like this and try to understand what I’m saying, but based on what I’ve seen, they’re likely to seize on some tiny incidental point I made and ignore the bigger picture, thereby stopping the spread of the meme in its tracks. Insofar as it makes reckless drives towards AGI less probable, that’s a good thing.

In the end, I don’t think that a million dollars a year and a dozen supergeniuses is enough. We need more resources, more talent, because the challenge of AGI is huge. It looks like the probability of success (by anyone) before 2015 is quite low, and the good guys have a significant theoretical head-start. I think we can afford (and in fact require) more exposure, until the necessary philanthropists and supergeniuses step forward. A major software project is not cheap, and taking the planning fallacy into account, things are going to take more work than we suspect. But once we reach that threshold — stop! Don’t keep plugging ahead for exposure like a mindless robot. That’s just what we’re trying to avoid, y’know?

And wait — you said there are smart bloggers out there that actually aren’t writing about this stuff?