My disagreement with Dale Carrico, Mike Treder, James Hughes, Ray Kurzweil, Richard Jones, Charles Stross, Kevin Kelly, Max More, David Brin, and many others is relatively boring and straightforward, I think. It is simply this: I believe that it is possible that a being more powerful than the entirety of humanity could emerge in a relatively covert and quick way, and they don’t. A singleton, a Maximillian, an unrivaled superintelligence, a transcending upload, whatever you want to call it.
If you believe that such a being could be created and become unrivaled, then it is obvious that you would want to have some impact on its motivations. If you don’t, then clearly you would see such preparations to be silly and misguided.
Why do people make this more complex than it needs to be? It has nothing to do with politics. It has everything to do with our estimated probabilities of the likelihood of a more-powerful-than-humanity being emerging quickly. I am practically willing to concede all other points, because I think that this is the crux of the argument. Boring and simple, if I am indeed correct.
I am fairly confident that, at this point in history, superintelligence is the MacGuffin — the key element that determines how the story of humanity will go. I could be entirely wrong, of course, but that is my current position, and it is derived from cogsci and economics-based arguments about takeoff curves, not political nonsense. If it is wrong, it should be entirely simple to refute the hard takeoff hypothesis at the locus of cogsci and economics-based arguments rather than political or sociological arguments. Particularly, I think that James Hughes, as a sociologist, seems to have a desire to search for a “sociological” (social signaling/subcultural) explanation for other people’s beliefs, rather than looking at the economics/cogsci side of the arguments, which is their entire substance. You have to note that the people that believe in hard takeoff hypotheses are mostly subculturally isolated from one another, and barely even come into geographical contact. What wins us over are abstract arguments like, “humans are qualitatively smarter than chimps and have a huge advantage over them; why couldn’t there exist a superintelligence that has a similar qualitative advantage over us?”