A Boring Disagreement?
My disagreement with Dale Carrico, Mike Treder, James Hughes, Ray Kurzweil, Richard Jones, Charles Stross, Kevin Kelly, Max More, David Brin, and many others is relatively boring and straightforward, I think. It is simply this: I believe that it is possible that a being more powerful than the entirety of humanity could emerge in a relatively covert and quick way, and they don't. A singleton, a Maximillian, an unrivaled superintelligence, a transcending upload, whatever you want to call it.
If you believe that such a being could be created and become unrivaled, then it is obvious that you would want to have some impact on its motivations. If you don't, then clearly you would see such preparations to be silly and misguided.
Why do people make this more complex than it needs to be? It has nothing to do with politics. It has everything to do with our estimated probabilities of the likelihood of a more-powerful-than-humanity being emerging quickly. I am practically willing to concede all other points, because I think that this is the crux of the argument. Boring and simple, if I am indeed correct.
I am fairly confident that, at this point in history, superintelligence is the MacGuffin -- the key element that determines how the story of humanity will go. I could be entirely wrong, of course, but that is my current position, and it is derived from cogsci and economics-based arguments about takeoff curves, not political nonsense. If it is wrong, it should be entirely simple to refute the hard takeoff hypothesis at the locus of cogsci and economics-based arguments rather than political or sociological arguments. Particularly, I think that James Hughes, as a sociologist, seems to have a desire to search for a "sociological" (social signaling/subcultural) explanation for other people's beliefs, rather than looking at the economics/cogsci side of the arguments, which is their entire substance. You have to note that the people that believe in hard takeoff hypotheses are mostly subculturally isolated from one another, and barely even come into geographical contact. What wins us over are abstract arguments like, "humans are qualitatively smarter than chimps and have a huge advantage over them; why couldn't there exist a superintelligence that has a similar qualitative advantage over us?"
August 21st, 2009 - 22:35
for me, its not probability but possibility that matters. if its even remotely possible quick, repeated self-modification could occur than the implications, if true, are so large they are worthy of massive amounts attention from everybody.
its like even if there’s a 1% probability of a nuke being detonated in NYC, the fact that such destruction is possible compels everybody to agree with massive expenditures towards preventing that sort of terrorism.
i think you should push that sort of 1% argument, rather than get into an argument over what is more likely. because as i see it, even preparing for the rapid-scenario encompasses preparation for the slow takeoff scenario aswell.
August 22nd, 2009 - 00:51
I think that Hughes probably takes the sociological ground assumptions of assuming that people’s beliefs should be described exclusively in terms of their sociological causes, not as the results of rational arguments. The idea of truth causing beliefs is mistakently seens as superstitious because the sociologists don’t understand the mechanisms by which map entangles territory but they do see memetic conflicts. Object level arguments also win with Robin, but he also ignores them, which is less defensible from an econ perspective. In any event, almost everyone generalizes from history and concludes that in practice people won’t be honestly swayed by object level arguments so they fear technocracy. They aren’t open to object level arguments for why this time will be different because there were supposedly such arguments last time.
August 22nd, 2009 - 01:59
Microsoft Windows arose “relatively quickly” – and spread its DNA all over the planet – mostly obliterating its rivals. Machine Intelligence will probably be a bit like that.
On the other hand, computers have been developing for a hundred years or so now. Their rise has happened quickly on geological scales – but has not been *that* rapid on a human timescale.
August 22nd, 2009 - 02:25
Google popped up fairly quickly too – and it clearly represent a self-improving ecosystem. If either they or Microsoft are responsible for the first superintelligent machine, we should presumably have started the clock ticking back when those companies were formed – right?
August 22nd, 2009 - 02:30
> Why do people make this more complex than it needs to be?
Because the conclusions that an honest analysis would lead to are rarely the ones that would be most convenient to the person doing the analysis.
August 22nd, 2009 - 03:53
Machine intelligence has taken some 50 years to “emerge” so far – at least. So: we know it takes at least that long.
Didn’t start the clock ticking back then? Don’t even have the the clock ticking yet?!? When is it going to get started, then?
August 22nd, 2009 - 07:59
Boring indeed! What is NOT boring are the techniques how it will be done, one sunny day in the quite near future.
But I wouldn’t discuss that. Who will?
August 22nd, 2009 - 10:34
It is indeed a boring disagreement because even with some small but significant probability of a super-intelligent agent emerging with surprising quickness, the important question is not the *narrow* one of how to ensure that the first such singleton is “friendly” (however you might attempt to define it), but rather, how best to promote a *broad* network of robust and resilient greater-than-human-level intelligence reflecting the evolving values of humanity.
August 22nd, 2009 - 12:36
jefallbright,
When you take as the starting point of your analysis the scenario of a super-intelligent agent emerging into a position of power with surprising quickness, any hypothetical “broad network” representing positive values will have to *include* the first such superintelligence. If the first superintelligence achieving great power doesn’t represent positive values, it doesn’t matter if you later attempt to launch a broader network, since the first one has already won (according to the specification of the scenario under analysis).
So what you suggest isn’t actually an alternative to what Michael has suggested. Instead, a solution to what Michael sees as the key problem is *included* in the requirements of your approach as well.
August 22nd, 2009 - 21:16
“economics-based arguments about takeoff curves”
What does economics have to do with it?
August 22nd, 2009 - 22:06
Aleksei,
It sounds like you assume that superintelligences cannot be result from augmented human minds. Something like this, while probably not capable of the rapid self-improvement by leaps and bounds that Michael speaks of, would only need to be slightly more intelligent than any other human to be considered a super-human intelligence.
A world containing a substantial number of enhanced people who have become, say, 50% smarter than Newton or Von Neumann or Terry Tao [or your favorite genius here] would go a long way toward improving our world, assuming their emotional maturity and moral reasoning is at least as strong as the best of today. (And I sincerely hope we are moving toward this.) They could continue to improve themselves, and make it easier for others to enter on the same or similar paths. And one of the things they likely would guard against is a malicious or amoral AGI.
I wonder if this is not too dissimilar from what Jeff has in mind above.
(By the way, it’s a safe bet that these minds could answer with greater confidence than ours the question of “how probable is the emergence of a hidden, quick-bootstrapping AGI?” Of course, that doesn’t help us to answer it at the moment…)
August 23rd, 2009 - 12:12
Why do you call a Maximilian a Maximilian? Where did you originate the term? I’m just curious.
August 23rd, 2009 - 17:37
Mitchell, because simple economic arguments about the productivity and costs of AI reproduction contribute a lot of probability mass to hard takeoffs. A preliminary modeler of that idea is here.
Kingraven, Maximillian means “the greatest” in Latin, and a truly smarter-than-human being would indeed be the greatest entity on the Earth in terms of its power. I also like the term because it refers to historical rulers and I think that the world under a human-derived superintelligence would plausibly be much more like past monarchies than an AI-derived superintelligence would be. A human-derived superintelligence could still want a lot of human typical things, like social power, recognition, human status, etc. (Of course, it might not, if it transcended those desires.) I doubt that an AI would be programmed to specifically value these things. A Maximillian could also just be a plain old human that has a contained superintelligence under his or her control, though that scenario might not last long.
August 23rd, 2009 - 18:45
Thank you for explaining, that was enlightening and very well put.
August 23rd, 2009 - 19:23
Michael – OK. I thought you meant arguments from economic theory, not from Moore’s law.
August 25th, 2009 - 16:48
It’s not from Moore’s law. It is from the economic tradition of making very simple assumptions and seeing how they play out in a model.