Possible Views on the Future of AGI
Obviously, different people have different views on the future of AGI (Artificial General Intelligence) and its policy consequences for us in the here and now. They depend primarily on two variables: the power and controllability of advanced AGI. This produces four rough domains of opinion:
- Low power, low controllability
- Low power, significant controllability
- Great power, low controllability
- Great power, significant controllability
The low power, low controllability group are the human exceptionalists, AI denialists, and technology skeptics. They don't believe powerful AI can be created because there's something special about humans that can't be duplicated in machines. Furthermore, software (including AI) is unwieldy and difficult to get a handle on. This group emphasizes we should not rely too much on technology and need to maintain a low-tech infrastructure to cover our asses in case the technology base somehow falls out from under us. You might think this the view of the old lady you see at the supermarket, or the survivalist stockpiling his shack in the woods with handguns, but actual philosophers believe it too.
The low power, significant controllability group are the human supremacists, who see economically valuable applications for AI but don't believe it will become so powerful as to surpass us in all areas. Because advanced AI will be controllable in their eyes, they welcome the technology as long as it is nicely integrated within the preexisting human system and social structure. I also call this the Jetsons view because their belief is that future AIs will behave similarly to Rosie from the Jetsons, useful and subservient, and not intimidating or truly autonomous. Many transhumanists uncomfortable with the implications of superior AI take this route, as do many science fiction authors.
The great power, low controllability group are the de Garises, and to a slightly lesser degree the Moravecs. I whimsically refer to this group as the gigadeath view, to build on Hugo de Garis' term, meaning that this group thinks the promise of powerful advanced AI will create a deep and profound divide among human beings that leads to an all-out war between AI enthusiasts and the traditionalists, in which billions will die. In this view, programming benevolent AIs is impossible because as soon as these machines become superintelligent, all of their previous programming will be discarded outright. Presumably then we must merge directly with these AIs to make it into the future, and everybody else is out of luck. This view is pretty prominent in Hollywood, and among a few philosophers.
The great power, significant controllability group primarily originates with Eliezer Yudkowsky of the Singularity Institute. As such I will call it the SingInst view. The SingInst view acknowledges that after a certain point, AI will become self-improving and radically superintelligent and capable, but emphasizes that this doesn't mean that all is lost. According to this view, by setting the initial conditions for AI carefully, we can expect certain invariants to persist after the roughly human-equivalent stage, even if we have no control over the AI directly. For instance, an AI with a fundamentally unselfish goal system would not suddenly transform into a selfish dictator AI, because future states of the AI are contingent upon specific self-modification choices continuous with the initial AI. So, if the second AI is not the type of person the first AI wants to be, then it will ensure that it never becomes it, even if it reprograms itself a bajillion times over. This is my view, and the view of maybe a few hundred SingInst supporters.
May 19th, 2007 - 18:22
“This group emphasizes we should not rely too much on technology and need to maintain a low-tech infrastructure to cover our asses in case the technology base somehow falls out from under us.”
This has happened several times in human history, not often, but it does happen, most notably when the Roman Empire fell and when the Soviet Union dissolved. It’s one of those Black Swan scenarios Eliezer discusses.
May 19th, 2007 - 20:03
Very interesting. I have some thoughts on how we might start putting those goals together, here:
http://www.blog.speculist.com/archives/001300.html
May 19th, 2007 - 20:21
“So I propose the following Three Goals of Artificial Intelligence:
1. Ensure the survival of life and intelligence.
2. Ensure the safety of individual sentient beings.
3. Maximize the happiness, freedom, and well-being of individual sentient beings.
Will they work? If not, what goals would work better? I’d be interested to see some discussion on this.”
Any set of “laws” is probably doomed to fail, because there’s no a priori reason why human morality, which is embedded in terabytes of neural information, can be encapsulated in a few simple laws. For example, since the first law takes precedence, the AI will run roughshod over the other two even when the difference in the likelihood of our survival is too small to notice.
May 20th, 2007 - 07:50
Phil, have you read Coherent Extrapolated Volition or any of Creating Friendly AI? Tom is right about the inevitable failure of laws. Locking a few inviolable rules into the AI for all time is a Really Bad Idea. An AI must be able to reason about its goal system and override any preexisting goals that end up having potential negative results – which inevitably many will, having been designed by dumb humans.
May 20th, 2007 - 11:40
Well, these aren’t laws, and they aren’t inviolable rules. They’re goals — design objectives, if you will. The more I think about it, the more I like them as goals for humanity, irrespective of whether they are workable goals for AI.
May 21st, 2007 - 09:08
Tom: regarding non-programmable standards of morality: are you familiar with game theory?
(personal bloviation linkage):
http://www.functionalisminaction.com/2007/05/secular-morality-contradiction-in-terms.html
Of course, there are problems translating that into a recursively improving intelligence (be it AI or IA), but there *IS* a route.
May 21st, 2007 - 12:53
“Tom: regarding non-programmable standards of morality: are you familiar with game theory?
(personal bloviation linkage):
http://www.functionalisminaction.com/2007/05/secular-morality-contradiction-in-terms.html”
There’s a very nice essay on how evolutionary psychology and not religion is responsible for human morality, but I don’t see what that has to do with the encapsulability of human morality. As for game theory, there are certainly a few fundamental pieces of morality (eg, retaliation theory) that derive from it, but I’ve never heard of someone replicating anything close to human morality from mathematics alone.
May 21st, 2007 - 13:35
IF morality has a basis in evolutionary responses, then it is fundamentally dictated that morality itself follows some model of computation; i.e.; it’s software that comes with the package that is the human brain; like the hardwired analog OpSys chips that some computers had/have.
It was my *argument* ( :) ) that game theory could be more universally applied across the board, to replicate pretty much *ALL* of the “universals” of morality. Of course, since the topic is “Friendliness”, this only gives us a basis to work off of — rather than a mechanism.
I’m essentially simply saying: there is plausibility to the expectation that it is *possible* to mathematically compute morality; or at least something sufficiently like it that it will be useful to our goal.
May 22nd, 2007 - 10:39
IanC Says: “there is plausibility to the expectation that it is *possible* to mathematically compute morality”
Yeah, I’ll second that! It is something that I have been thinking about lately. It’s a difficult problem, but if anyone tells me that difficulty implies impossibility i’ll just say “mind projection fallacy”.
I think that morality could be defined in a mathematically rigorous way if we knew how to define value, that is if we knew what counts as a good state of the world. Morality would then be a set of enforcable rules in a society that promotes better states of the world.
As a practical example, consider the laws, the legal system and the police that we have today. They are designed to allow a society that is “maximally free” – because it turns out that a free society tends to be an interesting and valuable thing. Of course there are subtleties here. What do you do when people have disagreements? Why, fundamentally, is a free society a good thing?
As I’ve said before, If we can work out what’s going on with value we will be in a better position to write a friendly AI.
May 22nd, 2007 - 13:33
I think one of the problems for “high power+low controllability” group is that they mix up the goals of the AI with the process of accomplishment. That is, it may be true that *how* it accomplishes what it wants is not predictable/controllable, but the goal that it’s aiming for is.
On morality: Human morality seems incredibly complicated, and possibly even inconsistent in many ways, so any attempt to formalize it in a few simple laws is bound to fail. Even if we could get the AI to perfectly execute those goals.
Semi-related: For a “totalist” utilitarian, the problem of imperfect outcomes is even worse. We accept that a small percentage of people today are unjustly imprisoned for crimes they did not commit, partly because this works out to less (?) than millions per year. What if we are 10^40 of sentients post-sing, is it still acceptable?
May 22nd, 2007 - 16:26
Johan — you mistake morality for moral behavior. It turns out that there *are* a few — granted they aren’t many in number — elements that are universally considered, “moral”. Some people choose to behave immorally.
That, in fact, is one of the areas where AI/IA would be quite possibly *more moral* than us unmodified souls.
Imagine: we quantify through game theory analysis of social outcomes the core, base minimum of what is or is not moral. We then dictate that if you want IA, you have to accept the “Universal Morality Governor” enhancement as well.
I suggest this as a legal requirement, knowing full well of the risks implied, for two reasons: 1) It avoids many of the unFriendliness issues in IA.
2) It avoids many of the bio-Luddite arguments regarding forcible changes in society.
May 22nd, 2007 - 17:17
Johan isn’t mistaking morality for anything. Morality can be universally agreed on in the abstract and still incredibly complicated and not formalized in terms of laws. The laws are just words, and not even close to a true mathematically and rigorously defined morality.
I also agree that people who use IA should have to have a certain degree of morality and benevolence, although I would phrase it differently than “Universal Morality Governor”, which would offend obsessive individualists.
May 23rd, 2007 - 12:52
Maybe I was unclear. Many values are indeed almost universally shared but the devil is in the details . Even if we took for granted that what we actually want is what we *think* we want, it’s still too complex to formalize.
For example: it’s easy to make a simple list of values or possibly even an ordinal ranking. But in many cases you need a cardinal ranking to get exactly what you want.
Simplified Experiment:
Ask people:
1. Do you want hapiness?
2. Do you want it more than freedom?
3. How much more?
Then take all the results and feed it into a utility calculation. What is the chance of a few simple rules emerging from the result?
(and this is all presuming stated prefences are what we should be trying to maximize, which seems a good guess but not at all certain.)
May 25th, 2007 - 13:35
Re: “obsessive individualists” & “Universal Morality Governor” — It is my experience that most obsessive individualists are simply contrarians who aren’t interested in learning the actual facts over what perspective they prefer. As a person active in the libertarian community of the Western united states, I feel rather uniquely qualified to make that statement.
Re: Morality; frankly, Michael, after learning what I’ve learned on the topic — and have conveyed elsewhere — I feel able to make the following statement: you are quite wrong.
Re: what we *say* we want — that is an extremely horrible goal qualification. Not a good guess at all.