Nick Bostrom on Superintelligence
Quoting from Ethical Issues in Advanced Artificial Intelligence:
It seems that the best way to ensure that a superintelligence will have a beneficial impact on the world is to endow it with philanthropic values. Its top goal should be friendliness. How exactly friendliness should be understood and how it should be implemented, and how the amity should be apportioned between different people and nonhuman creatures is a matter that merits further consideration. I would argue that at least all humans, and probably many other sentient creatures on earth should get a significant share in the superintelligence’s beneficence. If the benefits that the superintelligence could bestow are enormously vast, then it may be less important to haggle over the detailed distribution pattern and more important to seek to ensure that everybody gets at least some significant share, since on this supposition, even a tiny share would be enough to guarantee a very long and very good life. One risk that must be guarded against is that those who develop the superintelligence would not make it generically philanthropic but would instead give it the more limited goal of serving only some small group, such as its own creators or those who commissioned it.
This seems pretty straightforward. Another important factor may be whether we assume that superintelligence will acquire a favorable morality with little real work (as J. Storrs Hall seems to think will happen), or whether we have to figure out what "it" is and program it in by hand. The latter seems to be a huge hassle, yet also unavoidable. The "lazy way" around the problem would seem to be to enhance a human's intelligence until it becomes recursively self-improving, but that also introduces problems of its own. Most Singularity transition scenarios seem to assume that no one intelligence will jump significantly ahead of the others, and that the most powerful agent at any given time will still be dependent on the collective. I prefer transition strategies that work even if the most powerful agent at any given time is unrivaled.
I am very concerned about humans that program a superintelligence to serve only a group or even a single individual. If a human being had a superintelligence on its side with the goal only to serve its master, that human might be impossible to bring down. You can think it's BS if you want, but I am concerned about the possibility. If this is possible, then who cares about "advanced science and technologies" like nanotech, biotech, and infotech? We can develop all the fancy gadgets we want, and still be caught naked when someone creates a superintelligence and uses it to achieve his or her goals. Eventually, the genie might get out of control, which could simply bring down everything. I doubt that a human could rein in a superintelligence without its own complete morality indefinitely, but maybe it is possible.
The idea is to infuse superintelligence with its own complete morality so you don't have to worry endlessly. If we are lazy, we will be stuck with a half-completed work, and get to experience it for the rest of eternity. Once a superintelligence is created, it might be impossible to get rid of. We'd also be doing our universe a disservice by creating a monster that may spread outwards as fast as it can. It is rude to create a self-propagating superintelligent entity that does not put your best face forward as a species. Consider it the galactic equivalent of farting in a crowded elevator.
August 22nd, 2009 - 03:41
“I am very concerned about humans that program a superintelligence
to serve only a group or even a single individual.”
Michael, I’m grateful that you picked up this point.
As I mentioned in one of my recent posts on your blog,
its the biggest of my concerns regarding AGI, too.
Without discussing all the possible scenarios that could lead to an
functional AGI engine, here. I think it’s likely that ones big investments starts
to flow in the AGI field, the current idealism of some researcher may become
say ‘diminished.” We could all end up with an AGI called Lord Vader that is an immensely
powerful toy for one person or a small group of people, while the rest of the world is left
in the dust or worse.
Therefore I have a critical view on the huge donations that are made
to the SIAI by some single persons. Without knowledge about their true motives,
I will ever suspect they’re not purely “human-friendly”.
I don’t know how thoroughly this point is already discussed in the AGI field.
But if it’s not, it will be wise to do so.
-Ben
August 22nd, 2009 - 05:46
“The usually reliable Michael Anissimov has claimed that I seem to think that “superintelligence will automatically acquire a favorable morality without any real work.†Now I’m not all that sensitive about such things; but it bothers me that SIAI, of all people, should fail to understand the basic parameters of the problem, and thus have a completely unworkable notion of what to do about it. …”
http://www.foresight.org/nanodot/?p=3254
August 22nd, 2009 - 08:32
I see the problem and agree it’s a legitimate worry. I’m impressed by the audacity of your solution. Make the a friendly superintelligence before anyone else does and before anyone thinks to stop you. Particularly given the broad changes you desire, that’s the work of a revolutionary. I respect such vision and courage.
I assume you recognize the potential pitfalls of this approach. It can only succeed if the authorities either sympathize with you or underestimate the effect of a superintelligence on their power. Honestly, I have trouble taking the scenario seriously. It’s similar to assuming civilian scientists could have developed the atom bomb before the US government did. The military tends to investigate even dubious technologies that make great claims. Why would they blithely ignore AGI once the prospects looked good?
Furthermore, imagine the political reaction if the scheme came to fruition. Unilaterally creating the new dominant species would arouse widespread outrage. People would ask why they hadn’t been consulted in such a critical matter. They’d condemn the researchers for hubris. Now, these complaints might be brief. An end to scarcity and a better political system would go a long way towards pacifying the critics. But if the desired gains failed to materialize, the turmoil and vilification would likely continue unabated.
Instead of frantically attempting to win the race, you might endeavor to build a global coalition for the development of philanthropic superintelligence. The movement could simultaneously oppose the creation of any AGI beholden to narrow interests. I don’t know that this approach would be safer or more likely to succeed. I propose it only to demonstrate the presence of alternatives.
August 22nd, 2009 - 12:07
to infuse superintelligence with its own complete morality
Hold on – my doggy does not infuse me with whatever she wants. I infuse her with what I want, because I am smarter. Why should it be any different with AIs?
August 22nd, 2009 - 12:47
Giulio, because it would be infused at the embryonic stage. The morality of a superintelligence has to come from somewhere, and since humans will build it, it will initially come from humans. There are two possible positions on the issue — 1) ignore that starting point content (“initial motivations” as Nick Bostrom says), or 2) care about them. Position 2 makes sense to me, even if there is a chance that initial motivations will change throughout self-improvement.
August 22nd, 2009 - 12:56
Hold on – my doggy does not infuse me with whatever she wants. I infuse her with what I want, because I am smarter. Why should it be any different with AIs?
Excellent point, Giulio, which is why I think the whole FAI project, though clearly well-intentioned, is flawed from the start.
First you must assume that it’s possible to create sentience outside a biological mind (which is not a certainty), and then you must assume that the artificial intelligence will be capable of recursively self-improving (also not a certainty), and THEN you must assume — with stunning grandiosity — that the human designers not only can program the AGI with “friendly” directives but also can somehow insure that the AGI is not able (or doesn’t want?) to adjust its own programming in a way the humans would not prefer. Seems to me that’s a whole lot of questionable uncertainties and assumptions stacked up together.
August 22nd, 2009 - 13:06
Eh, assumptions one and two strike me as highly likely. You’ll find no certainty in the future, only good bets. And as Michael noted in the post directly above yours, FAI folks don’t necessarily assume they can prevent the AGI from adjust its motivations. Wouldn’t you prefer an artificial intelligence that at least began its existence friendly?
August 22nd, 2009 - 13:55
I agree with Ben on this — I see assumptions 1 and 2 as well-supported and likely, just like Nick Bostrom does. I also see a risk of hard takeoff even in biological intelligence enhancement.
As Ben says, the point is not to promise that we will surely get it right with absolute certainty, just set the starting position favorably. That’s all we can do.
As I wrote in my response to Giulio, you can either care about the starting point or not. I prefer to do so. You seem not to.
We disagree on the probabilities of AI and hard takeoffs, that doesn’t mean that we have political differences, or that I am anti-democratic, or crazy. It just means we disagree on those issues on what are probably scientific grounds.
August 22nd, 2009 - 15:28
Giulio:
> Hold on – my doggy does not infuse me with whatever she wants. I infuse her with what I want, because I am smarter.
Giulio, that kind of thing is illegal, you should be ashamed of yourself. ;-0
August 22nd, 2009 - 15:29
Unless by “my doggy” you mean your wife or partner. In which case, it’s ok.
August 22nd, 2009 - 16:49
giulio: Because you are smarter *and* want to do so. Giant cheesecake fallacy.
August 22nd, 2009 - 17:21
The example of pets actually demonstrates the difficulty of controlling what we classify as lesser intelligences. Just ask my dad about his recent frustrations with one of the family dogs. And that’s with an allegedly obedient animal. Cats, by contrast, tend to train humans rather than vice versa.
August 22nd, 2009 - 17:54
Roko, you should not make such dumb/immature jokes in these comments threads, even if you are bored of arguing and want to amuse yourself. It really makes my blog look bad.
August 23rd, 2009 - 08:20
Sorry! Just couldn’t help myself… I guess these debates can get a little old.
But just to clarify, this is the core debate, and it does need to get plugged.
To quote Treder:
“THEN you must assume — with stunning grandiosity — that the human designers not only can program the AGI with “friendly†directives but also can somehow insure that the AGI is not able (or doesn’t want?) to adjust its own programming in a way the humans would not prefer”
putting aside the anthropomorphism for a second, *if* you program an agent to like you, and it considers a possible self-modification that will cause it to not like you, then it will see that making that self modification would be a bad thing. Why? Because it sees a world which contains an agent that dislikes you and might kill you, which is not a good thing according to its current goals.
The problem is making the above rigorous, in light of the fact that we are not sure how self-modification really works.
August 23rd, 2009 - 11:23
Roko, it’s hardly less rigorous than saying that the AI won’t press a button that kills a person if it doesn’t want to kill people.
August 23rd, 2009 - 12:14
> Roko, it’s hardly less rigorous than saying that the AI won’t press a button that kills a person if it doesn’t want to kill people.
that is true. The above is anthropomorphic language as well.