To put it in a single sentence, I’d say that it’s because only a minority of cognitively possible goal sets place a high priority on the continued survival of human beings and the structures we value.

Another reason is that we can’t specify what we value in enough mathematical detail to transfer it to a new species without a lot of requisite hassle.

It would be easy if we could just transfer over the goal set of a “typical human” or a “nice person” and hope for the best. But there’s a problem: we have no experimental evidence of what happens when a human being can modify its own goals, or increase its own intelligence and/or physical power exponentially.

What little evidence we have of scenarios where people acquire a lot of power in a short amount of time indicates that the outcomes are usually not pretty. In fact, we have complicated democratic mechanisms built into our society to guard against these types of outcomes.

Most AI designers are missing the challenge because no one wants to have to take the responsibility of creating the first truly intelligent being. They just want to play with their program. The idea of taking any responsibility for the products of one’s research is a relatively recent notion, one that only holds weight with a minority of scientists and engineers, even today. This is usually because scientists and engineers are embedded in a large institutional apparatus that places responsibility so far up the chain of command that the actual researchers are absolved of most, if not all responsibility.

Back to the original issue of goal sets. Here are some likely applications for the most advanced AI technologies in the next 10-20 years:

  • Intelligence analysis and wargaming. (link)
  • Law enforcement (link)
  • Analyzing interstate politics (link)
  • Finance, banking, & investing (link)
  • Controlling combat robots (link)
  • Automating work flows (link)

There are many others, but I put these on the top of the list because they have the most economic or political importance, and therefore will be getting the most research money.

As AI in these areas progresses, the systems will go from outputting decisions only when explicitly requested, to outputting decisions continually and automatically. When a human worker consults the machine for input, it will be more like dipping a cup into a stream and tapping into the preexisting flow of knowledge consolidation and decision-making, rather than flicking on a light switch or pressing “run” for a conventional computer program.

Being continuously thinking, continuously decision-making entities, these AI systems will have implicit top goals, whether people explicitly program them or not. The implicit top goal of a workflow automator will be to accelerate the completion of productive tasks. The implicit top goal of the finance bot will be to pick stocks that maximize return on investment. The implicit top goal of the combat robot AIs will be to take out or capture people specified by certain data files in its memory.

What makes AI potentially so dangerous is the lack of background common sense and humanness that we take for granted. When the clock hits 5, most workers put down their tasks and are done for the day. They go home and spend time with their family, watch TV or play games, or just relax. An artificial worker would have no such “background normality” unless we program it in. It’s on task, 24 hours a day, 7 days a week, as long as its computer continues to suck power from the wall.

It’s that kind of monomaniacal devotion that puts humanity at risk from AI when it begins to step out of the lab and into the real world. An AI with implicit top goals will want to reinforce those goals and achieve them more effectively, where the “goals” are not the same as what you’d see in a human that was handed a piece of paper with those goals written on it, but as they are represented in the context of the AI’s decision structure and worldview.

Reasonableness and sensibility about goals are not easy to transfer over to a mind without the knowledge and common sense built into every neurologically normal human being. A blank slate intelligence sitting in the middle of a forest would be able to build models and make inferences about numerous aspects of its surroundings – that trees are tall, that animals are mobile but plants aren’t, that the weather changes in cycles. But inferences about “the right thing to do”? You can’t derive an ought from an is. Putting an AI in a social environment with humans or other AIs doesn’t help, because without some deep-seated motivation to care about this weird “morality” thing in the first place, an AI will just happily go about accomplishing the subtlety-devoid goals it was originally assigned. As it gains the ability to improve on its own intelligence or tap into the power of robotics, it will continue to get better and better at achieving those goals and harder and harder for humans to reach in and grant it the motivation to care about morality in the abstract.

If AIs in any of the applications I listed before gained the ability to improve upon themselves significantly, either mentally or physically, the implicit top goals they were given will be magnified many times over. There would be little reason for the AI to modify those goals unless such flexibility mechanisms were explicitly programmed in. When a human sees someone starving, they tend to feel sorry for them and at the very least wish they could help. When a human sees someone attacking a defenseless child, they tend to get angry. To your typical AI, a person starving or a child being attacked is only relevant in the context of the goals it already has – “how does this starving human affect stock prices?”, or “can this starving human give me information regarding the location of my next target?” are two inquiries that might come to mind.

Freedom, empathy, self-determination, consensus-building, conflict resolution, aesthetics, camaraderie and rapport – these values and inclinations are built in automatically for every human without serious brain defects. For an AI to share them, they have to be put in terms of lines of code and mathematical rigor. What programmer has the time to do all that work when general intelligence without the human-like morality will be significantly easier to achieve?

It’s that difficulty disparity between stripped-down general intelligence and morally-sophisticated general intelligence that makes AI so dangerous in the long term.