At the core of Singularitarian philosophy and values lies the following proposition:

Smarter-than-human, kinder-than-human intelligence is possible and desirable.

A central Singularitarian concern is the idea of risk mediation. Even if we have a very good theory, the chance of successfully carrying out a given goal can never be 100%. All we can do is push the probability as high as we can. Creating a superintelligence is an action that inherently has an element of risk. But given the continuing acceleration of computing power and cognitive science knowledge, we do not see avoiding it as plausible.

We see the eventual creation of superintelligence as overwhelmingly likely. However, we do not see the eventual creation of benevolent superintelligence as overwhelmingly likely. What we intuitively regard as "benevolent" actually corresponds to a goal system with a substantial amount of cognitive complexity - we shouldn't expect it to just pop up in any intelligence that is "sufficiently rational".

Making a benevolent Artificial Intelligence from scratch is quite different than raising a child to be benevolent. Some children may refuse to be benevolent regardless of their upbringing, while others may be benevolent in spite of it. The primary difference lies in cognitive hardware biases: the way their brains are wired by their genes.

In coding the first AI, the programmers will be the ones setting the initial cognitive hardware biases. Rather than creating selfish organisms in the way that biological evolution does, the first AI should be a selfless entity, one that is only interested in self-preservation or personal accomplishments insofar as they are of help to others.

The first AI should make a large effort to understand human issues and preferences before taking action to fulfill them in ways we approve of. This shouldn't require the AI to become a human itself; but high-quality mental simulations (better than the internal simulations humans have of each other) may be necessary to reach favorable outcomes.

next->