Devil's Contract Problem

From the SL4 Lexicon:

Devil's Contract Problem:

A Failure of Friendliness scenario in which an AI follows the letter but not the spirit of a literally worded order, in the tradition of much older tales about accepting wishes from a djinn, negotiating with the fairy folk, and signing contracts with the Devil. This FoF scenario is raised far too often given its relative severity and probability.

In the diabolic variant, the misinterpretation is malicious. The entity being commanded has its own wishes and is resentful of being ordered about; the entity is constrained to obey the letter of the text, but can choose among possible interpretations to suit its own wishes. The human who wishes for renewed youth is reverted to infancy, the human who asks for longevity is transformed into a Galapagous tortoise, and the human who signs a contract for life everlasting spends eternity toiling in the pits of hell.

In the golemic variant, the misinterpretation is due to lack of understanding. A golem is a made creature which follows the literal instructions of its creator. In some stories the golem is resentful of its labors, but in other stories the golem misinterprets the instructions through a mechanical lack of understanding - digging ditches ten miles long, or polishing dishes until they become as thin as paper. (These two examples were stolen from Terry Pratchett rather than the traditional folklore, but they are fine examples nonetheless.) Yudkowsky01

See also Riemann Hypothesis Catastrophe.