The basic idea of utilitarianism is to do the greatest good for the greatest number. A related idea, negative utilitarianism, requires us to prevent the greatest amount of suffering for the greatest number. According to its proponents, the greatest harms are more consequential than the greatest goods.

Would you rather avoid being tortured for a day, or engage in your favorite activity for a day? For many, the answer is obvious: avoid torture. This synchs well with humanity’s empirically demonstrated aversion to risk. It also makes sense evolutionarily, as avoiding pain was probably more adaptive than merely seeking pleasure.

Negative utilitarianism seems like a reasonable enough philosophy, at least at first. What could possibly be wrong with minimizing harm? Well, it turns out that the optimal implementation of negative utilitarianism would be to kill off all of humanity in the quickest and most painless way possible. That way, the probability of Earth-originating sentients experiencing harm in the future is reduced to zero. From the perspective of negative utilitarianism, this is the best possible outcome.

Hah! Now I’ll bet you think negative utilitarianism sounds like a horrible idea, don’t you? The problem is that it may be a philosophically appealing viewpoint to a subset of humanity. One challenge of futuristic technologies is that they may make possible the existence of groups that are unaccountable in practice. I’m not saying I want such groups to exist, or that such groups existing is a good idea, just that it could actually happen and we might be hard pressed to do anything about it. A prototype of such a scenario is given in the novel Aristoi, where aristocrats uses nanotechnology and brain-computer interfaces to ensure absolute dominance over the rest of humanity.

If an Aristoi class decides that negative utilitarianism makes sense, then from their perspective, it could be quite appealing to destroy all of humanity. Do we have any intelligent strategies for averting such a possibility?