Suppose that a very reliable source told you that, somewhere in Turkmenistan, there was a world-destroying bomb set to go off in one month. The only way to disable the bomb is by cutting the red wire. Clearly, destroying the world is bad, and so the next day, you fly out to Turkmenistan, locate the bomb, and snip the red wire. The bomb then blows up, taking you and the world with it, as you needed to cut the blue wire before you cut the red wire.
This scenario results an obvious negative payoff from new information, both relative to doing nothing and relative to omniscience. The new information didn’t make anyone’s model of the world worse; a model of the world with a bomb and one wire is clearly superior to a model with no bomb and no wires, relative to an actual world with a bomb and two wires. It appears that the more inaccurate model leads to a better outcome. Can you rig a scenario where you know (in advance) that improving your world-model will result in a bad outcome, and that you should leave it unchanged?
In population ethics, a linear summing over utilities leads you to the Repugnant Conclusion: if you get 3^^^^3 people together, and have them all experience the smallest possible amount of joy, the expected positive utility will exceed that of our entire civilization. This is the negation of Pascal’s Mugging, although it has wider implications: if true, it means that we should fill the universe with cheaply reproducible happiness, even if we have to eliminate most of the happiness-related mind states that humans currently experience. Peter de Blanc has proven that, if you have an unbounded utility function, you will always wind up in situations like these, and so the problem must be with our estimates of our utility functions and not our ideas about happiness.
However, given the current state of the world, we should still consider the Repugnant Hypothesis: What if the sum of utility over most human lives is negative, rather than positive? In the real world, even if this is true, it shouldn’t matter that much; we still have hope. But it could have raised some awkward questions if we were rational enough to consider it in, eg., AD 1200. Should we no longer have children? Should we try and kill as many people as possible, except for those who are known to be happy? How does the utility created by destroying the planet balance against the potential future utility, if life in the future turned out to be better?
A quick thought experiment: Suppose that, in 1945, you had all of the high-ranking Nazi war criminals in a Star Trek-style holodeck. The obvious course of action is to shoot them, or torture them in various ways, but this won’t do anyone any good; your actions are never revealed to the public, and so you won’t deter future dictators or prevent another Holocaust. The only causal result of one additional unit of pain is one additional unit of pain. Yet, even though a lot of people accept the principle that pain is always bad and pleasure is always good, I suspect that many of them would be uncomfortable with letting the Nazis live out the remainder of their lives in the most pleasant way possible. What’s the right thing to do, in this situation?