Robin seems to endorse the Modesty Argument in papers such as Are Disagreements Honest?  I, on the other hand, have held that it can be rational for an individual to not adjust their own probability estimate in the direction of someone else who disagrees with them.

How can I maintain this position in the face of Aumann's Agreement Theorem, which proves that genuine Bayesians cannot have common knowledge of a dispute about probability estimates?  If genunie Bayesians will always agree with each other once they've exchanged probability estimates, shouldn't we Bayesian wannabes do the same?

To explain my reply, I begin with a metaphor:  If I have five different accurate maps of a city, they will all be consistent with each other.  Some philosophers, inspired by this, have held that "rationality" consists of having beliefs that are consistent among themselves.  But, although accuracy necessarily implies consistency, consistency does not necessarily imply accuracy.  If I sit in my living room with the curtains drawn, and make up five maps that are consistent with each other, but I don't actually walk around the city and make lines on paper that correspond to what I see, then my maps will be consistent but not accurate.  When genuine Bayesians agree in their probability estimates, it's not because they're trying to be consistent - Aumann's Agreement Theorem doesn't invoke any explicit drive on the Bayesians' part to be consistent.  That's what makes AAT surprising!  Bayesians only try to be accurate; in the course of seeking to be accurate, they end up consistent.  The Modesty Argument, that we can end up accurate in the course of seeking to be consistent, does not necessarily follow.

How can I maintain my position in the face of my admission that disputants will always improve their average score if they average together their individual probability distributions?

Suppose a creationist comes to me and offers:  "You believe that natural selection is true, and I believe that it is false.  Let us both agree to assign 50% probability to the proposition."  And suppose that by drugs or hypnosis it was actually possible for both of us to contract to adjust our probability estimates in this way.  This unquestionably improves our combined log-score, and our combined squared error.  If as a matter of altruism, I value the creationist's accuracy as much as my own - if my loss function is symmetrical around the two of us - then I should agree.  But what if I'm trying to maximize only my own individual accuracy?  In the former case, the question is absolutely clear, and in the latter case it is not absolutely clear, to me at least, which opens up the possibility that they are different questions.

If I agree to a contract with the creationist in which we both use drugs or hypnosis to adjust our probability estimates, because I know that the group estimate must be improved thereby, I regard that as pursuing the goal of social altruism.  It doesn't make creationism actually true, and it doesn't mean that I think creationism is true when I agree to the contract.  If I thought creationism was 50% probable, I wouldn't need to sign a contract - I would have already updated my beliefs!  It is tempting but false to regard adopting someone else's beliefs as a favor to them, and rationality as a matter of fairness, of equal compromise.  Therefore it is written:  "Do not believe you do others a favor if you accept their arguments; the favor is to you."  Am I really doing myself a favor by agreeing with the creationist to take the average of our probability distributions?

I regard rationality in its purest form as an individual thing - not because rationalists have only selfish interests, but because of the form of the only admissible question:  "Is is actually true?"  Other considerations, such as the collective accuracy of a group that includes yourself, may be legitimate goals, and an important part of human existence - but they differ from that single pure question.

In Aumann's Agreement Theorem, all the individual Bayesians are trying to be accurate as individuals.  If their explicit goal was to maximize group accuracy, AAT would not be surprising.  So the improvement of group score is not a knockdown argument as to what an individual should do if they are trying purely to maximize their own accuracy, and it is that last quest which I identify as rationality.  It is written:  "Every step of your reasoning must cut through to the correct answer in the same movement.  More than anything, you must think of carrying your map through to reflecting the territory.  If you fail to achieve a correct answer, it is futile to protest that you acted with propriety."  From the standpoint of social altruism, someone may wish to be Modest, and enter a drug-or-hypnosis-enforced contract of Modesty, even if they fail to achieve a correct answer thereby.

The central argument for Modesty proposes something like a Rawlsian veil of ignorance - how can you know which of you is the honest truthseeker, and which the stubborn self-deceiver?  The creationist believes that he is the sane one and you are the fool.  Doesn't this make the situation symmetric around the two of you?  If you average your estimates together, one of you must gain, and one of you must lose, since the shifts are in opposite directions; but by Jensen's inequality it is a positive-sum game.  And since, by something like a Rawlsian veil of ignorance, you don't know which of you is really the fool, you ought to take the gamble.  This argues that the socially altruistic move is also always the individually rational move.

And there's also the obvious reply:  "But I know perfectly well who the fool is.  It's the other guy.  It doesn't matter that he says the same thing - he's still the fool."

This reply sounds bald and unconvincing when you consider it abstractly.  But if you actually face a creationist, then it certainly feels like the correct answer - you're right, he's wrong, and you have valid evidence to know that, even if the creationist can recite exactly the same claim in front of a TV audience.

Robin Hanson sides with symmetry - this is clearest in his paper Uncommon Priors Require Origin Disputes - and therefore endorses the Modesty Argument.  (Though I haven't seen him analyze the particular case of the creationist.)

I respond:  Those who dream do not know they dream; but when you wake you know you are awake.  Dreaming, you may think you are awake.  You may even be convinced of it.  But right now, when you really are awake, there isn't any doubt in your mind - nor should there be.  If you, persuaded by the clever argument, decided to start doubting right now that you're really awake, then your Bayesian score would go down and you'd become that much less accurate.  If you seriously tried to make yourself doubt that you were awake - in the sense of wondering if you might be in the midst of an ordinary human REM cycle - then you would probably do so because you wished to appear to yourself as rational, or because it was how you conceived of "rationality" as a matter of moral duty.  Because you wanted to act with propriety.  Not because you felt genuinely curious as to whether you were awake or asleep.  Not because you felt you might really and truly be asleep.  But because you didn't have an answer to the clever argument, just an (ahem) incommunicable insight that you were awake.

Russell Wallace put it thusly:  "That we can postulate a mind of sufficiently low (dreaming) or distorted (insane) consciousness as to genuinely not know whether it's Russell or Napoleon doesn't mean I (the entity currently thinking these thoughts) could have been Napoleon, any more than the number 3 could have been the number 7. If you doubt this, consider the extreme case: a rock doesn't know whether it's me or a rock. That doesn't mean I could have been a rock."

There are other problems I see with the Modesty Argument, pragmatic matters of human rationality - if a fallible human tries to follow the Modesty Argument in practice, does this improve or disimprove personal rationality?  To me it seems that the adherents of the Modesty Argument tend to profess Modesty but not actually practice it.

For example, let's say you're a scientist with a controversial belief - like the Modesty Argument itself, which is hardly a matter of common accord - and you spend some substantial amount of time and effort trying to prove, argue, examine, and generally forward this belief.  Then one day you encounter the Modesty Argument, and it occurs to you that you should adjust your belief toward the modal belief of the scientific field.  But then you'd have to give up your cherished hypothesis.  So you do the obvious thing - I've seen at least two people do this on two different occasions - and say:  "Pursuing my personal hypothesis has a net expected utility to Science.  Even if I don't really believe that my theory is correct, I can still pursue it because of the categorical imperative: Science as a whole will be better off if scientists go on pursuing their own hypotheses."  And then they continue exactly as before.

I am skeptical to say the least.  Integrating the Modesty Argument as new evidence ought to produce a large effect on someone's life and plans.  If it's being really integrated, that is, rather than flushed down a black hole.  Your personal anticipation of success, the bright emotion with which you anticipate the confirmation of your theory, should diminish by literally orders of magnitude after accepting the Modesty Argument.  The reason people buy lottery tickets is that the bright anticipation of winning ten million dollars, the dancing visions of speedboats and mansions, is not sufficiently diminished - as a strength of emotion - by the probability factor, the odds of a hundred million to one.  The ticket buyer may even profess that the odds are a hundred million to one, but they don't anticipate it properly - they haven't integrated the mere verbal phrase "hundred million to one" on an emotional level.

So, when a scientist integrates the Modesty Argument as new evidence, should the resulting nearly total loss of hope have no effect on real-world plans originally formed in blessed ignorance and joyous anticipation of triumph?  Especially when you consider that the scientist knew about the social utility to start with, while making the original plans?  I think that's around as plausible as maintaining your exact original investment profile after the expected returns on some stocks change by a factor of a hundred.  What's actually happening, one naturally suspects, is that the scientist finds that the Modesty Argument has uncomfortable implications; so they reach for an excuse, and invent on-the-fly the argument from social utility as a way of exactly cancelling out the Modesty Argument and preserving all their original plans.

But of course if I say that this is an argument against the Modesty Argument, that is pure ad hominem tu quoque.  If its adherents fail to use the Modesty Argument properly, that does not imply it has any less force as logic.

Rather than go into more detail on the manifold ramifications of the Modesty Argument, I'm going to close with the thought experiment that initially convinced me of the falsity of the Modesty Argument.  In the beginning it seemed to me reasonable that if feelings of 99% certainty were associated with a 70% frequency of true statements, on average across the global population, then the state of 99% certainty was like a "pointer" to 70% probability.  But at one point I thought:  "What should an (AI) superintelligence say in the same situation?  Should it treat its 99% probability estimates as 70% probability estimates because so many human beings make the same mistake?"  In particular, it occurred to me that, on the day the first true superintelligence was born, it would be undeniably true that - across the whole of Earth's history - the enormously vast majority of entities who had believed themselves superintelligent would be wrong.  The majority of the referents of the pointer "I am a superintelligence" would be schizophrenics who believed they were God.

A superintelligence doesn't just believe the bald statement that it is a superintelligence - it presumably possesses a very detailed, very accurate self-model of its own cognitive systems, tracks in detail its own calibration, and so on.  But if you tell this to a mental patient, the mental patient can immediately respond:  "Ah, but I too possess a very detailed, very accurate self-model!"  The mental patient may even come to sincerely believe this, in the moment of the reply.  Does that mean the superintelligence should wonder if it is a mental patient?  This is the opposite extreme of Russell Wallace asking if a rock could have been you, since it doesn't know if it's you or the rock.

One obvious reply is that human beings and superintelligences occupy different classes - we do not have the same ur-priors, or we are not part of the same anthropic reference class; some sharp distinction renders it impossible to group together superintelligences and schizophrenics in probability arguments.  But one would then like to know exactly what this "sharp distinction" is, and how it is justified relative to the Modesty Argument.  Can an evolutionist and a creationist also occupy different reference classes?  It sounds astoundingly arrogant; but when I consider the actual, pragmatic situation, it seems to me that this is genuinely the case.

Or here's a more recent example - one that inspired me to write today's blog post, in fact.  It's the true story of a customer struggling through five levels of Verizon customer support, all the way up to floor manager, in an ultimately futile quest to find someone who could understand the difference between .002 dollars per kilobyte and .002 cents per kilobyte.  Audio [27 minutes], Transcript.  It has to be heard to be believed.  Sample of conversation:  "Do you recognize that there's a difference between point zero zero two dollars and point zero zero two cents?"  "No."

The key phrase that caught my attention and inspired me to write today's blog post is from the floor manager:  "You already talked to a few different people here, and they've all explained to you that you're being billed .002 cents, and if you take it and put it on your calculator... we take the .002 as everybody has told you that you've called in and spoken to, and as our system bills accordingly, is correct."

Should George - the customer - have started doubting his arithmetic, because five levels of Verizon customer support, some of whom cited multiple years of experience, told him he was wrong?  Should he have adjusted his probability estimate in their direction?  A straightforward extension of Aumann's Agreement Theorem to impossible possible worlds, that is, uncertainty about the results of computations, proves that, had all parties been genuine Bayesians with common knowledge of each other's estimates, they would have had the same estimate.  Jensen's inequality proves even more straightforwardly that, if George and the five levels of tech support had averaged together their probability estimates, they would have improved their average log score.  If such arguments fail in this case, why do they succeed in other cases?  And if you claim the Modesty Argument carries in this case, are you really telling me that if George had wanted only to find the truth for himself, he would have been wise to adjust his estimate in Verizon's direction?  I know this is an argument from personal incredulity, but I think it's a good one.

On the whole, and in practice, it seems to me like Modesty is sometimes a good idea, and sometimes not.  I exercise my individual discretion and judgment to decide, even knowing that I might be biased or self-favoring in doing so, because the alternative of being Modest in every case seems to me much worse.

But the question also seems to have a definite anthropic flavor.  Anthropic probabilities still confuse me; I've read arguments but I have been unable to resolve them to my own satisfaction.  Therefore, I confess, I am not able to give a full account of how the Modesty Argument is resolved.

Modest, aren't I?