The Evidence Spectrum

If you look into things like religion and parapsychology and UFOs, you will notice people argue about a lot of circumstantial, dubious pieces of evidence. One approach here is to just try to judge each piece of evidence individually, then weigh them all. It’s important for there to be people looking at the details. But this approach leaves you open to systematic errors. A faster, more sloppy approach is to take a step back and compare the spectrum of evidence you see to the characteristic spectrum of evidence that you might expect.

  • The characteristic spectrum of a real class of phenomena includes a lot of evidence that’s too ambiguous to be of much use, plus some evidence that looks reasonably convincing, plus a little bit (or a lot) of flaming-letters-on-the-moon type undeniable evidence.
  • The characteristic spectrum of something that’s not real but that has had a lot of intellectual energy put into its defense includes a lot of ambiguous evidence and maybe a little bit of reasonably-convincing-looking evidence.
  • The characteristic spectrum of something real that’s half-heartedly or incompetently trying to hide itself might be a lot like that, too.
  • The characteristic spectrum of something real that’s actively and competently trying to hide itself is flat.

In most cases you can just look at the far end of the spectrum, where evidence is incontrovertible. I would say that this tactic is not perfect, but is quick and often works.

God Hates Implicature

(Follow-up to Evolution and God are Incompatible)

Paul Grice claimed that in conversations we assume a “cooperative principle“: when we try to figure out what people mean, we assume they are working together to make the exchange as useful as possible. He also stated some more specific maxims, such as “avoid ambiguity” and “make your contribution as informative as is required for the current purposes of the exchange”.

There are many parts of the Bible, like the creation story in Genesis, that the last few centuries of science have shown to be literally false. Some Christians are Biblical literalists and just deny the evidence. Other Christians think things like the creation story in Genesis should be taken metaphorically.

Strangely enough I have less respect for the latter view than the former. Being flat wrong is one thing, being confused about subtle philosophical points is often more dangerous.

The default presumption for any statement is that it’s a literal truth claim. If I say my aunt lives in Ireland, then unless context, background information, tone of voice, and so on suggest otherwise, I am trying to make you believe that my aunt literally lives in Ireland. It’s not as if saying my aunt lives in Ireland helps you understand a philosophical point or moves your spirit.

Saying God made the world in a week is like that. That the creation story contradicts scientific evidence is something that hasn’t been widely known until the 19th or 20th century. If 1) God exists, and 2) we can hold God to higher conversational standards than the average internet troll, and 3) Genesis is not literally true, then it follows God would have inserted some sort of warning to guard against misinterpretation. If you want to keep 3) — as you should — then you have to throw away 1) or 2).

My Favorite Conspiracy Theory

Did Shakespeare (the Stratford guy) write his own work? At various times it’s been suggested that “Shakespeare” was secretly someone else, like Francis Bacon, Edward de Vere, or Christopher Marlowe. Conspiracies are always implausible, and the experts mostly seem to be what they call “Stratfordians”. So as someone who knows nothing about this stuff, normally I’d say, that’s that.

Peter Farey has a web page claiming that the text on the monument near Shakespeare’s grave, if properly deciphered, says that Marlowe wrote Shakespeare’s works after faking his own death. It’s worth the read, even if only as an ingenious and creative misinterpretation. It begins:

So many ludicrous cryptograms have been offered as an alleged proof that someone other than William Shakespeare of Stratford-upon-Avon wrote the works attributed to him, that anyone attempting to suggest something even remotely along these lines is bound to receive a fairly cool reception. I would, therefore, have much preferred to be writing about some quite different discovery, but that which follows was what I actually found. Unfortunately, if one is looking for the truth, the evidence that crops up is not always of the type that one would have ideally picked.

The article has some probability calculations that purport to show it can’t be chance. I find these calculations naive, but they look like they could be loosened to something less naive and still work. So on the one hand, intuitively it seems like strong evidence. On the other hand, human brains are really good at finding secret messages where there aren’t any. And there’s a low prior for the evidence to overcome. (Though not for Farey himself, who already believed the faked death theory before the whole monument business).

So I’m not quite sure what and how to think here (perhaps some sort of Bayesian analysis would help, but it’s tricky.) Other than for amusement value, I’ve never been into conspiracy theories. But I’m not sure I’d dismiss this one easily. (And yes, I know that none of this has any real implications, but it’s interesting and perhaps a good case study.)

Bayes Bayes Revolution, Aumania

Writing the Aumann game posts got me thinking about games designed to promote rationality. Nick Bostrom and commenters discussed the topic on Overcoming Bias. Tom McCabe had a post about rational debating. In this post I’ll brainstorm about two ideas I’ve had for computer/video games.

First, Bayes Bayes Revolution.

The basic tool here is a probability slider that you move left/right with two keys or a mouse. It runs from 0 to 1, but probably not in a linear way. There should be more distance between 0.01 and 0.02 than 0.50 and 0.51, just for convenience. And maybe the results of your setting the slider should show up on a view-only linear scale so your intuitions don’t get warped the same way. Maybe a two-color pie diagram.

The game starts out posing you a question, maybe whether a hidden animal is red or green. You have some initial guess for the probability, either from having played the game before (50-50 if you haven’t), or because the game told you.

Then, one by one, at a rate depending on difficulty level, the game gives you pieces of evidence. You might see a blue ball flying across the screen, and you might know (again, from experience or because you’re told) that red animals throw blue balls three times as often as green animals. If your initial probability for a red animal was, say, 0.2, then if you crank the Reverend you find:

p(red animal | blue ball) = p(red animal) * p(blue ball | red animal) / p(blue ball) = 0.2 * 3q / (0.2 * 3q + 0.8 * q) = 3/7.

The game scores you based on how close you put the slider to this correct answer. Then when the next piece of evidence comes it scores you based on how close you put the slider to the correct answer after taking into account both pieces of evidence. And so on. From what I understand about existing games of this genre, they rate you “OK” or “excellent” or whatever each time according to some formula. (You’re being scored here on following the right procedure rather than placing a high probability on the right outcome; that would be cleaner in a way, but would introduce a luck factor.)

Starting from there, you can make the game as simple or complicated as you want. It can be spartan and abstract or full of bells and whistles, using simple independent drawings from an urn or bombarding the player with baroque causal diagrams.

In the end it would be like following a sort of dance routine. But the Bayes Dance is a very special kind of dance, because if you do it right, you never have any idea where you’re going. In fact, to those who can’t hear the Music of the Evidence, the Bayes Dance looks exactly like a random walk. Just like “if we knew what we were doing, it wouldn’t be research”, one can say that “if we knew where we were going, it wouldn’t be Bayes”. Take a Bayesian to a regular dance lesson, and he will say, “if I already know I’m going to have to step to the left, then that must be a better place for my foot, so why not put it there to start with?” Or he will say, “my leg has to stay here on average, so if you’re that sure it’s going to the left, then in the unlikely case that it does go to the right, it will have to fly off, all the way out of the window”.

So in Bayes Bayes Revolution, as indeed in life, there is no predictable routine you can practice. All you can do is align your gut feelings with the math so they can work with any input.

not photoshopped!

Second, Aumania.

I seem to have turned BBR into a rationality lecture more than an actual game idea. I guess if you really hated rationality lectures, you wouldn’t come here. Aumania is a different game idea, one that greatly resembles the Aumann game we’ve played on this blog before, and unlike BBR I could see it being quite enjoyable.

To stay with the metaphor from earlier on a little longer: Aumann’s agreement theorem says that if two Bayes dancers have gone out of synch because they heard different music, then when they watch each other after the music stops (or perhaps between notes if they’re quick), they will do a sequence of steps, each reacting to the last, that ends in them gradually merging back and standing still.

Unlike BBR, Aumania is necessarily multi-player. The point of the game is to react smartly to the opinions of others.

Here’s how it works. Each player has an individual probability slider, like in BBR. Maybe they’re arranged in a circle on the screen. The game presents the players with randomly generated claims, perhaps like the ones we used for the Aumann game. Players have maybe ten seconds to move their slider. The key is that they should take special note of how all the other sliders are being set. When time is up, the answer is revealed and everyone gets scored.

Your score is the logarithm of the probability you put on the correct answer. That way, entering your true subjective probability maximizes your expected score. Score, defined this way, is always a negative number; the game could either accept this, or transform back to percentages, or give players a fixed amount of free points for every question, or whatever.

Score could also map to hit point damage. Learning to be rational so you can help humankind find and act on true beliefs is nice and all, but learning to be rational because else Mario gets eaten just before killing that level 6 boss, that’s motivation.

For every setting on the probability slider there is a corresponding score size if the claim is true and if the claim is false. These could be displayed as two bars to show the trade-offs you’re making.

Aumania is partly about knowing your trivia, but not as much as you might think. Players will do better in groups with much knowledge dispersed in them than in groups with little. But the world’s most ignorant person can score the same as the world’s most knowledgeable person, just by always copying the probability. If he’s also the world’s most stubborn person, that’s when he has a problem.

I’m not sure whether Aumania would work best as a cooperative game (players maximize the group’s total score), an independent game (players maximize their own score), or a competitive game (players minimize the rank order of their score compared to the other players). In a cooperative or competitive game, there might be incentives for deception, which means Aumann’s theorem won’t work. Unpredictable deadlines might help.

Again, you can add as many complications as you want. Random pieces of evidence, as in BBR, would be one possibility. Other means of communicating evidence, such as chat, would be another.

If we can teach the next generation to update beliefs as smoothly as this, our species will be utterly unstoppable.

The New Adventures of Aumann

I should still say something about lessons from the last Aumann Game, but in the mean time here are some new claims. Remember, the object is to maximize the sum of the logarithms of the probabilities you assign to the right answers. Putting a .1 probability on a true claim means you get one negative cookie, putting a .01 probability on a true claim means you get two negative cookies, and so on. In the best case you get no cookies. To play, post your honest estimates and update them based on those posted by others. Initial scores will reflect your calibration and general knowledge/judgment. Later scores will reflect your calibration, your ability to correctly weigh the opinions of others, and possibly still your general knowledge/judgment.

Where possible I usually added the question before knowing the answer. I’d be curious to know if there are some kinds of question you think are particularly (un)suitable for this. It probably wouldn’t be too hard to generate these automatically and write a human brain calibrator program.

First, some more statistics claims, using NationMaster and StateMaster.

1. Belgium has more inhabitants than Sweden.

2. Bolivia has a greater total GDP (PPP) than Kenya.

3. Life expectancy at birth is greater in Israel than in Uruguay.

4. Hungary has more land area than Utah.

5. South Dakota has more murder and nonnegligent manslaughter per capita than Vermont.

6. Louisiana consumes more oil than Washington (the state).

7. California has a greater fraction of male inhabitants than Kentucky.

8. Austrians have won more total gold medals in the summer Olympics (all time) than New Zealanders.

9. Argentina is more urbanized than Latvia.

10. There are more tractors in use in Belarus than Zimbabwe (2000).

Some pure probability theory questions.

11. I will shuffle a deck of cards, take the first ten cards, and multiply the numbers on them. Face cards count as 1. Claim: The result will be over one hundred thousand.

12. I will shuffle a deck of cards and flip them over one by one. Claim: I will reach the third 8 or the third 9 before I reach the second 3.

13. I will add the results of ten 6-sided dice. Claim: The result will be at least 43.

14. I will flip a coin until I get heads four times in a row or tails five times in a row. Claim: This will happen before or on the 30th flip.

15. I will place two black rooks, two black bishops, a black queen, five white pawns, and a white king on a chessboard at random (no pawns on the first or last row). Claim: the white king will be in check.

Times and distances.

16. Cairo and Bombay are farther apart than Vancouver and Mexico City.

17. London and Dublin are farther apart than Milwaukee and Philadelphia.

18. Bucharest and Prague are farther apart than Sao Paulo and Rio de Janeiro.

19. Jerusalem and San Francisco are farther apart than Sydney and Helsinki.

20. Nietzsche died before Wagner.

21. Eisenhower died before Truman.

22. Leonidas died before Socrates.

23. Caesar was born before Virgil.

24. Aristotle was born before Ptolemy.

25. Shakespeare died before Newton was born.

26. Mao Zedong died before Britney Spears was born.

27. The Chinese invented paper before the Romans invaded Britain.

28. Buddhism was introduced in Japan before Norwegians started colonizing Iceland.

29. Ankara was founded before Kiev.

30. The Eighth Crusade happened before the War of the Roses.

Some internet questions. Google searches are including quote marks.

31. I will choose a random message from the Extropians mailing list from January 1998. Claim: The message body (including any quoted text) will contain the word “technology”.

32. Same as previous claim, but for SL4, January 2008, and “maybe”.

33. Googling “cow” gives more search results than “horse”.

34. Googling “Paris Hilton” gives more search results than “Eiffel Tower”.

35. Googling “Oort cloud” gives more search results than “Kuiper belt”.

36. Googling “Lance Armstrong” gives more search results than “Neil Armstrong”.

37. Googling “John McCain” gives more search results than “Michael Jackson”.

38. This blog gets more visitors from India than Ireland.

39. Gandalf has a longer Wikipedia page than James Randi.

40. Robert Aumann has a longer Wikipedia page than Spiderman.