Auto-Calibrate

Heuristics and biases studies show that people are usually too confident when they assign probability; in one study, people who assigned “98%” confidence levels to answers were wrong 46% of the time. Now, through the power of the Internet, you can check your calibration by answering trivia questions, assigning confidence levels, and then checking to see whether your confidence levels were justified. Below are five trivia quizzes, with calibration levels added; feel free to post your results in the comments section. Trivia questions are from the open source MisterHouse project.

Quiz #1

Quiz #2

Quiz #3

Quiz #4

Quiz #5

5 thoughts on “Auto-Calibrate

  1. I did quizzes one and two:

    Quiz q:
    96% confidence: 2 marked, 2 correct, success 100%
    90% confidence: 2 marked, 2 correct, success 100%
    75% confidence: 5 marked, 3 correct, success 60%
    50% confidence: 16 marked, 5 correct, success 32.25%
    25% confidence: 26 marked, 8 correct, success 30.77%

    Quiz 2:
    96% confidence: 2 marked, 1 correct, success 50%
    90% confidence: 2 marked, 2 correct, success 100%
    75% confidence: 3 marked, 2 correct, success 66.6%
    50% confidence: 16 marked, 8 correct, success 50%
    25% confidence: 27 marked, 8 correct, success 29.63%

    I was so embarrassed about missing a 96% on the second quiz that I went back and looked at my answers again, and discovered that I had only marked one answer as 96%. I also forgot to fill out one of the questions. It turns out that the quiz assumes you’re 96% confident by default if you don’t mark anything. That’s definitely the wrong behavior.

    Note: the likelihood ratios for 25%, 50%, 75%, 90%, and 96% confidence are 1, 3, 9, 27, and 72.

    Note 2: Since your confidence should never be below 25%, and should almost never be exactly 25%, people should always appear underconfident on the 25% answers.

  2. Incidentally, I have created a similar website (http://calibratedprobabilityassessment.org/). Though it lacks much content right now, and it only does binary questions (though this would not be too difficult to change). The only question set up there has the problem that your answers will not be indedpendent which will mess up the results. If would want to work on combining work, I would be happy to do so.

  3. Also, I looked at the MisterHouse project, and I don’t see how you got the trivia questions from it. It says it is a Home Automation system, which doesn’t seem to have anything to do with trivia. What am I missing?

  4. Pingback: Calibrated Probability Assessment.org « Good Morning, Economics

  5. I did the first two also, and here are my results:

    Quiz 1:
    96% confidence: 5 marked, 5 correct, success 100%
    90% confidence: 6 marked, 5 correct, success 83%
    75% confidence: 6 marked, 3 correct, success 50%
    50% confidence: 14 marked, 10 correct, success 71%
    25% confidence: 20 marked, 9 correct, success 45%

    Quiz 2:
    96% confidence: 5 marked, 5 correct, success 100%
    90% confidence: 4 marked, 4 correct, success 100%
    75% confidence: 5 marked, 3 correct, success 60%
    50% confidence: 14 marked, 7 correct, success 50%
    25% confidence: 22 marked, 8 correct, success 36%

    I thought for both of these that the questions were a little too hard. Choosing 25% confidence means I am basically “punting” (giving up) on the question, and I did this almost half the time. Several times though I was able to eliminate one answer with reasonable confidence, but usually I answered with 25% on those. Maybe it would make sense to have a 33% confidence level.

    John Salvatier cites the “hard easy effect” where people tend to be over-confident on hard question and under-confident on easy questions. This manifests as close to 100% correctness on 90%-confident answers, and lower than 50% correctness on 50%-confident answers. I had read about that before doing these and so I tried to defend against it.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>