The Martial Art of Rationality
I often use the metaphor that rationality is the martial art of mind. You don't need huge, bulging muscles to learn martial arts - there's a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. Some human beings are faster or stronger than others; but a martial art does not train the variance between humans. A martial art just trains your muscles - if you have the human-universal complex machinery of a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist. How can you train a variance? What does it mean to train +2 standard deviations of muscle? It's equally unclear what it would mean to train an IQ of 132. But if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you're a fast learner, you might learn faster - but the art of rationality isn't about that; it's about training brain machinery we all have in common.
Alas, our minds respond less readily to our will than our hands. Muscles are evolutionarily ancient subjects of neural control, while cognitive reflectivity is a comparatively more recent innovation. We shouldn't be surprised that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth.
If you live in an urban area, you probably don't need to walk very far to find a martial arts dojo. Why aren't there dojos that teach rationality? One reason, perhaps, is that it's harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so. Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student. It is also harder to give impressive public exhibitions of rationality. This may partially explain why there are no rationality dojos as yet.
Very recently - in just the last few decades - the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.
Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of Artificial General Intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declared goals in using it. Deliberately we decide that we want to seek only the truth; but our brains have hardwired support for rationalizing falsehoods. We can try to compensate for what we choose to regard as flaws of the machinery; but we can't actually rewire the neural circuitry. Nor may a martial artist plate titanium over his bones - not today, at any rate.
Trying to synthesize a personal art of rationality, using the science of rationality, may prove awkward: One imagines trying to invent a martial art using an abstract theory of physics, game theory, and human anatomy. But humans are not reflectively blind; we do have a native instinct for introspection. The inner eye is not sightless; but it sees blurrily, with systematic distortions. We need, then, to apply the science to our intuitions, to use the abstract knowledge to correct our mental movements and augment our metacognitive skills. We are not writing a computer program to make a string puppet execute martial arts forms; it is our own mental limbs that we must move. Therefore we must connect theory to practice. We must come to see what the science means, for ourselves, for our daily inner life.
And we must, above all, figure out how to communicate the skill; which may not be a matter for declarative statements alone. Martial artists spar with each other, execute standard forms, and are watched throughout by their teachers. Calculus students do homework, and check their answers. Olympic runners continually try to beat their best previous time, as measured by a stopwatch.
How to communicate procedural skills of rationality, or measure them, is probably the single largest open issue that stands between humanity and rationality dojos - at least it's the part of the problem that most baffles me. Meanwhile I lecture. So does anyone out there have ideas?
November 26, 2006
Why truth? And...
Some of the comments in this blog have touched on the question of why we ought to seek truth. (Thankfully not many have questioned what truth is.) Our shaping motivation for configuring our thoughts to rationality, which determines whether a given configuration is "good" or "bad", comes from whyever we wanted to find truth in the first place.
It is written: "The first virtue is curiosity." Curiosity is one reason to seek truth, and it may not be the only one, but it has a special and admirable purity. If your motive is curiosity, you will assign priority to questions according to how the questions, themselves, tickle your personal aesthetic sense. A trickier challenge, with a greater probability of failure, may be worth more effort than a simpler one, just because it is more fun.
Some people, I suspect, may object that curiosity is an emotion and is therefore "not rational". I label an emotion as "not rational" if it rests on mistaken beliefs, or rather, on irrational epistemic conduct: "If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, the Way opposes your calm." Conversely, then, an emotion which is evoked by correct beliefs or epistemically rational thinking is a "rational emotion"; and this has the advantage of letting us regard calm as an emotional state, rather than a privileged default. When people think of "emotion" and "rationality" as opposed, I suspect that they are really thinking of System 1 and System 2 - fast perceptual judgments versus slow deliberative judgments. Deliberative judgments aren't always true, and perceptual judgments aren't always false; so it is very important to distinguish that dichotomy from "rationality". Both systems can serve the goal of truth, or defeat it, according to how they are used.
Besides sheer emotional curiosity, what other motives are there for desiring truth? Well, you might want to accomplish some specific real-world goal, like building an airplane, and therefore you need to know some specific truth about aerodynamics. Or more mundanely, you want chocolate milk, and therefore you want to know whether the local grocery has chocolate milk, so you can choose whether to walk there or somewhere else. If this is the reason you want truth, then the priority you assign to your questions will reflect the expected utility of their information - how much the possible answers influence your choices, how much your choices matter, and how much you expect to find an answer that changes your choice from its default.
To seek truth merely for its instrumental value may seem impure - should we not desire the truth for its own sake? - but such investigations are extremely important because they create an outside criterion of verification: if your airplane drops out of the sky, or if you get to the store and find no chocolate milk, it's a hint that you did something wrong. You get back feedback on which modes of thinking work, and which don't. Pure curiosity is a wonderful thing, but it may not linger too long on verifying its answers, once the attractive mystery is gone. Curiosity, as a human emotion, has been around since long before the ancient Greeks. But what set humanity firmly on the path of Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world. As far as sheer curiosity goes, spinning campfire tales of gods and heroes satisfied that desire just as well, and no one realized that anything was wrong with that.
Are there motives for seeking truth besides curiosity and pragmatism? The third reason that I can think of is morality: You believe that to seek the truth is noble and important and worthwhile. Though such an ideal also attaches an intrinsic value to truth, it's a very different state of mind from curiosity. Being curious about what's behind the curtain doesn't feel the same as believing that you have a moral duty to look there. In the latter state of mind, you are a lot more likely to believe that someone else should look behind the curtain, too, or castigate them if they deliberately close their eyes. For this reason, I would also label as "morality" the belief that truthseeking is pragmatically important to society, and therefore is incumbent as a duty upon all. Your priorities, under this motivation, will be determined by your ideals about which truths are most important (not most useful or most intriguing); or your moral ideals about when, under what circumstances, the duty to seek truth is at its strongest.
I tend to be suspicious of morality as a motivation for rationality, not because I reject the moral ideal, but because it invites certain kinds of trouble. It is too easy to acquire, as learned moral duties, modes of thinking that are dreadful missteps in the dance. Consider Mr. Spock of Star Trek, a naive archetype of rationality. Spock's emotional state is always set to "calm", even when wildly inappropriate. He often gives many significant digits for probabilities that are grossly uncalibrated. (E.g: "Captain, if you steer the Enterprise directly into that black hole, our probability of surviving is only 2.234%" Yet nine times out of ten the Enterprise is not destroyed. What kind of tragic fool gives four significant digits for a figure that is off by two orders of magnitude?) Yet this popular image is how many people conceive of the duty to be "rational" - small wonder that they do not embrace it wholeheartedly. To make rationality into a moral duty is to give it all the dreadful degrees of freedom of an arbitrary tribal custom. People arrive at the wrong answer, and then indignantly protest that they acted with propriety, rather than learning from their mistake.
And yet if we're going to improve our skills of rationality, go beyond the standards of performance set by hunter-gatherers, we'll need deliberate beliefs about how to think with propriety. When we write new mental programs for ourselves, they start out in System 2, the deliberate system, and are only slowly - if ever - trained into the neural circuitry that underlies System 1. So if there are certain kinds of thinking that we find we want to avoid - like, say, biases - it will end up represented, within System 2, as an injunction not to think that way; a professed duty of avoidance.
If we want the truth, we can most effectively obtain it by thinking in certain ways, rather than others; and these are the techniques of rationality. Some of the techniques of rationality involve overcoming a certain class of obstacles, the biases...
(Continued in next post: "What's a bias, again?")
November 26, 2006
...What's a bias, again?
(Continued from previous post: "Why truth? And...")
A bias is a certain kind of obstacle to our goal of obtaining truth - its character as an "obstacle" stems from this goal of truth - but there are many obstacles that are not "biases".
If we start right out by asking "What is bias?", it comes at the question in the wrong order. As the proverb goes, "There are forty kinds of lunacy but only one kind of common sense." The truth is a narrow target, a small region of configuration space to hit. "She loves me, she loves me not" may be a binary question, but E=MC^2 is a tiny dot in the space of all equations, like a winning lottery ticket in the space of all lottery tickets. Error is not an exceptional condition; it is success which is a priori so improbable that it requires an explanation.
We don't start out with a moral duty to "reduce bias", because biases are bad and evil and Just Not Done. This is the sort of thinking someone might end up with if they acquired a deontological duty of "rationality" by social osmosis, which leads to people trying to execute techniques without appreciating the reason for them. (Which is bad and evil and Just Not Done, according to Surely You're Joking, Mr. Feynman, which I read as a kid.)
Rather, we want to get to the truth, for whatever reason, and we find various obstacles getting in the way of our goal. These obstacles are not wholly dissimilar to each other - for example, there are obstacles that have to do with not having enough computing power available, or information being expensive. It so happens that a large group of obstacles seem to have a certain character in common - to cluster in a region of obstacle-to-truth space - and this cluster has been labeled "biases".
What is a bias? Can we look at the empirical cluster and find a compact test for membership? Perhaps we will find that we can't really give any explanation better than pointing to a few extensional examples, and hoping the listener understands. If you are a scientist just beginning to investigate fire, it might be a lot wiser to point to a campfire and say "Fire is that orangey-bright hot stuff over there," rather than saying "I define fire as an alchemical transmutation of substances which releases phlogiston." As I said in The Simple Truth, you should not ignore something just because you can't define it. I can't quote the equations of General Relativity from memory, but nonetheless if I walk off a cliff, I'll fall. And we can say the same of biases - they won't hit any less hard if it turns out we can't define compactly what a "bias" is. So we might point to conjunction fallacies, to overconfidence, to the availability and representativeness heuristics, to base rate neglect, and say: "Stuff like that."
With all that said, we seem to label as "biases" those obstacles to truth which are produced, not by the cost of information, nor by limited computing power, but by the shape of our own mental machinery. For example, the machinery is evolutionarily optimized to purposes that actively oppose epistemic accuracy; for example, the machinery to win arguments in adaptive political contexts. Or the selection pressure ran skew to epistemic accuracy; for example, believing what others believe, to get along socially. Or, in the classic heuristic-and-bias, the machinery operates by an identifiable algorithm that does some useful work but also produces systematic errors: the availability heuristic is not itself a bias, but it gives rise to identifiable, compactly describable biases. Our brains are doing something wrong, and after a lot of experimentation and/or heavy thinking, someone identifies the problem in a fashion that System 2 can comprehend; then we call it a "bias". Even if we can do no better for knowing, it is still a failure that arises, in an identifiable fashion, from a particular kind of cognitive machinery - not from having too little machinery, but from the shape of the machinery itself.
"Biases" are distinguished from errors that arise from cognitive content, such as adopted beliefs, or adopted moral duties. These we call "mistakes", rather than "biases", and they are much easier to correct, once we've noticed them for ourselves. (Though the source of the mistake, or the source of the source of the mistake, may ultimately be some bias.)
"Biases" are distinguished from errors that arise from damage to an individual human brain, or from absorbed cultural mores; biases arise from machinery that is humanly universal.
Plato wasn't "biased" because he was ignorant of General Relativity - he had no way to gather that information, his ignorance did not arise from the shape of his mental machinery. But if Plato believed that philosophers would make better kings because he himself was a philosopher - and this belief, in turn, arose because of a universal adaptive political instinct for self-promotion, and not because Plato's daddy told him that everyone has a moral duty to promote their own profession to governorship, or because Plato sniffed too much glue as a kid - then that was a bias, whether Plato was ever warned of it or not.
Biases may not be cheap to correct. They may not even be correctable. But where we look upon our own mental machinery and see a causal account of an identifiable class of errors; and when the problem seems to come from the evolved shape of the machinery, rather from there being too little machinery, or bad specific content; then we call that a bias.
Personally, I see our quest in terms of acquiring personal skills of rationality, in improving truthfinding technique. The challenge is to attain the positive goal of truth, not to avoid the negative goal of failure. Failurespace is wide, infinite errors in infinite variety. It is difficult to describe so huge a space: "What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world." Success-space is narrower, and therefore more can be said about it.
While I am not averse (as you can see) to discussing definitions, we should remember that is not our primary goal. We are here to pursue the great human quest for truth: for we have desperate need of the knowledge, and besides, we're curious. To this end let us strive to overcome whatever obstacles lie in our way, whether we call them "biases" or not.
December 01, 2006
The Proper Use of Humility
It is widely recognized that good science requires some kind of humility. What sort of humility is more controversial.
Consider the creationist who says: "But who can really know whether evolution is correct? It is just a theory. You should be more humble and open-minded." Is this humility? The creationist practices a very selective underconfidence, refusing to integrate massive weights of evidence in favor of a conclusion he finds uncomfortable. I would say that whether you call this "humility" or not, it is the wrong step in the dance.
What about the engineer who humbly designs fail-safe mechanisms into machinery, even though he's damn sure the machinery won't fail? This seems like a good kind of humility to me. Historically, it's not unheard-of for an engineer to be damn sure a new machine won't fail, and then it fails anyway.
What about the student who humbly double-checks the answers on his math test? Again I'd categorize that as good humility.
What about a student who says, "Well, no matter how many times I check, I can't ever be certain my test answers are correct," and therefore doesn't check even once? Even if this choice stems from an emotion similar to the emotion felt by the previous student, it is less wise.
You suggest studying harder, and the student replies: "No, it wouldn't work for me; I'm not one of the smart kids like you; nay, one so lowly as myself can hope for no better lot." This is social modesty, not humility. It has to do with regulating status in the tribe, rather than scientific process. If you ask someone to "be more humble", by default they'll associate the words to social modesty - which is an intuitive, everyday, ancestrally relevant concept. Scientific humility is a more recent and rarefied invention, and it is not inherently social. Scientific humility is something you would practice even if you were alone in a spacesuit, light years from Earth with no one watching. Or even if you received an absolute guarantee that no one would ever criticize you again, no matter what you said or thought of yourself. You'd still double-check your calculations if you were wise.
The student says: "But I've seen other students double-check their answers and then they still turned out to be wrong. Or what if, by the problem of induction, 2 + 2 = 5 this time around? No matter what I do, I won't be sure of myself." It sounds very profound, and very modest. But it is not coincidence that the student wants to hand in the test quickly, and go home and play video games.
The end of an era in physics does not always announce itself with thunder and trumpets; more often it begins with what seems like a small, small flaw... But because physicists have this arrogant idea that their models should work all the time, not just most of the time, they follow up on small flaws. Usually, the small flaw goes away under closer inspection. Rarely, the flaw widens to the point where it blows up the whole theory. Therefore it is written: "If you do not seek perfection you will halt before taking your first steps."
But think of the social audacity of trying to be right all the time! I seriously suspect that if Science claimed that evolutionary theory is true most of the time but not all of the time - or if Science conceded that maybe on some days the Earth is flat, but who really knows - then scientists would have better social reputations. Science would be viewed as less confrontational, because we wouldn't have to argue with people who say the Earth is flat - there would be room for compromise. When you argue a lot, people look upon you as confrontational. If you repeatedly refuse to compromise, it's even worse. Consider it as a question of tribal status: scientists have certainly earned some extra status in exchange for such socially useful tools as medicine and cellphones. But this social status does not justify their insistence that only scientific ideas on evolution be taught in public schools. Priests also have high social status, after all. Scientists are getting above themselves - they won a little status, and now they think they're chiefs of the whole tribe! They ought to be more humble, and compromise a little.
Many people seem to possess rather hazy views of "rationalist humility". It is dangerous to have a prescriptive principle which you only vaguely comprehend; your mental picture may have so many degrees of freedom that it can adapt to justify almost any deed. Where people have vague mental models that can be used to argue anything, they usually end up believing whatever they started out wanting to believe. This is so convenient that people are often reluctant to give up vagueness. But the purpose of our ethics is to move us, not be moved by us.
"Humility" is a virtue that is often misunderstood. This doesn't mean we should discard the concept of humility, but we should be careful using it. It may help to look at the actions recommended by a "humble" line of thinking, and ask: "Does acting this way make you stronger, or weaker?" If you think about the problem of induction as applied to a bridge that needs to stay up, it may sound reasonable to conclude that nothing is certain no matter what precautions are employed; but if you consider the real-world difference between adding a few extra cables, and shrugging, it seems clear enough what makes the stronger bridge.
The vast majority of appeals that I witness to "rationalist's humility" are excuses to shrug. The one who buys a lottery ticket, saying, "But you can't know that I'll lose." The one who disbelieves in evolution, saying, "But you can't prove to me that it's true." The one who refuses to confront a difficult-looking problem, saying, "It's probably too hard to solve." The problem is motivated skepticism aka disconfirmation bias - more heavily scrutinizing assertions that we don't want to believe. Humility, in its most commonly misunderstood form, is a fully general excuse not to believe something; since, after all, you can't be sure. Beware of fully general excuses!
A further problem is that humility is all too easy to profess. Dennett, in "Breaking the Spell", points out that while many religious assertions are very hard to believe, it is easy for people to believe that they ought to believe them. Dennett terms this "belief in belief". What would it mean to really assume, to really believe, that three is equal to one? It's a lot easier to believe that you should, somehow, believe that three equals one, and to make this response at the appropriate points in church. Dennett suggests that much "religious belief" should be studied as "religious profession" - what people think they should believe and what they know they ought to say.
It is all too easy to meet every counterargument by saying, "Well, of course I could be wrong." Then, having dutifully genuflected in the direction of Modesty, having made the required obeisance, you can go on about your way without changing a thing.
The temptation is always to claim the most points with the least effort. The temptation is to carefully integrate all incoming news in a way that lets us change our beliefs, and above all our actions, as little as possible. John Kenneth Galbraith said: "Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof." And the greater the inconvenience of changing one's mind, the more effort people will expend on the proof.
But y'know, if you're gonna do the same thing anyway, there's no point in going to such incredible lengths to rationalize it. Often I have witnessed people encountering new information, apparently accepting it, and then carefully explaining why they are going to do exactly the same thing they planned to do previously, but with a different justification. The point of thinking is to shape our plans; if you're going to keep the same plans anyway, why bother going to all that work to justify it? When you encounter new information, the hard part is to update, to react, rather than just letting the information disappear down a black hole. And humility, properly misunderstood, makes a wonderful black hole - all you have to do is admit you could be wrong. Therefore it is written: "To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty."
December 10, 2006
The Modesty Argument
The Modesty Argument states that when two or more human beings have common knowledge that they disagree about a question of simple fact, they should each adjust their probability estimates in the direction of the others'. (For example, they might adopt the common mean of their probability distributions. If we use the logarithmic scoring rule, then the score of the average of a set of probability distributions is better than the average of the scores of the individual distributions, by Jensen's inequality.)
Put more simply: When you disagree with someone, even after talking over your reasons, the Modesty Argument claims that you should each adjust your probability estimates toward the other's, and keep doing this until you agree. The Modesty Argument is inspired by Aumann's Agreement Theorem, a very famous and oft-generalized result which shows that genuine Bayesians literally cannot agree to disagree; if genuine Bayesians have common knowledge of their individual probability estimates, they must all have the same probability estimate. ("Common knowledge" means that I know you disagree, you know I know you disagree, etc.)
I've always been suspicious of the Modesty Argument. It's been a long-running debate between myself and Robin Hanson.
Robin seems to endorse the Modesty Argument in papers such as Are Disagreements Honest? I, on the other hand, have held that it can be rational for an individual to not adjust their own probability estimate in the direction of someone else who disagrees with them.
How can I maintain this position in the face of Aumann's Agreement Theorem, which proves that genuine Bayesians cannot have common knowledge of a dispute about probability estimates? If genunie Bayesians will always agree with each other once they've exchanged probability estimates, shouldn't we Bayesian wannabes do the same?
To explain my reply, I begin with a metaphor: If I have five different accurate maps of a city, they will all be consistent with each other. Some philosophers, inspired by this, have held that "rationality" consists of having beliefs that are consistent among themselves. But, although accuracy necessarily implies consistency, consistency does not necessarily imply accuracy. If I sit in my living room with the curtains drawn, and make up five maps that are consistent with each other, but I don't actually walk around the city and make lines on paper that correspond to what I see, then my maps will be consistent but not accurate. When genuine Bayesians agree in their probability estimates, it's not because they're trying to be consistent - Aumann's Agreement Theorem doesn't invoke any explicit drive on the Bayesians' part to be consistent. That's what makes AAT surprising! Bayesians only try to be accurate; in the course of seeking to be accurate, they end up consistent. The Modesty Argument, that we can end up accurate in the course of seeking to be consistent, does not necessarily follow.
How can I maintain my position in the face of my admission that disputants will always improve their average score if they average together their individual probability distributions?
Suppose a creationist comes to me and offers: "You believe that natural selection is true, and I believe that it is false. Let us both agree to assign 50% probability to the proposition." And suppose that by drugs or hypnosis it was actually possible for both of us to contract to adjust our probability estimates in this way. This unquestionably improves our combined log-score, and our combined squared error. If as a matter of altruism, I value the creationist's accuracy as much as my own - if my loss function is symmetrical around the two of us - then I should agree. But what if I'm trying to maximize only my own individual accuracy? In the former case, the question is absolutely clear, and in the latter case it is not absolutely clear, to me at least, which opens up the possibility that they are different questions.
If I agree to a contract with the creationist in which we both use drugs or hypnosis to adjust our probability estimates, because I know that the group estimate must be improved thereby, I regard that as pursuing the goal of social altruism. It doesn't make creationism actually true, and it doesn't mean that I think creationism is true when I agree to the contract. If I thought creationism was 50% probable, I wouldn't need to sign a contract - I would have already updated my beliefs! It is tempting but false to regard adopting someone else's beliefs as a favor to them, and rationality as a matter of fairness, of equal compromise. Therefore it is written: "Do not believe you do others a favor if you accept their arguments; the favor is to you." Am I really doing myself a favor by agreeing with the creationist to take the average of our probability distributions?
I regard rationality in its purest form as an individual thing - not because rationalists have only selfish interests, but because of the form of the only admissible question: "Is is actually true?" Other considerations, such as the collective accuracy of a group that includes yourself, may be legitimate goals, and an important part of human existence - but they differ from that single pure question.
In Aumann's Agreement Theorem, all the individual Bayesians are trying to be accurate as individuals. If their explicit goal was to maximize group accuracy, AAT would not be surprising. So the improvement of group score is not a knockdown argument as to what an individual should do if they are trying purely to maximize their own accuracy, and it is that last quest which I identify as rationality. It is written: "Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory. If you fail to achieve a correct answer, it is futile to protest that you acted with propriety." From the standpoint of social altruism, someone may wish to be Modest, and enter a drug-or-hypnosis-enforced contract of Modesty, even if they fail to achieve a correct answer thereby.
The central argument for Modesty proposes something like a Rawlsian veil of ignorance - how can you know which of you is the honest truthseeker, and which the stubborn self-deceiver? The creationist believes that he is the sane one and you are the fool. Doesn't this make the situation symmetric around the two of you? If you average your estimates together, one of you must gain, and one of you must lose, since the shifts are in opposite directions; but by Jensen's inequality it is a positive-sum game. And since, by something like a Rawlsian veil of ignorance, you don't know which of you is really the fool, you ought to take the gamble. This argues that the socially altruistic move is also always the individually rational move.
And there's also the obvious reply: "But I know perfectly well who the fool is. It's the other guy. It doesn't matter that he says the same thing - he's still the fool."
This reply sounds bald and unconvincing when you consider it abstractly. But if you actually face a creationist, then it certainly feels like the correct answer - you're right, he's wrong, and you have valid evidence to know that, even if the creationist can recite exactly the same claim in front of a TV audience.
Robin Hanson sides with symmetry - this is clearest in his paper Uncommon Priors Require Origin Disputes - and therefore endorses the Modesty Argument. (Though I haven't seen him analyze the particular case of the creationist.)
I respond: Those who dream do not know they dream; but when you wake you know you are awake. Dreaming, you may think you are awake. You may even be convinced of it. But right now, when you really are awake, there isn't any doubt in your mind - nor should there be. If you, persuaded by the clever argument, decided to start doubting right now that you're really awake, then your Bayesian score would go down and you'd become that much less accurate. If you seriously tried to make yourself doubt that you were awake - in the sense of wondering if you might be in the midst of an ordinary human REM cycle - then you would probably do so because you wished to appear to yourself as rational, or because it was how you conceived of "rationality" as a matter of moral duty. Because you wanted to act with propriety. Not because you felt genuinely curious as to whether you were awake or asleep. Not because you felt you might really and truly be asleep. But because you didn't have an answer to the clever argument, just an (ahem) incommunicable insight that you were awake.
Russell Wallace put it thusly: "That we can postulate a mind of sufficiently low (dreaming) or distorted (insane) consciousness as to genuinely not know whether it's Russell or Napoleon doesn't mean I (the entity currently thinking these thoughts) could have been Napoleon, any more than the number 3 could have been the number 7. If you doubt this, consider the extreme case: a rock doesn't know whether it's me or a rock. That doesn't mean I could have been a rock."
There are other problems I see with the Modesty Argument, pragmatic matters of human rationality - if a fallible human tries to follow the Modesty Argument in practice, does this improve or disimprove personal rationality? To me it seems that the adherents of the Modesty Argument tend to profess Modesty but not actually practice it.
For example, let's say you're a scientist with a controversial belief - like the Modesty Argument itself, which is hardly a matter of common accord - and you spend some substantial amount of time and effort trying to prove, argue, examine, and generally forward this belief. Then one day you encounter the Modesty Argument, and it occurs to you that you should adjust your belief toward the modal belief of the scientific field. But then you'd have to give up your cherished hypothesis. So you do the obvious thing - I've seen at least two people do this on two different occasions - and say: "Pursuing my personal hypothesis has a net expected utility to Science. Even if I don't really believe that my theory is correct, I can still pursue it because of the categorical imperative: Science as a whole will be better off if scientists go on pursuing their own hypotheses." And then they continue exactly as before.
I am skeptical to say the least. Integrating the Modesty Argument as new evidence ought to produce a large effect on someone's life and plans. If it's being really integrated, that is, rather than flushed down a black hole. Your personal anticipation of success, the bright emotion with which you anticipate the confirmation of your theory, should diminish by literally orders of magnitude after accepting the Modesty Argument. The reason people buy lottery tickets is that the bright anticipation of winning ten million dollars, the dancing visions of speedboats and mansions, is not sufficiently diminished - as a strength of emotion - by the probability factor, the odds of a hundred million to one. The ticket buyer may even profess that the odds are a hundred million to one, but they don't anticipate it properly - they haven't integrated the mere verbal phrase "hundred million to one" on an emotional level.
So, when a scientist integrates the Modesty Argument as new evidence, should the resulting nearly total loss of hope have no effect on real-world plans originally formed in blessed ignorance and joyous anticipation of triumph? Especially when you consider that the scientist knew about the social utility to start with, while making the original plans? I think that's around as plausible as maintaining your exact original investment profile after the expected returns on some stocks change by a factor of a hundred. What's actually happening, one naturally suspects, is that the scientist finds that the Modesty Argument has uncomfortable implications; so they reach for an excuse, and invent on-the-fly the argument from social utility as a way of exactly cancelling out the Modesty Argument and preserving all their original plans.
But of course if I say that this is an argument against the Modesty Argument, that is pure ad hominem tu quoque. If its adherents fail to use the Modesty Argument properly, that does not imply it has any less force as logic.
Rather than go into more detail on the manifold ramifications of the Modesty Argument, I'm going to close with the thought experiment that initially convinced me of the falsity of the Modesty Argument. In the beginning it seemed to me reasonable that if feelings of 99% certainty were associated with a 70% frequency of true statements, on average across the global population, then the state of 99% certainty was like a "pointer" to 70% probability. But at one point I thought: "What should an (AI) superintelligence say in the same situation? Should it treat its 99% probability estimates as 70% probability estimates because so many human beings make the same mistake?" In particular, it occurred to me that, on the day the first true superintelligence was born, it would be undeniably true that - across the whole of Earth's history - the enormously vast majority of entities who had believed themselves superintelligent would be wrong. The majority of the referents of the pointer "I am a superintelligence" would be schizophrenics who believed they were God.
A superintelligence doesn't just believe the bald statement that it is a superintelligence - it presumably possesses a very detailed, very accurate self-model of its own cognitive systems, tracks in detail its own calibration, and so on. But if you tell this to a mental patient, the mental patient can immediately respond: "Ah, but I too possess a very detailed, very accurate self-model!" The mental patient may even come to sincerely believe this, in the moment of the reply. Does that mean the superintelligence should wonder if it is a mental patient? This is the opposite extreme of Russell Wallace asking if a rock could have been you, since it doesn't know if it's you or the rock.
One obvious reply is that human beings and superintelligences occupy different classes - we do not have the same ur-priors, or we are not part of the same anthropic reference class; some sharp distinction renders it impossible to group together superintelligences and schizophrenics in probability arguments. But one would then like to know exactly what this "sharp distinction" is, and how it is justified relative to the Modesty Argument. Can an evolutionist and a creationist also occupy different reference classes? It sounds astoundingly arrogant; but when I consider the actual, pragmatic situation, it seems to me that this is genuinely the case.
Or here's a more recent example - one that inspired me to write today's blog post, in fact. It's the true story of a customer struggling through five levels of Verizon customer support, all the way up to floor manager, in an ultimately futile quest to find someone who could understand the difference between .002 dollars per kilobyte and .002 cents per kilobyte. Audio [27 minutes], Transcript. It has to be heard to be believed. Sample of conversation: "Do you recognize that there's a difference between point zero zero two dollars and point zero zero two cents?" "No."
The key phrase that caught my attention and inspired me to write today's blog post is from the floor manager: "You already talked to a few different people here, and they've all explained to you that you're being billed .002 cents, and if you take it and put it on your calculator... we take the .002 as everybody has told you that you've called in and spoken to, and as our system bills accordingly, is correct."
Should George - the customer - have started doubting his arithmetic, because five levels of Verizon customer support, some of whom cited multiple years of experience, told him he was wrong? Should he have adjusted his probability estimate in their direction? A straightforward extension of Aumann's Agreement Theorem to impossible possible worlds, that is, uncertainty about the results of computations, proves that, had all parties been genuine Bayesians with common knowledge of each other's estimates, they would have had the same estimate. Jensen's inequality proves even more straightforwardly that, if George and the five levels of tech support had averaged together their probability estimates, they would have improved their average log score. If such arguments fail in this case, why do they succeed in other cases? And if you claim the Modesty Argument carries in this case, are you really telling me that if George had wanted only to find the truth for himself, he would have been wise to adjust his estimate in Verizon's direction? I know this is an argument from personal incredulity, but I think it's a good one.
On the whole, and in practice, it seems to me like Modesty is sometimes a good idea, and sometimes not. I exercise my individual discretion and judgment to decide, even knowing that I might be biased or self-favoring in doing so, because the alternative of being Modest in every case seems to me much worse.
But the question also seems to have a definite anthropic flavor. Anthropic probabilities still confuse me; I've read arguments but I have been unable to resolve them to my own satisfaction. Therefore, I confess, I am not able to give a full account of how the Modesty Argument is resolved.
Modest, aren't I?
December 21, 2006
"I don't know."
An edited transcript of a long instant-messenger conversation that took place regarding the phrase, "I don't know", sparked by Robin Hanson's previous post, "You Are Never Entitled to Your Opinion."
[08:50] Eliezer: http://www.overcomingbias.com/2006/12/you_are_never_e.html
[09:01] X: it still seems that saying "i don't know" in some situations is better than giving your best guess
[09:01] X: especially if you are dealing with people who will take you at your word who are not rationalists
[09:02] Eliezer: in real life, you have to choose, and bet, at some betting odds
[09:02] Eliezer: i.e., people who want to say "I don't know" for cryonics still have to sign up or not sign up, and they'll probably do the latter
[09:03] Eliezer: "I don't know" is usually just a screen that people think is defensible and unarguable before they go on to do whatever they feel like, and it's usually the wrong thing because they refused to admit to themselves what their guess was, or examine their justifications, or even realize that they're guessing
[09:02] X: how many apples are in a tree outside?
[09:02] X: i've never seen it and neither have you
[09:02] Eliezer: 10 to 1000
[09:04] Eliezer: if you offer to bet me a million dollars against one dollar that the tree outside has fewer than 20 apples, when neither of us have seen it, I will take your bet
[09:04] X: is it better to say "maybe 10 to 1000" to make it clear that you are guessing when talking to people
[09:04] Eliezer: therefore I have assigned a nonzero and significant probability to apples < 20 whether I admit it or not
[09:05] Eliezer: what you *say* is another issue, especially when speaking to nonrationalists, and then it is well to bear in mind that words don't have fixed meanings; the meaning of the sounds that issue from your lips is whatever occurs in the mind of the listener. If they're going to misinterpret something then you shouldn't say it to them no matter what the words mean inside your own head
[09:06] Eliezer: often you are just screwed unless you want to go back and teach them rationality from scratch, and in a case like that, all you can do is say whatever creates the least inaccurate image
[09:06] X: 10 to 1000 is misleading when you say it to a nonrationalist?
[09:06] Eliezer: "I don't know" is a good way to duck when you say it to someone who doesn't know about probability distributions
[09:07] Eliezer: if they thought I was certain, or that my statement implied actual knowledge of the tree
[09:07] Eliezer: then the statement would mislead them
[09:07] Eliezer: and if I knew this, and did it anyway for my own purposes, it would be a lie
[09:08] Eliezer: if I just couldn't think of anything better to say, then it would be honest but not true, if you can see the distinction
[09:08] Eliezer: honest for me, but the statement that formed in their minds would still not be true
[09:09] X: most people will say to you.... but you said....10-1000 apples
[09:09] Eliezer: then you're just screwed
[09:10] Eliezer: nothing you can do will create in their minds a true understanding, not even "I don't know"
[09:10] X: why bother, why not say i don't know?
[09:10] Eliezer: honesty therefore consists of misleading them the least and telling them the most
[09:10] X: it's better than misleading them with 10-1000
[09:10] Eliezer: as for "why bother", well, if you're going to ask that question, just don't reply to their email or whatever
[09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?
[09:11] X: they may act on your guess, and then say "but you said...." and lose money or get in trouble or have less respect for you
[09:11] Eliezer: then you choose to wave them off
[09:11] Eliezer: with "I don't know"
[09:11] Eliezer: but it's for your own sake, not for their sake
[09:12] X: [09:11] Eliezer: what if you're dealing with someone who thinks my saying "I don't know" is a license for them to make up their own ideas, which will be a lot worse?
[09:12] X: here i could see why
[09:12] X: but it's difficult working with typical people in the real world
[09:13] Eliezer: the first thing to decide is, are you trying to accomplish something for yourself (like not getting in trouble) or are you trying to improve someone else's picture of reality
[09:13] Eliezer: "I don't know" is often a good way of not getting in trouble
[09:13] Eliezer: as for it being difficult to work with people in the real world, well, yeah
[09:13] X: if you say...10-1000, and you are wrong, and they are mad, then you say, i don't know, they will be even madder
[09:13] Eliezer: are you trying not to get in trouble?
[09:14] Eliezer: or are you trying to improve their picture of reality?
[09:14] Eliezer: these are two different tasks
[09:14] X: especially if they have lost money or have been proven wrong by someone else
[09:14] Eliezer: if they intersect you have to decide what your tradeoff is
[09:14] Eliezer: which is more important to you
[09:14] Eliezer: then decide whether to explain for their benefit or say "I don't know" for yours
[09:15] X: well, if it was my job, i would say i don't know rather than be wrong, because who knows what your boss will do after he loses money listening to you
[09:15] Eliezer: okay
[09:16] Eliezer: just be clear that this is not because "I don't know" is the rational judgment, but because "I don't know" is the political utterance
[09:16] X: he may take your guess, and try to turn it into an actual anwser because no one around you has a better plan
[09:17] Eliezer: you can easily see this by looking at your stated reason: you didn't talk about evidence and reality and truth, but, how you might get in trouble based on someone's reaction
[09:17] X: yes
[09:17] X: that's what you have to put up with in the real world
[09:17] Eliezer: if you're really worried about your boss's welfare then you should consider that if you say "I don't know" he must do something anyway - refusing to choose is also a choice, and refusing to act is like refusing to let time pass - and he will construct that plan based on some information, which doesn't include your information
[09:18] Eliezer: if your life isn't worth more than someone else's, neither it is worth any less, and it is often proper to let fools make their own mistakes
[09:18] Eliezer: you can only throw yourself in front of so many bullets before you run out of flesh to stop them with
[09:19] X: ?
[09:19] Eliezer: in other words, you cannot always save people from themselves
[09:23] Eliezer: but all of this is wandering away from the original point, which is true and correct, that no one is ever entitled to their own opinion
[09:26] X: what is his name?
[09:26] Eliezer: ?
[09:26] X: a man outside
[09:26] X: random guy
[09:26] Eliezer: It's probably not "Xpchtl Vaaaaaarax"
[09:26] X: probably not
[09:27] Eliezer: I suppose I could construct a second-order Markov transition diagram for the letters in names expressed in English, weighted by their frequency
[09:27] Eliezer: but that would be a lot of work
[09:28] Eliezer: so I could say "I don't know" as shorthand for the fact that, although I possess a lot of knowledge about possible and probable names, I don't know anything *more* than you do
[09:28] X: ok, so you say ruling out what you see as likely not correct is ok?
[09:28] Eliezer: what I'm saying is that I possess a large amount of knowledge about possible names
[09:28] Eliezer: all of which influences what I would bet on
[09:28] Eliezer: if I had to take a real-world action, like, guessing someone's name with a gun to my head
[09:29] Eliezer: if I had to choose it would suddenly become very relevant that I knew Michael was one of the most statistically common names, but couldn't remember for which years it was the most common, and that I knew Michael was more likely to be a male name than a female name
[09:29] Eliezer: if an alien had a gun to its head, telling it "I don't know" at this point would not be helpful
[09:29] Eliezer: because there's a whole lot I know that it doesn't
[09:30] X: ok
[09:33] X: what about a question for which you really don't have any information?
[09:33] X: like something only an alien would know
[09:34] Eliezer: if I have no evidence I use an appropriate Ignorance Prior, which distributes probability evenly across all possibilities, and assigns only a very small amount to any individual possibility because there are so many
[09:35] Eliezer: if the person I'm talking to already knows to use an ignorance prior, I say "I don't know" because we already have the same probability distribution and I have nothing to add to that
[09:35] Eliezer: the ignorance prior tells me my betting odds
[09:35] Eliezer: it governs my choices
[09:35] X: and what if you don't know how to use an ignorance prior
[09:36] X: have never heard of it etc
[09:36] Eliezer: if I'm dealing with someone who doesn't know about ignorance priors, and who is dealing with the problem by making up this huge elaborate hypothesis with lots of moving parts and many places to go wrong, then the truth is that I automatically know s/he's wrong
[09:36] Eliezer: it may not be possible to explain this to them, short of training them from scratch in rationality
[09:36] Eliezer: but it is true
[09:36] Eliezer: and if the person trusts me for a rationalist, it may be both honest and helpful to tell them, "No, that's wrong"
[09:36] X: what if that person says, "i don't know what their name is", that ok?
[09:37] Eliezer: in real life you cannot choose "I don't know", it's not an option on your list of available actions
[09:37] Eliezer: in real life it's always, "I don't know, so I'm going to say Vkktor Blackdawn because I think it sounds cool"
[09:39] Eliezer: Vkktor Blackdawn is as (im)probable as anything else, but if you start assigning more probability to it than the ignorance prior calls for - because it sounds cool, because you don't have room in your mind for more than one possibility, or because you've started to construct an elaborate mental explanation of how the alien might end up named Vkktor Blackdawn
[09:39] Eliezer: then I know better
[09:40] Eliezer: and if you trust me, I may be able to honestly and usefully tell you so
[09:40] Eliezer: rather than saying "I don't know", which is always something to say, not to think
[09:40] Eliezer: this is important if someone asks you, "At what odds would you bet that the alien is named Vkktor Blackdawn?"
[09:41] Eliezer: or if you have to do anything else, based on your guesses and the weight you assign to them
[09:41] Eliezer: which is what probability is all about
[09:41] X: and if they say "I don't know, I don't know anything about probability"?
[09:41] Eliezer: then either they trust me blindly or I can't help them
[09:41] Eliezer: that's how it goes
[09:41] Eliezer: you can't always save people from themselves
[09:42] X: trust you blindly about what you are saying or about your guess as to what the alien's name is?
[09:42] Eliezer: trust me blindly when I tell them, "Don't bet at those odds."
December 22, 2006
A Fable of Science and Politics
In the time of the Roman Empire, civic life was divided between the Blue and Green factions. The Blues and the Greens murdered each other in single combats, in ambushes, in group battles, in riots. Procopius said of the warring factions: "So there grows up in them against their fellow men a hostility which has no cause, and at no time does it cease or disappear, for it gives place neither to the ties of marriage nor of relationship nor of friendship, and the case is the same even though those who differ with respect to these colors be brothers or any other kin." Edward Gibbon wrote: "The support of a faction became necessary to every candidate for civil or ecclesiastical honors."
Who were the Blues and the Greens? They were sports fans - the partisans of the blue and green chariot-racing teams.
Imagine a future society that flees into a vast underground network of caverns and seals the entrances. We shall not specify whether they flee disease, war, or radiation; we shall suppose the first Undergrounders manage to grow food, find water, recycle air, make light, and survive, and that their descendants thrive and eventually form cities. Of the world above, there are only legends written on scraps of paper; and one of these scraps of paper describes the sky, a vast open space of air above a great unbounded floor. The sky is cerulean in color, and contains strange floating objects like enormous tufts of white cotton. But the meaning of the word "cerulean" is controversial; some say that it refers to the color known as "blue", and others that it refers to the color known as "green".
In the early days of the underground society, the Blues and Greens contested with open violence; but today, truce prevails - a peace born of a growing sense of pointlessness. Cultural mores have changed; there is a large and prosperous middle class that has grown up with effective law enforcement and become unaccustomed to violence. The schools provide some sense of historical perspective; how long the battle between Blues and Greens continued, how many died, how little changed as a result. Minds have been laid open to the strange new philosophy that people are people, whether they be Blue or Green.
The conflict has not vanished. Society is still divided along Blue and Green lines, and there is a "Blue" and a "Green" position on almost every contemporary issue of political or cultural importance. The Blues advocate taxes on individual incomes, the Greens advocate taxes on merchant sales; the Blues advocate stricter marriage laws, while the Greens wish to make it easier to obtain divorces; the Blues take their support from the heart of city areas, while the more distant farmers and watersellers tend to be Green; the Blues believe that the Earth is a huge spherical rock at the center of the universe, the Greens that it is a huge flat rock circling some other object called a Sun. Not every Blue or every Green citizen takes the "Blue" or "Green" position on every issue, but it would be rare to find a city merchant who believed the sky was blue, and yet advocated an individual tax and freer marriage laws.
The Underground is still polarized; an uneasy peace. A few folk genuinely think that Blues and Greens should be friends, and it is now common for a Green to patronize a Blue shop, or for a Blue to visit a Green tavern. Yet from a truce originally born of exhaustion, there is a quietly growing spirit of tolerance, even friendship.
One day, the Underground is shaken by a minor earthquake. A sightseeing party of six is caught in the tremblor while looking at the ruins of ancient dwellings in the upper caverns. They feel the brief movement of the rock under their feet, and one of the tourists trips and scrapes her knee. The party decides to turn back, fearing further earthquakes. On their way back, one person catches a whiff of something strange in the air, a scent coming from a long-unused passageway. Ignoring the well-meant cautions of fellow travellers, the person borrows a powered lantern and walks into the passageway. The stone corridor wends upward... and upward... and finally terminates in a hole carved out of the world, a place where all stone ends. Distance, endless distance, stretches away into forever; a gathering space to hold a thousand cities. Unimaginably far above, too bright to look at directly, a searing spark casts light over all visible space, the naked filament of some huge light bulb. In the air, hanging unsupported, are great incomprehensible tufts of white cotton. And the vast glowing ceiling above... the color... is...
Now history branches, depending on which member of the sightseeing party decided to follow the corridor to the surface.
Aditya the Blue stood under the blue forever, and slowly smiled. It was not a pleasant smile. There was hatred, and wounded pride; it recalled every argument she'd ever had with a Green, every rivalry, every contested promotion. "You were right all along," the sky whispered down at her, "and now you can prove it." For a moment Aditya stood there, absorbing the message, glorying in it, and then she turned back to the stone corridor to tell the world. As Aditya walked, she curled her hand into a clenched fist. "The truce," she said, "is over."
Barron the Green stared incomprehendingly at the chaos of colors for long seconds. Understanding, when it came, drove a pile-driver punch into the pit of his stomach. Tears started from his eyes. Barron thought of the Massacre of Cathay, where a Blue army had massacred every citizen of a Green town, including children; he thought of the ancient Blue general, Annas Rell, who had declared Greens "a pit of disease; a pestilence to be cleansed"; he thought of the glints of hatred he'd seen in Blue eyes and something inside him cracked. "How can you be on their side?" Barron screamed at the sky, and then he began to weep; because he knew, standing under the malevolent blue glare, that the universe had always been a place of evil.
Charles the Blue considered the blue ceiling, taken aback. As a professor in a mixed college, Charles had carefully emphasized that Blue and Green viewpoints were equally valid and deserving of tolerance: The sky was a metaphysical construct, and cerulean a color that could be seen in more than one way. Briefly, Charles wondered whether a Green, standing in this place, might not see a green ceiling above; or if perhaps the ceiling would be green at this time tomorrow; but he couldn't stake the continued survival of civilization on that. This was merely a natural phenomenon of some kind, having nothing to do with moral philosophy or society... but one that might be readily misinterpreted, Charles feared. Charles sighed, and turned to go back into the corridor. Tomorrow he would come back alone and block off the passageway.
Daria, once Green, tried to breathe amid the ashes of her world. I will not flinch, Daria told herself, I will not look away. She had been Green all her life, and now she must be Blue. Her friends, her family, would turn from her. Speak the truth, even if your voice trembles, her father had told her; but her father was dead now, and her mother would never understand. Daria stared down the calm blue gaze of the sky, trying to accept it, and finally her breathing quietened. I was wrong, she said to herself mournfully; it's not so complicated, after all. She would find new friends, and perhaps her family would forgive her... or, she wondered with a tinge of hope, rise to this same test, standing underneath this same sky? "The sky is blue," Daria said experimentally, and nothing dire happened to her; but she couldn't bring herself to smile. Daria the Blue exhaled sadly, and went back into the world, wondering what she would say.
Eddin, a Green, looked up at the blue sky and began to laugh cynically. The course of his world's history came clear at last; even he couldn't believe they'd been such fools. "Stupid," Eddin said, "stupid, stupid, and all the time it was right here." Hatred, murders, wars, and all along it was just a thing somewhere, that someone had written about like they'd write about any other thing. No poetry, no beauty, nothing that any sane person would ever care about, just one pointless thing that had been blown out of all proportion. Eddin leaned against the cave mouth wearily, trying to think of a way to prevent this information from blowing up the world, and wondering if they didn't all deserve it.
Ferris gasped involuntarily, frozen by sheer wonder and delight. Ferris's eyes darted hungrily about, fastening on each sight in turn before moving reluctantly to the next; the blue sky, the white clouds, the vast unknown outside, full of places and things (and people?) that no Undergrounder had ever seen. "Oh, so that's what color it is," Ferris said, and went exploring.
January 20, 2007
Some Claims Are Just Too Extraordinary
"I would sooner believe that two Yankee professors would lie, than that stones would fall from heaven."
-- Thomas Jefferson, on meteors
"How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn't. It isn't going to happen."
-- Eliezer Yudkowsky, "A Technical Explanation of Technical Explanation"
"If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.†Then when I discovered a “Flux Capacitor†type thing I would finally stop and say, “Hey, cool gadget!†Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest "Grey†(they are the tall nice ones), “where the hell did this come from?†Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor†and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations."
-- Peter K. Bertine, on the Extropian mailing list
We underestimate the power of science, and overestimate the power of personal observation. A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes. Our own eyes can deceive us. People can fool themselves, hallucinate, and even go insane. The controls on publication in major journals are more trustworthy than the very fabric of your brain. If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.
This is not what most scientists will tell you, of course; but I think it is pragmatically true. Because in real life, what happens is that your eyes have a little malfunction and decide that the sky is green, and science will tell you that the sky is blue.
A replicated scientific report is a special kind of extraordinary claim, designed by the surrounding process to be more extraordinary evidence than simple verbal claims. It is more extraordinary evidence because the surrounding process - and I would place a far greater premium on the replication than on the peer review, by the way - is constructed to deny entrance to claims that are in fact false. In this way, the replicated scientific report becomes capable of overcoming greater burdens of prior improbability.
There are some burdens of prior improbability so great that simple verbal claims cannot overcome them. I would not believe someone who claimed that their coffee was disobeying conservation of angular momentum - but I might believe the same report published in Physics Today, with at least three replications. Who would believe in quantum mechanics if a stranger walked up to us on the street and whispered it to us?
Are there some burdens of prior improbability so great that science itself cannot overcome them?
What about the claim that 2 + 2 = 5?
What about journals that claim to publish replicated reports of ESP?
Sometimes, even claims deliberately constructed to be extraordinary evidence end up just not being extraordinary enough.
January 20, 2007
Outside the Laboratory
"Outside the laboratory, scientists are no wiser than anyone else." Sometimes this proverb is spoken by scientists, humbly, sadly, to remind themselves of their own fallibility. Sometimes this proverb is said for rather less praiseworthy reasons, to devalue unwanted expert advice. Is the proverb true? Probably not in an absolute sense. It seems much too pessimistic to say that scientists are literally no wiser than average, that there is literally zero correlation.
But the proverb does appear true to some degree, and I propose that we should be very disturbed by this fact. We should not sigh, and shake our heads sadly. Rather we should sit bolt upright in alarm. Why? Well, suppose that an apprentice shepherd is laboriously trained to count sheep, as they pass in and out of a fold. Thus the shepherd knows when all the sheep have left, and when all the sheep have returned. Then you give the shepherd a few apples, and say: "How many apples?" But the shepherd stares at you blankly, because they weren't trained to count apples - just sheep. You would probably suspect that the shepherd didn't understand counting very well.
Now suppose we discover that a Ph.D. economist buys a lottery ticket every week. We have to ask ourselves: Does this person really understand expected utility, on a gut level? Or have they just been trained to perform certain algebra tricks?
One thinks of Richard Feynman's account of a failing physics education program:
"The students had memorized everything, but they didn't know what anything meant. When they heard 'light that is reflected from a medium with an index', they didn't know that it meant a material such as water. They didn't know that the 'direction of the light' is the direction in which you see something when you're looking at it, and so on. Everything was entirely memorized, yet nothing had been translated into meaningful words. So if I asked, 'What is Brewster's Angle?' I'm going into the computer with the right keywords. But if I say, 'Look at the water,' nothing happens - they don't have anything under 'Look at the water'!"
Suppose we have an apparently competent scientist, who knows how to design an experiment on N subjects; the N subjects will receive a randomized treatment; blinded judges will classify the subject outcomes; and then we'll run the results through a computer and see if the results are significant at the 0.05 confidence level. Now this is not just a ritualized tradition. This is not a point of arbitrary etiquette like using the correct fork for salad. It is a ritualized tradition for testing hypotheses experimentally. Why should you test your hypothesis experimentally? Because you know the journal will demand so before it publishes your paper? Because you were trained to do it in college? Because everyone else says in unison that it's important to do the experiment, and they'll look at you funny if you say otherwise?
No: because, in order to map a territory, you have to go out and look at the territory. It isn't possible to produce an accurate map of a city while sitting in your living room with your eyes closed, thinking pleasant thoughts about what you wish the city was like. You have to go out, walk through the city, and write lines on paper that correspond to what you see. It happens, in miniature, every time you look down at your shoes to see if your shoelaces are untied. Photons arrive from the Sun, bounce off your shoelaces, strike your retina, are transduced into neural firing frequences, and are reconstructed by your visual cortex into an activation pattern that is strongly correlated with the current shape of your shoelaces. To gain new information about the territory, you have to interact with the territory. There has to be some real, physical process whereby your brain state ends up correlated to the state of the environment. Reasoning processes aren't magic; you can give causal descriptions of how they work. Which all goes to say that, to find things out, you've got to go look.
Now what are we to think of a scientist who seems competent inside the laboratory, but who, outside the laboratory, believes in a spirit world? We ask why, and the scientist says something along the lines of: "Well, no one really knows, and I admit that I don't have any evidence - it's a religious belief, it can't be disproven one way or another by observation." I cannot but conclude that this person literally doesn't know why you have to look at things. They may have been taught a certain ritual of experimentation, but they don't understand the reason for it - that to map a territory, you have to look at it - that to gain information about the environment, you have to undergo a causal process whereby you interact with the environment and end up correlated to it. This applies just as much to a double-blind experimental design that gathers information about the efficacy of a new medical device, as it does to your eyes gathering information about your shoelaces.
Maybe our spiritual scientist says: "But it's not a matter for experiment. The spirits spoke to me in my heart." Well, if we really suppose that spirits are speaking in any fashion whatsoever, that is a causal interaction and it counts as an observation. Probability theory still applies. If you propose that some personal experience of "spirit voices" is evidence for actual spirits, you must propose that there is a favorable likelihood ratio for spirits causing "spirit voices", as compared to other explanations for "spirit voices", which is sufficient to overcome the prior improbability of a complex belief with many parts. Failing to realize that "the spirits spoke to me in my heart" is an instance of "causal interaction", is analogous to a physics student not realizing that a "medium with an index" means a material such as water.
It is easy to be fooled, perhaps, by the fact that people wearing lab coats use the phrase "causal interaction" and that people wearing gaudy jewelry use the phrase "spirits speaking". Discussants wearing different clothing, as we all know, demarcate independent spheres of existence - "separate magisteria", in Stephen J. Gould's immortal blunder of a phrase. Actually, "causal interaction" is just a fancy way of saying, "Something that makes something else happen", and probability theory doesn't care what clothes you wear.
In modern society there is a prevalent notion that spiritual matters can't be settled by logic or observation, and therefore you can have whatever religious beliefs you like. If a scientist falls for this, and decides to live their extralaboratorial life accordingly, then this, to me, says that they only understand the experimental principle as a social convention. They know when they are expected to do experiments and test the results for statistical significance. But put them in a context where it is socially conventional to make up wacky beliefs without looking, and they just as happily do that instead.
The apprentice shepherd is told that if "seven" sheep go out, and "eight" sheep go out, then "fifteen" sheep had better come back in. Why "fifteen" instead of "fourteen" or "three"? Because otherwise you'll get no dinner tonight, that's why! So that's professional training of a kind, and it works after a fashion - but if social convention is the only reason why seven sheep plus eight sheep equals fifteen sheep, then maybe seven apples plus eight apples equals three apples. Who's to say that the rules shouldn't be different for apples?
But if you know why the rules work, you can see that addition is the same for sheep and for apples. Isaac Newton is justly revered, not for his outdated theory of gravity, but for discovering that - amazingly, surprisingly - the celestial planets, in the glorious heavens, obeyed just the same rules as falling apples. In the macroscopic world - the everyday ancestral environment - different trees bear different fruits, different customs hold for different people at different times. A genuinely unified universe, with stationary universal laws, is a highly counterintuitive notion to humans! It is only scientists who really believe it, though some religions may talk a good game about the "unity of all things".
As Richard Feynman put it:
"If we look at a glass closely enough we see the entire universe. There are the things of physics: the twisting liquid which evaporates depending on the wind and weather, the reflections in the glass, and our imaginations adds the atoms. The glass is a distillation of the Earth's rocks, and in its composition we see the secret of the universe's age, and the evolution of the stars. What strange array of chemicals are there in the wine? How did they come to be? There are the ferments, the enzymes, the substrates, and the products. There in wine is found the great generalization: all life is fermentation. Nobody can discover the chemistry of wine without discovering, as did Louis Pasteur, the cause of much disease. How vivid is the claret, pressing its existence into the consciousness that watches it! If our small minds, for some convenience, divide this glass of wine, this universe, into parts — physics, biology, geology, astronomy, psychology, and so on — remember that Nature does not know it! So let us put it all back together, not forgetting ultimately what it is for. Let it give us one more final pleasure: drink it and forget it all!"
A few religions, especially the ones invented or refurbished after Isaac Newton, may profess that "everything is connected to everything else". (Since there is a trivial isomorphism between graphs and their complements, this profound wisdom conveys exactly the same useful information as a graph with no edges.) But when it comes to the actual meat of the religion, prophets and priests follow the ancient human practice of making everything up as they go along. And they make up one rule for women under twelve, another rule for men over thirteen; one rule for the Sabbath and another rule for weekdays; one rule for science and another rule for sorcery...
Reality, we have learned to our shock, is not a collection of separate magisteria, but a single unified process governed by mathematically simple low-level rules. Different buildings on a university campus do not belong to different universes, though it may sometimes seem that way. The universe is not divided into mind and matter, or life and nonlife; the atoms in our heads interact seamlessly with the atoms of the surrounding air. Nor is Bayes's Theorem different from one place to another.
If, outside of their specialist field, some particular scientist is just as susceptible as anyone else to wacky ideas, then they probably never did understand why the scientific rules work. Maybe they can parrot back a bit of Popperian falsificationism; but they don't understand on a deep level, the algebraic level of probability theory, the causal level of cognition-as-machinery. They've been trained to behave a certain way in the laboratory, but they don't like to be constrained by evidence; when they go home, they take off the lab coat and relax with some comfortable nonsense. And yes, that does make me wonder if I can trust that scientist's opinions even in their own field - especially when it comes to any controversial issue, any open question, anything that isn't already nailed down by massive evidence and social convention.
Maybe we can beat the proverb - be rational in our personal lives, not just our professional lives. We shouldn't let a mere proverb stop us: "A witty saying proves nothing," as Voltaire said. Maybe we can do better, if we study enough probability theory to know why the rules work, and enough experimental psychology to see how they apply in real-world cases - if we can learn to look at the water. An ambition like that lacks the comfortable modesty of being able to confess that, outside your specialty, you're no better than anyone else. But if our theories of rationality don't generalize to everyday life, we're doing something wrong. It's not a different universe inside and outside the laboratory.
Addendum: If you think that (a) science is purely logical and therefore opposed to emotion, or (b) that we shouldn't bother to seek truth in everyday life, see "Why Truth?" For new readers, I also recommend "Twelve Virtues of Rationality."
February 18, 2007
Politics is the Mind-Killer
People go funny in the head when talking about politics. The evolutionary reasons for this are so obvious as to be worth belaboring: In the ancestral environment, politics was a matter of life and death. And sex, and wealth, and allies, and reputation... When, today, you get into an argument about whether "we" ought to raise the minimum wage, you're executing adaptations for an ancestral environment where being on the wrong side of the argument could get you killed. Being on the right side of the argument could let you kill your hated rival!
If you want to make a point about science, or rationality, then my advice is to not choose a domain from contemporary politics if you can possibly avoid it. If your point is inherently about politics, then talk about Louis XVI during the French Revolution. Politics is an important domain to which we should individually apply our rationality - but it's a terrible domain in which to learn rationality, or discuss rationality, unless all the discussants are already rational.
Politics is an extension of war by other means. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back - providing aid and comfort to the enemy. People who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue.
In Artificial Intelligence, and particularly in the domain of nonmonotonic reasoning, there's a standard problem: "All Quakers are pacifists. All Republicans are not pacifists. Nixon is a Quaker and a Republican. Is Nixon a pacifist?"
What on Earth was the point of choosing this as an example? To rouse the political emotions of the readers and distract them from the main question? To make Republicans feel unwelcome in courses on Artificial Intelligence and discourage them from entering the field? (And no, before anyone asks, I am not a Republican. Or a Democrat.)
Why would anyone pick such a distracting example to illustrate nonmonotonic reasoning? Probably because the author just couldn't resist getting in a good, solid dig at those hated Greens. It feels so good to get in a hearty punch, y'know, it's like trying to resist a chocolate cookie.
As with chocolate cookies, not everything that feels pleasurable is good for you. And it certainly isn't good for our hapless readers who have to read through all the angry comments your blog post inspired.
I'm not saying that I think Overcoming Bias should be apolitical, or even that we should adopt Wikipedia's ideal of the Neutral Point of View. But try to resist getting in those good, solid digs if you can possibly avoid it. If your topic legitimately relates to attempts to ban evolution in school curricula, then go ahead and talk about it - but don't blame it explicitly on the whole Republican Party; some of your readers may be Republicans, and they may feel that the problem is a few rogues, not the entire party. As with Wikipedia's NPOV, it doesn't matter whether (you think) the Republican Party really is at fault. It's just better for the spiritual growth of the community to discuss the issue without invoking color politics.
(Now that I've been named as a co-moderator, I guess I'd better include a disclaimer: This article is my personal opinion, not a statement of official Overcoming Bias policy. This will always be the case unless explicitly specified otherwise.)
February 24, 2007
Just Lose Hope Already
Casey Serin, a 24-year-old web programmer with no prior experience in real estate, owes banks 2.2 million dollars after lying on mortgage applications in order to simultaneously buy 8 different houses in different states. He took cash out of the mortgage (applied for larger amounts than the price of the house) and spent the money on living expenses and real-estate seminars. He was expecting the market to go up, it seems.
That's not even the sad part. The sad part is that he still hasn't given up. Casey Serin does not accept defeat. He refuses to declare bankruptcy, or get a job; he still thinks he can make it big in real estate. He went on spending money on seminars. He tried to take out a mortgage on a 9th house. He hasn't failed, you see, he's just had a learning experience.
That's what happens when you refuse to lose hope.
While this behavior may seem to be merely stupid, it also puts me in mind of two Nobel-Prize-winning economists...
...namely Merton and Scholes of Long-Term Capital Management.
While LTCM raked in giant profits over its first three years, in 1998 the inefficiences that LTCM were exploiting had started to vanish - other people knew about the trick, so it stopped working.
LTCM refused to lose hope. Addicted to 40% annual returns, they borrowed more and more leverage to exploit tinier and tinier margins. When everything started to go wrong for LTCM, they had equity of $4.72 billion, leverage of $124.5 billion, and derivative positions of $1.25 trillion.
Every profession has a different way to be smart - different skills to learn and rules to follow. You might therefore think that the study of "rationality", as a general discipline, wouldn't have much to contribute to real-life success. And yet it seems to me that how to not be stupid has a great deal in common across professions. If you set out to teach someone how to not turn little mistakes into big mistakes, it's nearly the same art whether in hedge funds or romance, and one of the keys is this: Be ready to admit you lost.
March 02, 2007
You Are Not Hiring the Top 1%
Today's statistical fallacy (slightly redacted by editor) comes from Joel on Software:
Everyone thinks they're hiring the top 1%. Martin Fowler said, “We are still working hard to hire only the very top fraction of software developers (the target is around the top 0.5 to 1%).†I hear this from almost every software company. "We hire the top 1% or less," they all say. Could they all be hiring the top 1%? Where are all the other 99%? General Motors?
When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%? Think about what happens to the other 199 that you didn't hire. They go look for another job.
The entire world could consist of 1,000,000 programmers, of whom the worst 199 keep applying for every job and never getting them, but the best 999,801 always get jobs as soon as they apply for one. So every time a job is listed the 199 losers apply, as usual, and one guy from the pool of 999,801 applies, and he gets the job, of course, because he's the best, and now, in this contrived example, every employer thinks they're getting the top 0.5% when they're actually getting the top 99.9801%.
I'm exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don't give up and go into plumbing (although I wish they would... plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer's self-delusions about how selective they are.
This might explain some other phenomena I've heard of, such as the "slush heap" of inconceivably awful stories received in the mail by every fiction publisher that accepts unsolicited manuscripts. When a story is good enough to be published, it's accepted and removed from the system. Otherwise the hapless author sends it to another publisher!
Perhaps most novice writers aren't quite as dreadful (on average) as editors seem to believe? Editors will disproportionately encounter the work of novice writers who are not only awkward, but so incompetent as to be unaware of their own incompetence, and so proud that they can't take a hint.
PS: Two other areas where this might apply: Students who apply to your program/department for admission. And, grant proposals.
PPS: Robert Scarth comments, "This might also explain why some women often think that 'all men are bastards' - there are a few cads out there in continual circulation and with no intention to settle down, whereas men with honourable intentions have much fewer relationships, and settle down more quickly."
March 03, 2007
Policy Debates Should Not Appear One-Sided
Robin Hanson recently proposed stores where banned products could be sold. There are a number of excellent arguments for such a policy - an inherent right of individual liberty, the career incentive of bureaucrats to prohibit everything, legislators being just as biased as individuals. But even so (I replied), some poor, honest, not overwhelmingly educated mother of 5 children is going to go into these stores and buy a "Dr. Snakeoil's Sulfuric Acid Drink" for her arthritis and die, leaving her orphans to weep on national television.
I was just making a simple factual observation. Why did some people think it was an argument in favor of regulation?
On questions of simple fact (for example, whether Earthly life arose by natural selection) there's a legitimate expectation that the argument should be a one-sided battle; the facts themselves are either one way or another, and the so-called "balance of evidence" should reflect this. Indeed, under the Bayesian definition of evidence, "strong evidence" is just that sort of evidence which we only expect to find on one side of an argument.
But there is no reason for complex actions with many consequences to exhibit this onesidedness property. Why do people seem to want their policy debates to be one-sided?
Politics is the mind-killer. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back. If you abide within that pattern, policy debates will also appear one-sided to you - the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary.
One should also be aware of a related failure pattern, thinking that the course of Deep Wisdom is to compromise with perfect evenness between whichever two policy positions receive the most airtime. A policy may legitimately have lopsided costs or benefits. If policy questions were not tilted one way or the other, we would be unable to make decisions about them. But there is also a human tendency to deny all costs of a favored policy, or deny all benefits of a disfavored policy; and people will therefore tend to think policy tradeoffs are tilted much further than they actually are.
If you allow shops that sell otherwise banned products, some poor, honest, poorly educated mother of 5 kids is going to buy something that kills her. This is a prediction about a factual consequence, and as a factual question it appears rather straightforward - a sane person should readily confess this to be true regardless of which stance they take on the policy issue. You may also think that making things illegal just makes them more expensive, that regulators will abuse their power, or that her individual freedom trumps your desire to meddle with her life. But, as a matter of simple fact, she's still going to die.
We live in an unfair universe. Like all primates, humans have strong negative reactions to perceived unfairness; thus we find this fact stressful. There are two popular methods of dealing with the resulting cognitive dissonance. First, one may change one's view of the facts - deny that the unfair events took place, or edit the history to make it appear fair. Second, one may change one's morality - deny that the events are unfair.
Some libertarians might say that if you go into a "banned products shop", passing clear warning labels that say "THINGS IN THIS STORE MAY KILL YOU", and buy something that kills you, then it's your own fault and you deserve it. If that were a moral truth, there would be no downside to having shops that sell banned products. It wouldn't just be a net benefit, it would be a one-sided tradeoff with no drawbacks.
Others argue that regulators can be trained to choose rationally and in harmony with consumer interests; if those were the facts of the matter then (in their moral view) there would be no downside to regulation.
Like it or not, there's a birth lottery for intelligence - though this is one of the cases where the universe's unfairness is so extreme that many people choose to deny the facts. The experimental evidence for a purely genetic component of 0.6-0.8 is overwhelming, but even if this were to be denied, you don't choose your parental upbringing or your early schools either.
I was raised to believe that denying reality is a moral wrong. If I were to engage in wishful optimism about how Sulfuric Acid Drink was likely to benefit me, I would be doing something that I was warned against and raised to regard as unacceptable. Some people are born into environments - we won't discuss their genes, because that part is too unfair - where the local witch doctor tells them that it is right to have faith and wrong to be skeptical. In all goodwill, they follow this advice and die. Unlike you, they weren't raised to believe that people are responsible for their individual choices to follow society's lead. Do you really think you're so smart that you would have been a proper scientific skeptic even if you'd been born in 500 C.E.? Yes, there is a birth lottery, no matter what you believe about genes.
Saying "People who buy dangerous products deserve to get hurt!" is not tough-minded. It is a way of refusing to live in an unfair universe. Real tough-mindedness is saying, "Yes, sulfuric acid is a horrible painful death, and no, that mother of 5 children didn't deserve it, but we're going to keep the shops open anyway because we did this cost-benefit calculation." Can you imagine a politician saying that? Neither can I. But insofar as economists have the power to influence policy, it might help if they could think it privately - maybe even say it in journal articles, suitably dressed up in polysyllabismic obfuscationalization so the media can't quote it.
I don't think that when someone makes a stupid choice and dies, this is a cause for celebration. I count it as a tragedy. It is not always helping people, to save them from the consequences of their own actions; but I draw a moral line at capital punishment. If you're dead, you can't learn from your mistakes.
Unfortunately the universe doesn't agree with me. We'll see which one of us is still standing when this is over.
March 07, 2007
Burch's Law
Greg Burch said:
"I think people should have a right to be stupid and, if they have that right, the market's going to respond by supplying as much stupidity as can be sold."
Greg Burch was speaking about sport-utility vehicles, which he feels are very poorly designed. Note that Burch was not advocating banning SUVs. Burch did not even advocate regulating SUVs. Burch thinks people should have a right to be stupid. But Burch also openly acknowledges the real-world consequence of that right, which is that the market will respond by supplying as much stupidity as can be sold. Perhaps Burch is strongly libertarian, and sees the case against regulation as a slam-dunk regardless of the consequences, and therefore has an easier time acknowledging the downside of his policy. Or perhaps Burch is just a skillful rationalist. Either way, I hereby canonize his observation as Burch's Law.
Burch's Law is a special case of a more general rule: Just because your ethics require an action doesn't mean the universe will exempt you from the consequences. If the universe were fair, like a sympathetic human, the universe would understand that you had overriding ethical reasons for your action, and would exempt you from the usual penalties. The judge would rule "justifiable homicide" instead of "murder" and exempt you from the usual prison term. Well, the universe isn't fair and it won't exempt you from the consequences. We know the equations of physics in enough detail to know that the equations don't contain any quantities reflective of ethical considerations.
We don't send automobile manufacturers to jail, even though manufactured cars kill an estimated 1.2 million people per year worldwide. (Roughly 2% of the annual planetary death rate.) Not everyone who dies in an automobile accident is someone who decided to drive a car. The tally of casualties includes pedestrians. It includes minor children who had to be pushed screaming into the car on the way to school. And yet we still manufacture automobiles, because, well, we're in a hurry. I don't even disagree with this decision. I drive a car myself. The point is that the consequences don't change no matter how good the ethical justification sounds. The people who die in automobile accidents are still dead. We can suspend the jail penalty, but we can't suspend the laws of physics.
Humanity hasn't had much luck suspending the laws of economics, either. If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold.
March 13, 2007
The Scales of Justice, the Notebook of Rationality
Lady Justice is widely depicted as carrying a scales. A scales has the property that whatever pulls one side down, pushes the other side up. This makes things very convenient and easy to track. It's also usually a gross distortion.
In human discourse there is a natural tendency to treat discussion as a form of combat, an extension of war, a sport; and in sports you only need to keep track of how many points have been scored by each team. There are only two sides, and every point scored against one side, is a point in favor of the other. Everyone in the audience keeps a mental running count of how many points each speaker scores against the other. At the end of the debate, the speaker who has scored more points is, obviously, the winner; so everything he says must be true, and everything the loser says must be wrong.
"The Affect Heuristic in Judgments of Risks and Benefits" studied whether subjects mixed up their judgments of the possible benefits of a technology (e.g. nuclear power), and the possible risks of that technology, into a single overall good or bad feeling about the technology. Suppose that I first tell you that a particular kind of nuclear reactor generates less nuclear waste than competing reactor designs. But then I tell you that the reactor is more unstable than competing designs, with a greater danger of undergoing meltdown if a sufficiently large number of things go wrong simultaneously.
If the reactor is more likely to melt down, this seems like a 'point against' the reactor, or a 'point against' someone who argues for building the reactor. And if the reactor produces less waste, this is a 'point for' the reactor, or a 'point for' building it. So are these two facts opposed to each other? No. In the real world, no. These two facts may be cited by different sides of the same debate, but they are logically distinct; the facts don't know whose side they're on. The amount of waste produced by the reactor arises from physical properties of that reactor design. Other physical properties of the reactor make the nuclear reaction more unstable. Even if some of the same design properties are involved, you have to separately consider the probability of meltdown, and the expected annual waste generated. These are two different physical questions with two different factual answers.
But studies such as the above show that people tend to judge technologies - and many other problems - by an overall good or bad feeling. If you tell people a reactor design produces less waste, they rate its probability of meltdown as lower. This means getting the wrong answer to physical questions with definite factual answers, because you have mixed up logically distinct questions - treated facts like human soldiers on different sides of a war, thinking that any soldier on one side can be used to fight any soldier on the other side.
A scales is not wholly inappropriate for Lady Justice if she is investigating a strictly factual question of guilt or innocence. Either John Smith killed John Doe, or not. We are taught (by E. T. Jaynes) that all Bayesian evidence consists of probability flows between hypotheses; there is no such thing as evidence that "supports" or "contradicts" a single hypothesis, except insofar as other hypotheses do worse or better. So long as Lady Justice is investigating a single, strictly factual question with a binary answer space, a scales would be an appropriate tool. If Justitia must consider any more complex issue, she should relinquish her scales or relinquish her sword.
Not all arguments reduce to mere up or down. Lady Rationality carries a notebook, wherein she writes down all the facts that aren't on anyone's side.
March 15, 2007
Blue or Green on Regulation?
In recent posts, I have predicted that, if not otherwise prevented from doing so, some people will behave stupidly and suffer the consequences: "If people have a right to be stupid, the market will respond by supplying all the stupidity that can be sold." People misinterpret this as indicating that I take a policy stance in favor of regulation. It indicates no such thing. It is meant purely as guess about empirical consequences - a testable prediction on a question of simple fact.
Perhaps I would be less misinterpreted if I also told "the other side of the story" - inveighed at length about the reasons why bureaucrats are not perfect rationalists guarding our net best interests. But ideally, I shouldn't have to go to such lengths. Ideally, I could make a prediction about a strictly factual question without this being interpreted as a policy stance, or as a stance on logically distinct factual questions.
Yet it would appear that there are two and only two sides to the issue - pro-regulation and anti-regulation. All arguments are either allied soldiers or enemy soldiers; they fight on one side or the other. Any allied soldier can be deployed to fight any enemy soldier and vice versa. Whatever argument pushes one side up, pushes the other side down.
I understand that there are continuing fights about regulation, that this battle is viewed as important, and that people caught up in such battle may not want to let a pro-Green point go past without parrying with a Blue counterpoint. But these battle reflexes have developed too far. If I remark that victims of car accidents include minor children who had to be pushed screaming into the car on the way to school, anyone who is anti-regulation instantly suspects me of trying to pull out an emotional trump card. But I was not trying to get cars banned. I was trying to make a point about how emotional trump cards fail to trump the universe.
I have previously predicted on the strictly factual matter of whether, in the absence of regulation, people will get hurt. (Yes.) I have also indicated as a matter of moral judgment that I do not think they deserve to get hurt, because being stupid is not the same as being malicious. Furthermore there are such things as minor children and pedestrians.
I shouldn't have to say this, but apparently I do, so, for the record, here is "the other side of the story":
The FDA prevents 5,000 casualties per year but causes at least 20,000-120,000 casualties by delaying approval of beneficial medications. The second number is calculated only by looking at delays in the introduction of medications eventually approved - not medications never approved, or medications for which approval was never sought. FDA fatalities are comparable to the annual number of fatal car accidents, but the noneffects of medications not approved don't make the evening news. A bureaucrat's chief incentive is not to approve anything that will ever harm anyone in a way that makes it into the newspaper; no other cost-benefit calculus is involved as an actual career incentive. The bureaucracy as a whole may have an incentive to approve at least some new products - if the FDA never approved a new medication, Congress would become suspicious - but any individual bureaucrat has an unlimited incentive to say no. Regulators have no career motive to do any sort of cost-benefit calculation - except of course for the easy career-benefit calculation. A product with a failure mode spectacular enough to make the newspapers will be banned regardless of what other good it might do; one-reason decisionmaking. As with the FAA banning toenail clippers on planes, "safety precautions" are primarily an ostentatious display of costly efforts so that, when a catastrophe does occur, the agency will be seen to have tried its hardest.
Government = ordinary human fallibility + poor incentives + organizational overhead + guns.
But this does not change the consequences of nonregulation. Children will still die horrible deaths in car accidents and they still will not deserve it.
I understand that debates are not conducted in front of perfectly rational audiences. We all know what happens when you try to trade off a sacred value against a nonsacred value. It's why, when someone says, "But if you don't ban cars, people will die in car crashes!" you don't say "Yes, people will die horrible flaming deaths and they don't deserve it. But it's worth it so I don't have to walk to work in the morning." Instead you say, "How dare you take away our freedom to drive? We'll decide for ourselves; we're just as good at making decisions as you are." So go ahead and say that, then. But think to yourself, in the silent privacy of your thoughts if you must: And yet they will still die, and they will not deserve it.
That way, when Sebastian Thrun comes up with a scheme to automate the highways, and claims it will eliminate nearly all traffic accidents, you can pay appropriate attention.
So too with those other horrible consequences of stupidity that I may dwell upon in later posts. Just because (you believe) regulation may not be able to solve these problems, doesn't mean we wouldn't be very interested in a proposal to solve them by other means.
People are hurt by free markets, just as they're hurt by automobiles - torn up by huge powerful mindless machines with imperfect human operators. It may not be the course of wisdom to fix these problems by resorting to the blunt sledgehammer of ban-the-bad-thing, by wishing to the fairy godmother of government and her magic wand of law. But then people will still get hurt. They will lose their jobs, lose their pensions, lose their health insurance, be ground down to bloody stumps by poverty, perhaps die, and they won't deserve it either.
So am I Blue or Green on regulation, then? I consider myself neither. Imagine, for a moment, that much of what the Greens said about the downside of the Blue policy was true - that, left to the mercy of the free market, many people would be crushed by powers far beyond their understanding, nor would they deserve it. And imagine that most of what the Blues said about the downside of the Green policy was also true - that regulators were fallible humans with poor incentives, whacking on delicately balanced forces with a sledgehammer.
Close your eyes and imagine it. Extrapolate the result. If that were true, then... then you'd have a big problem and no easy way to fix it, that's what you'd have. Does this universe look familiar?
March 16, 2007
Superstimuli and the Collapse of Western Civilization
At least three people have died playing online games for days without rest. People have lost their spouses, jobs, and children to World of Warcraft. If people have the right to play video games - and it's hard to imagine a more fundamental right - then the market is going to respond by supplying the most engaging video games that can be sold, to the point that exceptionally engaged consumers are removed from the gene pool.
How does a consumer product become so involving that, after 57 hours of using the product, the consumer would rather use the product for one more hour than eat or sleep? (I suppose one could argue that the consumer makes a rational decision that they'd rather play Starcraft for the next hour than live out the rest of their lives, but let's just not go there. Please.)
A candy bar is a superstimulus: it contains more concentrated sugar, salt, and fat than anything that exists in the ancestral environment. A candy bar matches taste buds that evolved in a hunter-gatherer environment, but it matches those taste buds much more strongly than anything that actually existed in the hunter-gatherer environment. The signal that once reliably correlated to healthy food has been hijacked, blotted out with a point in tastespace that wasn't in the training dataset - an impossibly distant outlier on the old ancestral graphs. Tastiness, formerly representing the evolutionarily identified correlates of healthiness, has been reverse-engineered and perfectly matched with an artificial substance. Unfortunately there's no equally powerful market incentive to make the resulting food item as healthy as it is tasty. We can't taste healthfulness, after all.
The now-famous Dove Evolution video shows the painstaking construction of another superstimulus: an ordinary woman transformed by makeup, careful photography, and finally extensive Photoshopping, into a billboard model - a beauty impossible, unmatchable by human women in the unretouched real world. Actual women are killing themselves (e.g. supermodels using cocaine to keep their weight down) to keep up with competitors that literally don't exist.
And likewise, a video game can be so much more engaging than mere reality, even through a simple computer monitor, that someone will play it without food or sleep until they literally die. I don't know all the tricks used in video games, but I can guess some of them - challenges poised at the critical point between ease and impossibility, intermittent reinforcement, feedback showing an ever-increasing score, social involvement in massively multiplayer games.
Is there a limit to the market incentive to make video games more engaging? You might hope there'd be no incentive past the point where the players lose their jobs; after all, they must be able to pay their subscription fee. This would imply a "sweet spot" for the addictiveness of games, where the mode of the bell curve is having fun, and only a few unfortunate souls on the tail become addicted to the point of losing their jobs. As of 2007, playing World of Warcraft for 58 hours straight until you literally die is still the exception rather than the rule. But video game manufacturers compete against each other, and if you can make your game 5% more addictive, you may be able to steal 50% of your competitor's customers. You can see how this problem could get a lot worse.
If people have the right to be tempted - and that's what free will is all about - the market is going to respond by supplying as much temptation as can be sold. The incentive is to make your stimuli 5% more tempting than those of your current leading competitors. This continues well beyond the point where the stimuli become ancestrally anomalous superstimuli. Consider how our standards of product-selling feminine beauty have changed since the advertisements of the 1950s. And as candy bars demonstrate, the market incentive also continues well beyond the point where the superstimulus begins wreaking collateral damage on the consumer.
So why don't we just say no? A key assumption of free-market economics is that, in the absence of force and fraud, people can always refuse to engage in a harmful transaction. (To the extent this is true, a free market would be, not merely the best policy on the whole, but a policy with few or no downsides.)
An organism that regularly passes up food will die, as some video game players found out the hard way. But, on some occasions in the ancestral environment, a typically beneficial (and therefore tempting) act may in fact be harmful. Humans, as organisms, have an unusually strong ability to perceive these special cases using abstract thought. On the other hand we also tend to imagine lots of special-case consequences that don't exist, like ancestor spirits commanding us not to eat perfectly good rabbits.
Evolution seems to have struck a compromise, or perhaps just aggregated new systems on top of old. Homo sapiens are still tempted by food, but our oversized prefrontal cortices give us a limited ability to resist temptation. Not unlimited ability - our ancestors with too much willpower probably starved themselves to sacrifice to the gods, or failed to commit adultery one too many times. The video game players who died must have exercised willpower (in some sense) to keep playing for so long without food or sleep; the evolutionary hazard of self-control.
Resisting any temptation takes conscious expenditure of an exhaustible supply of mental energy. It is not in fact true that we can "just say no" - not just say no, without cost to ourselves. Even humans who won the birth lottery for willpower or foresightfulness still pay a price to resist temptation. The price is just more easily paid.
Our limited willpower evolved to deal with ancestral temptations; it may not operate well against enticements beyond anything known to hunter-gatherers. Even where we successfully resist a superstimulus, it seems plausible that the effort required would deplete willpower much faster than resisting ancestral temptations.
Is public display of superstimuli a negative externality, even to the people who say no? Should we ban chocolate cookie ads, or storefronts that openly say "Ice Cream"?
Just because a problem exists doesn't show (without further justification and a substantial burden of proof) that the government can fix it. The regulator's career incentive does not focus on products that combine low-grade consumer harm with addictive superstimuli; it focuses on products with failure modes spectacular enough to get into the newspaper. Conversely, just because the government may not be able to fix something, doesn't mean it isn't going wrong.
I leave you with a final argument from fictional evidence: Simon Funk's online novel After Life depicts (among other plot points) the planned extermination of biological Homo sapiens - not by marching robot armies, but by artificial children that are much cuter and sweeter and more fun to raise than real children. Perhaps the demographic collapse of advanced societies happens because the market supplies ever-more-tempting alternatives to having children, while the attractiveness of changing diapers remains constant over time. Where are the advertising billboards that say "BREED"? Who will pay professional image consultants to make arguing with sullen teenagers seem more alluring than a vacation in Tahiti?
"In the end," Simon Funk wrote, "the human species was simply marketed out of existence."
March 19, 2007
Useless Medical Disclaimers
I recently underwent a minor bit of toe surgery and had to sign a scary-looking disclaimer form in which I acknowledged that there was a risk of infection, repeat surgery, chronic pain, amputation, spontaneous combustion, meteor strikes, and a plague of locusts o'er the land.
It was the most pointless damned form I've ever seen in a doctor's office. What are the statistical incidences of any of these risks? Should I be more or less worried about dying in a car crash on the way home? Taken literally, that kind of "information" is absolutely useless for making decisions. You can't translate something into an expected utility, even a qualitative and approximate one, if it doesn't come with a probability attached.
Taken literally, saying that there is a "possibility" of infection tells me nothing. The probability could be 1/1,000,000,000,000 and it would still be technically correct to describe the outcome as "possible". I'm not the litigious type, but I seriously wonder if it would be possible to sue based on the theory that "possibilities" with no probabilities attached to them are not useful information and therefore should not constitute a "disclaimer" under the law.
Staring at this pointless list of disasters, I also wondered why the form contained no useful information.
The thought that occurred to me was that, innumeracy being so widespread, no one would dare put numbers on that sheet of paper. If "amputation" is listed as a consequence with a probability of 0.0001%, patients will run screaming out of the office, crying, "Not my toe! I don't want to lose my toe!" No amount of patient explanation will suffice to convince them that they ought to diminish the emotional force of their fear by a factor of one million. Each extra zero after the decimal point would only be one more symbol for their eyes to glaze over; it would not diminish the emotional force of the anticipation by an additional factor of ten.
And so I don't get any useful statistical information! Hmph.
Clearly, innumeracy produces negative externalities and it ought to be regulated. In particular, we should impose a tax on people who can't properly diminish the emotional impact of their anticipations by tiny probability factors.
Two classic objections to regulation are that (a) it infringes on personal freedom and (b) the individual always knows more about their own situation than the regulator. However, my proposed policy addresses both of these issues: rather than administering a math test, we can ask each individual whether or not they're innumerate. If they do declare themselves to be innumerate, they can decide for themselves the amount of the tax to pay.
What do you think? Would this tax give people an incentive to become less innumerate, as standard economics would predict?
March 23, 2007
Archimedes's Chronophone
Think of how many generations of humanity would have benefited if certain ideas had been invented sooner, rather than later - if the Greeks had invented science - if the Romans had possessed printing presses - if Western civilization had turned against slavery in the thirteenth century.
Archimedes of Syracuse was the greatest mathematician and engineer of the ancient world. Imagine that Archimedes invented a temporal telephone ("chronophone" for short) which lets him talk to you, here in the 21st century. You can make suggestions! For purposes of the thought experiment, ignore the morality of altering history - just assume that it is proper to optimize post-Archimedean history as though it were simply the ordinary future. If so, it would seem that you are in a position to accomplish a great deal of good.
Unfortunately, Archimedes's chronophone comes with certain restrictions upon its use: It cannot transmit information that is, in a certain sense, "too anachronistic".
You cannot suggest, for example, that women should have the vote. Maybe you could persuade Archimedes of Syracuse of the issue, and maybe not; but it is a moot point, the chronophone will not transmit the advice. Or rather, it will transmit the advice, but it will come out as: "Install a tyrant of great personal virtue, such as Hiero II, under whose rule Syracuse experienced fifty years of peace and prosperity." That's how the chronophone avoids transmitting overly anachronistic information - it transmits cognitive strategies rather than words. If you follow the policy of "Check my brain's memory to see what my contemporary culture recommends as a wise form of political organization", what comes out of the chronophone is the result of Archimedes following the same policy of looking up in his brain what his era lauds as a wise form of political organization.
You might think the next step would be to prepare a careful series of Plato-style philosophical arguments, starting from known territory, and intended to convince an impartial audience, with which to persuade Archimedes that all sentient beings should be equal before the law. Unfortunately, if you try this, what comes out on Archimedes's end is a careful series of Plato-style philosophical analogies which argue that wealthy male landowners should have special privileges. You followed the policy of "Come up with a line of philosophical argument intended to persuade a neutral observer to my own era's point of view on political privilege," so what comes out of the chronophone is what Archimedes would think up if he followed the same cognitive strategy.
In Archimedes's time, slavery was thought right and proper; in our time, it is held an abomination. If, today, you need to argue that slavery is bad, you can invent all sorts of moral arguments which lead to that conclusion - all sorts of justifications leap readily to mind. If you could talk to Archimedes of Syracuse directly, you might even be able to persuade him to your viewpoint (or not). But the really odd thing is that, at some point in time, someone must have turned against slavery - gone from pro-slavery to anti-slavery - even though they didn't start out wanting to persuade themselves against slavery. By the time someone gets to the point of wanting to construct persuasive anti-slavery arguments, they must have already turned against slavery. If you know your desired moral destination, you are already there. Thus, that particular cognitive strategy - searching for ways to persuade people against slavery - can't explain how we got here from there, how Western culture went from pro-slavery to anti-slavery.
The chronophone, to prevent paradox, will not transmit arguments that you constructed already knowing the desired destination. And because this is a law of physics governing time travel, the chronophone cannot be fooled. No matter how cleverly you construct your neutral-sounding philosophical argument, the chronophone "knows" you started with the desired conclusion already in mind.
The same dilemma applies to scientific issues. if you say "The Earth circles the Sun" it comes out of the chronophone as "The Sun circles the Earth". It doesn't matter that our civilization is right and their civilization is wrong - the chronophone takes no notice of facts, only beliefs and cognitive strategies. You tried to transmit your own belief about heavenly mechanics, so it comes out as Archimedes's belief about heavenly mechanics.
Obviously, what you need to transmit is the scientific method - that's how our own civilization went from geocentrism to heliocentrism without having the destination already in mind. Unfortunately, you also can't say to Archimedes, "Use mathematical laws instead of heroic mythology to explain empirical phenomena." It will come out as "If anyone should throw back his head and learn something by staring at the varied patterns on a ceiling, apparently you would think that he was contemplating with his reason, when he was only staring with his eyes... I cannot but believe that no study makes the soul look on high except that which is concerned with real being and the unseen." (Plato, The Republic, Book VII.) That is Archimedes's culture's stance on epistemology, just as science is your own culture's stance.
Can you suggest that Archimedes pay attention to facts, and authorities, and think about which one should ought to take precedence - by way of leading him down a garden path to the scientific method? But humanity did not invent the scientific method by setting out to invent the scientific method - by looking for a garden path that would lead to the scientific method. If you know your desired destination, you are already there. And no matter how you try to prevent your garden path from looking like a garden path, the laws of time travel know the difference.
So what can you say into the chronophone?
Suppose that, at some point in your life, you've genuinely thought that the scientific method might not be correct - that our culture's preferred method of factual investigation might be flawed. Then, perhaps, you could talk into the chronophone about how you've doubted that the scientific method as commonly practiced is correct, and it would come out of the chronophone as doubts about whether deference to authority is correct. After all, something like that must be how humanity got to science from nonscience - individuals who genuinely questioned whether their own culture's preferred method of epistemological investigation was correct.
If you try to follow this strategy, your own doubts had better be genuine. Otherwise what will come out of the chronophone is a line of Socratic questioning that argues for deference to authority. If your doubts are genuine, surface doubts will come out as surface doubts, deep doubts as deep doubts. The chronophone always knows how much you really doubted, and how much you merely tried to convince yourself you doubted so that you could say it into the chronophone. Such is the unavoidable physics of time travel.
Now... what advice do you give to Archimedes, and how do you say it into the chronophone?
Addendum: A basic principle of the chronophone is that to get nonobvious output, you need nonobvious input. If you say something that is considered obvious in your home culture, it comes out of the chronophone as something that is considered obvious in Archimedes's culture.
March 24, 2007
Chronophone Motivations
Followup to: Archimedes's Chronophone.
Suppose you could send messages back in time to Archimedes of Syracuse, using a chronophone which - to avoid transmitting anachronistic information - transmits the results of executing cognitive strategies, rather than words. If you say "Women should have the vote", it comes out as "Install a tyrant of great personal virtue", because you repeated what your culture considers a wise form of political arrangement, and what comes out of the chronophone is the result of executing the same cognitive policy in Archimedes's era.
The chronophone won't transmit arguments you rationalize using your home culture's foreknowledge of the desired conclusion - it will substitute the result of executing that cognitive policy using Archimedes's culture's belief as the intended conclusion. A basic principle of the chronophone is that if you say something considered obvious in your home culture, it comes out as something considered obvious in Archimedes's culture.
The challenge was to say something useful under this restriction. This challenge is supposed to be difficult. It's really hard to get somewhere when you don't already know your destination. If there were some simple cognitive policy you could follow to spark moral and technological revolutions, without your home culture having advance knowledge of the destination, you could execute that cognitive policy today - which is what the whole parable is about!
A surprising number of respondents seemed to completely miss the point of the chronophone, just thinking up things they would like to say directly to Archimedes. The classic question of "If you went back in time, how would you start up an industrial civilization?" has been done many times in science fiction (Lord Kalvan of Otherwhen, The Cross-Time Engineer). There are thousands of things we'd like to say to the Past. The difficult part of the question is: How do you get it to come out of the chronophone?
Ger suggested teaching Archimedes decimal notation. Well, if you speak decimal notation - our home culture's standard representation of numbers - into the chronophone, then the chronophone outputs the standard representation of numbers used in Syracuse. To get a culturally nonobvious output, you need a culturally nonobvious input. Place notation is revolutionary because it makes it easier for ordinary people, not just trained accountants, to manipulate large numbers. Maybe an equivalent new idea in our own era would be Python, which makes it easier for novices to program computers - or a mathematician trying to standardize on category theory instead of set theory as a foundation for mathematics. Coming up with that chronophone input suggests that maybe we should pay more attention, in this era, to Python or category theory! A new representation that makes math easier can add up to a lot of benefit over time.
Hertzlinger remarked: "Some of Archimedes's most potentially-important research involved things he regarded as trivial toys. So if we advise him to get interested in Rubik's cube..." Of course you cannot directly describe a Rubik's Cube into the chronophone. So I asked what corresponding input Hertzlinger would say into the chronophone - has Hertzlinger followed the cognitive policy of playing with toy ideas? Maybe if this would have been such a good policy for Archimedes to follow, we should follow it ourselves.
Robin Hanson proposed an (admittedly clever) meta-trick for fine-tuning the chronophone's output. If that worked, Robin wanted to suggest trying to make useful devices that make money, and creating a tradition of this activity. I asked Robin if he'd ever tried to make such useful devices himself - if this is so important to human progress, why isn't Robin doing it? Perhaps Robin could reply that we've already gotten a huge amount of progress out of inventing gadgets, so now this no longer offers the greatest marginal returns. But that, in turn, points up one of the essential difficulties of the challenge. In this era it is culturally obvious - a non-surprising idea - that money-making new technologies benefit humanity. What could you say into the chronophone that would correspond to the nonobviousness of that idea in Archimedes's era? I don't know if it's important enough to qualify, but, for example, Robin's thoughts about prediction markets are not considered obvious in modern culture. That makes them a better bet for chronophone input than if Robin were to describe his efforts to invent a fancy new gadget. Everyone's doing that these days; it would probably come out of the chronophone as a suggestion to become a great warrior.
Richard Hamming used to ask his fellow researchers two questions: "What are the most important problems of your field?" and "Why aren't you working on them?"
What kind of ideas have provided the greatest benefit to humanity? Why aren't you thinking them?
Most of what we desperately want to say to Archimedes is not obvious relative to Archimedes's culture. This strongly suggests that the most important things the Future would want to say to us are, amazingly enough, not things that everyone already knows. If you want to really benefit humanity, you've got to do some original thinking - come up with the sort of nonobvious idea that you would speak into a chronophone. And you have to do some hard thinking about areas of application, directions of effort. You can't just run off in the direction of what your contemporary culture has instilled as the reflex answer to the question "How can I benefit humanity?" In those orchards the low-hanging fruit is gone.
The point of the chronophone dilemma is to make us think about what kind of cognitive policies are good to follow when you don't know your destination in advance. If you can just tell Archimedes to build a capitalist society because your culture already knows this is a good idea, it defeats the purpose of the dilemma. The chronophone transmits cognitive policies, not sentences. What sort of thinking are we doing now that is analogous to the kind of thinking we wish Archimedes had done then?
March 26, 2007
Self-deception: Hypocrisy or Akrasia?
What are we to think when someone says with their lips that they desire truth, but by their other cognitive deeds choose comfortable illusions over reality (or comfortable cynicism over reality)?
Robin Hanson has labeled such individuals hypocrites. In the traditional sense of the term, a hypocrite is a moral liar: someone who says a morality which they do not, themselves, believe. On the other hand, we don't always live up to the goals we set for ourselves. If I really believe that I ought to exercise at least 3 times per week, but I don't always do so, am I properly termed a "hypocrite"? The term akrasia, meaning "weakness of will" or "failure of self-control", seems more appropriate. Even if I tell all my friends that they ought to exercise 3 times per week, that doesn't necessarily make me a hypocrite. It's good advice. (Now, if I claimed to always exercise 3 times per week, knowing that this claim was false, that would be dishonest.)
Accusations of hypocrisy garner a lot more attention than accusations of akrasia - because hypocrisy is a deliberate transgression. It is tempting to say "hypocrisy" when you really mean "akrasia", because you'll get more attention, but that can cause damage to innocent bystanders. In akrasia, your transgression is your failure of will - it's fine that you advocate going to the gym more often, you just need to live up to the principle yourself. In hypocrisy, the transgression is claiming to care: you have no right to publicly advocate the moral principle, because (the accuser says) you don't believe in it yourself.
Will Wilkinson asked Hanson: "Would it be a kind of victory if people who now say that they care about truth, but who really don't, started admitting that they really don't?"
But much more importantly: who says that people who claim to care about truth, and then deceive themselves, "really don't care" about the truth? Why not say that they really care about the truth (as is right and proper), but they aren't living up to their own morals?
It may be standard practice in economics to deduce "preferences" from actions rather than declarations, but that's because you're trying to predict, in a scientific sense, what the subject will do next - trying to build good economic models. Moral philosophy is a different bag o' worms. At the very least, it is a controversial step in moral reasoning to decide that people's emotional impulses and subconscious pressures, rather than their declarative moral reasoning processes and the words that issue from their lips, constitute their "real selves". We should then call akrasia, not weakness of will, but strength of will.
To put the dilemma more sharply: The one comes before you and pleads, "I know that I have many times been guilty of self-deception. I have bought lottery tickets, I have overestimated my driving skills, I have planned optimistically, I have refused to confront contradictory evidence. I am weak. And yet I desire to do better. Will you help me?"
So that is words issuing from the lips, which say one thing. And it may be that the one has committed other deeds which say something else. Who is the real person? Does that question have an answer, or only a definition?
I do not frame an answer. It is only needful for me to know that something has asked for my help. There is something here that can ally to me, in our quest for truth - whether or not you call it the "real self". Whether or not, for that matter, you call me my "real self". If the word "I", when I use it, does not refer to the cognitive pattern that authors these words on your computer screen, what does it refer to? And if the words that issue from some other's lips should declare me to be a ghost, then I will seek out my fellow truthseeking ghosts, and have company in my phantom quest.
March 27, 2007
Tsuyoku Naritai! (I Want To Become Stronger)
In Orthodox Judaism there is a saying: "The previous generation is to the next one as angels are to men; the next generation is to the previous one as donkeys are to men." This follows from the Orthodox Jewish belief that all Judaic law was given to Moses by God at Mount Sinai. After all, it's not as if you could do an experiment to gain new halachic knowledge; the only way you can know is if someone tells you (who heard it from someone else, who heard it from God). Since there is no new source of information, it can only be degraded in transmission from generation to generation.
Thus, modern rabbis are not allowed to overrule ancient rabbis. Crawly things are ordinarily unkosher, but it is permissible to eat a worm found in an apple - the ancient rabbis believed the worm was spontaneously generated inside the apple, and therefore was part of the apple. A modern rabbi cannot say, "Yeah, well, the ancient rabbis knew diddly-squat about biology. Overruled!" A modern rabbi cannot possibly know a halachic principle the ancient rabbis did not, because how could the ancient rabbis have passed down the answer from Mount Sinai to him? Knowledge derives from authority, and therefore is only ever lost, not gained, as time passes.
When I was first exposed to the angels-and-donkeys proverb in (religious) elementary school, I was not old enough to be a full-blown atheist, but I still thought to myself: "Torah loses knowledge in every generation. Science gains knowledge with every generation. No matter where they started out, sooner or later science must surpass Torah."
The most important thing is that there should be progress. So long as you keep moving forward you will reach your destination; but if you stop moving you will never reach it.
Tsuyoku naritai is Japanese. Tsuyoku is "strong"; naru is "becoming" and the form naritai is "want to become". Together it means "I want to become stronger" and it expresses a sentiment embodied more intensely in Japanese works than in any Western literature I've read. You might say it when expressing your determination to become a professional Go player - or after you lose an important match, but you haven't given up - or after you win an important match, but you're not a ninth-dan player yet - or after you've become the greatest Go player of all time, but you still think you can do better. That is tsuyoku naritai, the will to transcendence.
Tsuyoku naritai is the driving force behind my essay The Proper Use of Humility, in which I contrast the student who humbly double-checks his math test, and the student who modestly says "But how can we ever really know? No matter how many times I check, I can never be absolutely certain." The student who double-checks his answers wants to become stronger; he reacts to a possible inner flaw by doing what he can to repair the flaw, not with resignation.
Each year on Yom Kippur, an Orthodox Jew recites a litany which begins Ashamnu, bagadnu, gazalnu, dibarnu dofi, and goes on through the entire Hebrew alphabet: We have acted shamefully, we have betrayed, we have stolen, we have slandered...
As you pronounce each word, you strike yourself over the heart in penitence. There's no exemption whereby, if you manage to go without stealing all year long, you can skip the word gazalnu and strike yourself one less time. That would violate the community spirit of Yom Kippur, which is about confessing sins - not avoiding sins so that you have less to confess.
By the same token, the Ashamnu does not end, "But that was this year, and next year I will do better."
The Ashamnu bears a remarkable resemblance to the notion that the way of rationality is to beat your fist against your heart and say, "We are all biased, we are all irrational, we are not fully informed, we are overconfident, we are poorly calibrated..."
Fine. Now tell me how you plan to become less biased, less irrational, more informed, less overconfident, better calibrated.
There is an old Jewish joke: During Yom Kippur, the rabbi is seized by a sudden wave of guilt, and prostrates himself and cries, "God, I am nothing before you!" The cantor is likewise seized by guilt, and cries, "God, I am nothing before you!" Seeing this, the janitor at the back of the synagogue prostrates himself and cries, "God, I am nothing before you!" And the rabbi nudges the cantor and whispers, "Look who thinks he's nothing."
Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking. Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.
Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, "Do you think to set yourself above us?" We will shake our heads sadly and say, "You must not be very self-aware."
Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but that's not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!
March 28, 2007
Tsuyoku vs. the Egalitarian Instinct
Followup to: Tsuyoku naritai! (I Want To Become Stronger)
Hunter-gatherer tribes are usually highly egalitarian (at least if you're male) - the all-powerful tribal chieftain is found mostly in agricultural societies, rarely in the ancestral environment. Among most hunter-gatherer tribes, a hunter who brings in a spectacular kill will carefully downplay the accomplishment to avoid envy.
Maybe, if you start out below average, you can improve yourself without daring to pull ahead of the crowd. But sooner or later, if you aim to do the best you can, you will set your aim above the average.
If you can't admit to yourself that you've done better than others - or if you're ashamed of wanting to do better than others - then the median will forever be your concrete wall, the place where you stop moving forward. And what about people who are below average? Do you dare say you intend to do better than them? How prideful of you!
Maybe it's not healthy to pride yourself on doing better than someone else. Personally I've found it to be a useful motivator, despite my principles, and I'll take all the useful motivation I can get. Maybe that kind of competition is a zero-sum game, but then so is Go; it doesn't mean we should abolish that human activity, if people find it fun and it leads somewhere interesting.
But in any case, surely it isn't healthy to be ashamed of doing better.
And besides, life is not graded on a curve. The will to transcendence has no point beyond which it ceases and becomes the will to do worse; and the race that has no finish line also has no gold or silver medals. Just run as fast as you can, without worrying that you might pull ahead of other runners. (But be warned: If you refuse to worry about that possibility, someday you may pull ahead. If you ignore the consequences, they may happen to you.)
Sooner or later, if your path leads true, you will set out to mitigate a flaw that most people have not mitigated. Sooner or later, if your efforts bring forth any fruit, you will find yourself with fewer sins to confess.
Perhaps you will find it the course of wisdom to downplay the accomplishment, even if you succeed. People may forgive a touchdown, but not dancing in the end zone. You will certainly find it quicker, easier, more convenient, to publicly disclaim your worthiness, to pretend that you are just as much a sinner as everyone else. Just so long, of course, as everyone knows it isn't true. It can be fun to proudly display your modesty, so long as everyone knows how very much you have to be modest about.
But do not let that be the endpoint of your journeys. Even if you only whisper it to yourself, whisper it still: Tsuyoku, tsuyoku! Stronger, stronger!
And then set yourself a higher target. That's the true meaning of the realization that you are still flawed (though a little less so). It means always reaching higher, without shame.
Tsuyoku naritai! I'll always run as fast as I can, even if I pull ahead, I'll keep on running; and someone, someday, will surpass me; but even though I fall behind, I'll always run as fast as I can.
March 30, 2007
"Statistical Bias"
(Part one in a series on "statistical bias", "inductive bias", and "cognitive bias".)
"Bias" as used in the field of statistics refers to directional error in an estimator. Statistical bias is error you cannot correct by repeating the experiment many times and averaging together the results.
The famous bias-variance decomposition states that the expected squared error is equal to the squared directional error, or bias, plus the squared random error, or variance. The law of large numbers says that you can reduce variance, not bias, by repeating the experiment many times and averaging the results.
An experiment has some randomness in it, so if you repeat the experiment many times, you may get slightly different data each time; and if you run a statistical estimator over the data, you may get a slightly different estimate each time. In classical statistics, we regard the true value of the parameter as a constant, and the experimental estimate as a probabilistic variable. The bias is the systematic, or average, difference between these two values; the variance is the leftover probabilistic component.
Let's say you have a repeatable experiment intended to estimate, for example, the height of the Emperor of China. In fact, the Emperor's height is 200 cm. Suppose that every single American believes, without variation, that the Emperor's height is 180 cm. Then if you poll a random American and ask "How tall is the Emperor of China?", the answer is always "180 cm", the error is always -20 cm, and the squared error is always 400 (I shall omit the units on squared errors). But now suppose that Americans have normally distributed beliefs about the Emperor's height, with mean belief 180 cm, and standard deviation 10 cm. You conduct two independent repetitions of the poll, and one American says "190 cm", and the other says "170 cm", with errors respectively of -10 cm and -30 cm, and squared errors of 100 and 900. The average error is -20 cm, as before, but the average squared error is 100 + 900 / 2 = 500. So even though the average (directional) error didn't change as the result of adding noise to the experiments, the average squared error went up.
Although in one case the random perturbation of the answer happened to lead the American in the correct direction - the one who answered 190 cm, which is closer to the true value of 200 cm - the other American was led further away from the answer, replying 170 cm. Since these are equal deviations, the average answer did not change. But since the square increases faster than linear, the larger error corresponded to a still larger squared error, and the average squared error went up.
Furthermore, the new average squared error of 500 equals exactly the square of the directional error (-20 cm) plus the square of the random error (standard deviation of 10cm): 400 + 100 = 500.
In the long run, the above result is universal and exact: If the true value is constant X and the estimator is Y, then E[(X - Y)^2] = (X - E[Y])^2 + E[(E[Y] - Y)^2]. Expected squared error = squared expected bias + expected variance of estimator. This is the bias-variance decomposition.
If we averaged together the two Americans above, we would get an average estimate of 180 cm, with a squared error of 400, which is less than the average error of both experiments taken individually, but still erroneous.
If the true value is constant X and the estimator is Y, then by averaging many estimates together we converge toward the expected value of Y, E[Y], by the law of large numbers, and if we subtract this from X, we are left with a squared error of (X - E[Y])^2, which is the bias term of the bias-variance decomposition. If your estimator is all over the map and highly sensitive to noise in the experiment, then by repeating the experiment many times you can get the expected value of your estimator, and so you are left with only the systematic error of that estimator, and not the random noise in the estimator that varies from experiment to experiment. That's what the law of large numbers is good for.
April 01, 2007
Useful Statistical Biases
Friday's post on statistical bias and the bias-variance decomposition discussed how the squared error of an estimator equals the directional error of the estimator plus the variance of the estimator. All else being equal, bias is bad - you want to get rid of it. But all else is not always equal. Sometimes, by accepting a small amount of bias in your estimator, you can eliminate a large amount of variance. This is known as the "bias-variance tradeoff".
A linear regression tries to estimate a quantity by attaching weights to various signals associated with that quantity - for example, you could try to predict the gas mileage of a car using the car's mass and engine capacity.
A regularized linear regression tries to attach smaller variable weights, while still matching the data fairly well. A regularized regression may generalize to unseen data better than an unregularized regression - often quite a lot better. Assigning smaller variable weights is akin to finding a simpler explanation that fits the data almost as well. This drive for simplicity makes the regressor less sensitive to small random wobbles in the data, so it has lower variance: if you ran the regressor over different data samples, the estimates would look more similar to each other.
But the same regularization procedure also causes the estimator to ignore some actual data - and this is a systematic error, that would recur in the same direction if we repeated the experiment many times. The randomness goes in both directions, so by ignoring the noise in the data, you decrease your variance. But the real evidence goes in one direction, so if you ignore some real evidence in the process of ignoring noise - because you don't know which is which - then you end up with a directional error, an error that trends in the same direction when you repeat the experiment many times.
In statistics this is known as the bias-variance tradeoff. When your data is limited, it may be better to use a simplifying estimator that doesn't try to fit every tiny squiggle of the data, and this trades off a lot of variance against a little bias.
An "unbiased estimator" is one whose expected result equals the correct result, although it may have wide random swings in either direction. This is good if you are allowed to repeat the experiment as often as you like, because you can average together the estimates and get the correct answer to arbitrarily fine precision. That's the law of large numbers.
You might have the following bright idea - why not use an unbiased estimator, like an unregularized regression, to guess the bias of a regularized regression? Then you could just subtract out the systematic bias - you could have low bias and low variance. The problem with this, you see, is that while it may be easy to find an unbiased estimator of the bias, this estimate may have very large variance - so if you subtract out an estimate of the systematic bias, you may end up subtracting out way too much, or even subtracting in the wrong direction a fair fraction of the time. In statistics, "unbiased" is not the same as "good", unless the estimator also has low variance.
When you hear that a classroom gave an average estimate of 871 beans for a jar that contained 850 beans, and that only one individual student did better than the crowd, the astounding notion is not that the crowd can be more accurate than the individual. The astounding notion is that human beings are unbiased estimators of beans in a jar, having no significant directional error on the problem, yet with large variance. It implies that we tend to get the answer wrong but there's no systematic reason why. It requires that there be lots of errors that vary from individual to individual - and this is reliably true, enough so to keep most individuals from guessing the jar correctly. And yet there are no directional errors that everyone makes, or if there are, they cancel out very precisely in the average case, despite the large individual variations. Which is just plain odd. I find myself somewhat suspicious of the claim, and wonder whether other experiments that found less amazing accuracy were not as popularly reported.
Someone is bound to suggest that cognitive biases are useful, in the sense that they represent a bias-variance tradeoff. I think this is just mixing up words - just because the word "bias" is used by two different fields doesn't mean it has the same technical definition. When we accept a statistical bias in trade, we can't get strong information about the direction and magnitude of the bias - otherwise we would just subtract it out. We may be able to get an unbiased estimate of the bias, but "unbiased" is not the same as "reliable"; if the variance is huge, we really have very little information. Now with cognitive biases, we do have some idea of the direction of the systematic error, and the whole notion of "overcoming bias" is about trying to subtract it out. Once again, we see that cognitive biases are lemons, not lemonade. To the extent we can get strong information - e.g. from cognitive psychology experiments - about the direction and magnitude of a systematic cognitive error, we can do systematically better by trying to compensate.
April 01, 2007
The Error of Crowds
I've always been annoyed at the notion that the bias-variance decomposition tells us something about modesty or Philosophical Majoritarianism. For example, Scott Page rearranges the equation to get what he calls the Diversity Prediction Theorem:
Collective Error = Average Individual Error - Prediction Diversity
I think I've finally come up with a nice, mathematical way to drive a stake through the heart of that concept and bury it beneath a crossroads at midnight, though I fully expect that it shall someday rise again and shamble forth to eat the brains of the living.
Why should the bias-variance decomposition be relevant to modesty? Because, it seems to show, the error of averaging all the estimates together, is lower than the typical error of an individual estimate. Prediction Diversity (the variance) is positive when any disagreement exists at all, so Collective Error < Average Individual Error. But then how can you justify keeping your own estimate, unless you know that you did better than average? And how can you legitimately trust that belief, when studies show that everyone believes themselves to be above-average? You should be more modest, and compromise a little.
So what's wrong with this picture?
To begin with, the bias-variance decomposition is a mathematical tautology. It applies when we ask a group of experts to estimate the 2007 close of the NASDAQ index. It would also apply if you weighed the experts on a pound scale and treated the results as estimates of the dollar cost of oil in 2020.
As Einstein put it, "Insofar as the expressions of mathematics refer to reality they are not certain, and insofar as they are certain they do not refer to reality." The real modesty argument, Aumann's Agreement Theorem, has preconditions; AAT depends on agents computing their beliefs in a particular way. AAT's conclusions can be false in any particular case, if the agents don't reason as Bayesians.
The bias-variance decomposition applies to the luminosity of fireflies treated as estimates, just as much as a group of expert opinions. This tells you that you are not dealing with a causal description of how the world works - there are not necessarily any causal quantities, things-in-the-world, that correspond to "collective error" or "prediction diversity". The bias-variance decomposition is not about modesty, communication, sharing of evidence, tolerating different opinions, humbling yourself, overconfidence, or group compromise. It's an algebraic tautology that holds whenever its quantities are defined consistently, even if they refer to the silicon content of pebbles.
More importantly, the tautology depends on a particular definition of "error": error must go as the squared difference between the estimate and the true value. By picking a different error function, just as plausible as the squared difference, you can conjure a diametrically opposed recommendation:
The professor cleared his throat. "All right," he said to the gathered students, "you've each handed in your written estimates of the value of this expression here," and he gestured to a rather complex-looking string of symbols drawn on the blackboard. "Now it so happens," the professor continued, "that this question contains a hidden gotcha. All of you missed in the same direction - that is, you all underestimated or all overestimated the true value, but I won't tell you which. Now, I'm going to take the square root of the amount by which you missed the correct answer, and subtract it from your grade on today's homework. But before I do that, I'm going to give you a chance to revise your answers. You can talk with each other and share your thoughts about the problem, if you like; or alternatively, you could stick your fingers in your ears and hum. Which do you think is wiser?"
Here we are taking the square root of the difference between the true value and the estimate, and calling this the error function, or loss function. (It goes without saying that a student's utility is linear in their grade.)
And now, your expected utility is higher if you pick a random student's estimate than if you pick the average of the class! The students would do worse, on average, by averaging their estimates together! And this again is tautologously true, by Jensen's Inequality.
A brief explanation of Jensen's Inequality:
(I strongly recommend looking at this graph while reading the following.)
Jensen's Inequality says that if X is a probabilistic variable, F(X) is a function of X, and E[expr] stands for the probabilistic expectation of expr, then:
E[F(X)] <= F(E[X]) if F is concave (second derivative negative)
E[F(X)] >= F(E[X]) if F is convex (second derivative positive)
Why? Well, think of two values, x1 and x2. Suppose F is convex - the second derivative is positive, "the cup holds water". Now imagine that we draw a line between x=x1, y=F(x1) and x=x2, y=F(x2). Pick a point halfway along this line. At the halfway point, x will equal (x1 + x2)/2, and y will equal (F(x1)+F(x2))/2. Now draw a vertical line from this halfway point to the curve - the intersection will be at x=(x1 + x2)/2, y=F((x1 + x2)/2). Since the cup holds water, the chord between two points on the curve is above the curve, and we draw the vertical line downward to intersect the curve. Thus F((x1 + x2)/2) < (F(x1) + F(x2))/2. In other words, the F of the average is less than the average of the Fs.
So:
If you define the error as the squared difference, F(x) = x^2 is a convex function, with positive second derivative, and by Jensen's Inequality, the error of the average - F(E[X]) - is less than the average of the errors - E[F(X)]. So, amazingly enough, if you square the differences, the students can do better on average by averaging their estimates. What a surprise.
But in the example above, I defined the error as the square root of the difference, which is a concave function with a negative second derivative. Poof, by Jensen's Inequality, the average error became less than the error of the average. (Actually, I also needed the professor to tell the students that they all erred in the same direction - otherwise, there would be a cusp at zero, and the curve would hold water. The real-world equivalent of this condition is that you think the directional or collective bias is a larger component of the error than individual variance.)
If, in the above dilemma, you think the students would still be wise to share their thoughts with each other, and talk over the math puzzle - I certainly think so - then your belief in the usefulness of conversation has nothing to do with a tautology defined over an error function that happens, in the case of squared error, to be convex. And it follows that you must think the process of sharing thoughts, of arguing differences, is not like averaging your opinions together; or that sticking to your opinion is not like being a random member of the group. Otherwise, you would stuff your fingers in your ears and hum when the problem had a concave error function.
When a line of reasoning starts assigning negative expected utilities to knowledge - offers to pay to avoid true information - I usually consider that a reductio.
April 02, 2007
The Majority Is Always Wrong
Today my coworker Marcello pointed out to me an interesting anti-majoritarian effect. There are three major interpretations of probability: the "subjective" view of probabilities as measuring the uncertainty of agents, the "propensity" view of probabilities as chances inherent within objects, and the "frequentist" view of probabilities as the limiting value of long-run frequencies. I was remarking on how odd it was that frequentism, the predominant view in mainstream statistics, is the worst of the three major alternatives (in my view, you have to presume either uncertainty or propensity in order to talk about the limiting frequency of events that have not yet happened).
And Marcello said something along the lines of, "Well, of course. If anything were worse than frequentism, it wouldn't be there." I said, "What?" And Marcello said, "Like the saying that Mac users have, 'If Macs really were worse than Windows PCs, no one would use them.'"
At this point the light bulb went on over my head - a fluorescent light bulb - and I understood what Marcello was saying: an alternative to frequentism that was even worse than frequentism would have dropped off the radar screens long ago. You can survive by being popular, or by being superior, but alternatives that are neither popular nor superior quickly go extinct.
I can personally testify that Dvorak seems to be much easier on the fingers than Qwerty - but this is not surprising, since if Dvorak really were inferior to Qwerty, it would soon cease to exist. (Yes, I am familiar with the controversy in this area - bear in mind that this is a politically charged topic since it has been used to make accusations of market failure. Nonetheless, my fingers now sweat less, my hands feel less tired, my carpal tunnel syndrome went away, and none of this is surprising because I can feel my fingers traveling shorter distances.)
In any case where you've got (1) a popularity effect (it's easier to use something other people are using) and (2) a most dominant alternative, plus a few smaller niche alternatives, then the most dominant alternative will probably be the worst of the lot - or at least strictly superior to none of the others.
Can anyone else think of examples from their experience where there are several major alternatives that you've heard of, and a popularity effect (which may be as simple as journal editors preferring well-known usages), and the most popular alternative seems to be noticeably the worst?
Addendum: Metahacker said of this hypothesis, "It's wrong, but only sometimes." Sounds about right to me.
April 04, 2007
Knowing About Biases Can Hurt People
Once upon a time I tried to tell my mother about the problem of expert calibration, saying: "So when an expert says they're 99% confident, it only happens about 70% of the time." Then there was a pause as, suddenly, I realized I was talking to my mother, and I hastily added: "Of course, you've got to make sure to apply that skepticism evenhandedly, including to yourself, rather than just using it to argue against anything you disagree with -"
And my mother said: "Are you kidding? This is great! I'm going to use it all the time!"
Taber and Lodge's Motivated skepticism in the evaluation of political beliefs describes the confirmation of six predictions:
- Prior attitude effect. Subjects who feel strongly about an issue - even when encouraged to be objective - will evaluate supportive arguments more favorably than contrary arguments.
- Disconfirmation bias. Subjects will spend more time and cognitive resources denigrating contrary arguments than supportive arguments.
- Confirmation bias. Subjects free to choose their information sources will seek out supportive rather than contrary sources.
- Attitude polarization. Exposing subjects to an apparently balanced set of pro and con arguments will exaggerate their initial polarization.
- Attitude strength effect. Subjects voicing stronger attitudes will be more prone to the above biases.
- Sophistication effect. Politically knowledgeable subjects, because they possess greater ammunition with which to counter-argue incongruent facts and arguments, will be more prone to the above biases.
If you're irrational to start with, having more knowledge can hurt you. For a true Bayesian, information would never have negative expected utility. But humans aren't perfect Bayes-wielders; if we're not careful, we can cut ourselves.
I've seen people severely messed up by their own knowledge of biases. They have more ammunition with which to argue against anything they don't like. And that problem - too much ready ammunition - is one of the primary ways that people with high mental agility end up stupid, in Stanovich's "dysrationalia" sense of stupidity.
You can think of people who fit this description, right? People with high g-factor who end up being less effective because they are too sophisticated as arguers? Do you think you'd be helping them - making them more effective rationalists - if you just told them about a list of classic biases?
I recall someone who learned about the calibration / overconfidence problem. Soon after he said: "Well, you can't trust experts; they're wrong so often as experiments have shown. So therefore, when I predict the future, I prefer to assume that things will continue historically as they have -" and went off into this whole complex, error-prone, highly questionable extrapolation. Somehow, when it came to trusting his own preferred conclusions, all those biases and fallacies seemed much less salient - leapt much less readily to mind - than when he needed to counter-argue someone else.
I told the one about the problem of disconfirmation bias and sophisticated argument, and lo and behold, the next time I said something he didn't like, he accused me of being a sophisticated arguer. He didn't try to point out any particular sophisticated argument, any particular flaw - just shook his head and sighed sadly over how I was apparently using my own intelligence to defeat itself. He had acquired yet another Fully General Counterargument.
Even the notion of a "sophisticated arguer" can be deadly, if it leaps all too readily to mind when you encounter a seemingly intelligent person who says something you don't like.
I endeavor to learn from my mistakes. The last time I gave a talk on heuristics and biases, I started out by introducing the general concept by way of the conjunction fallacy and representativeness heuristic. And then I moved on to confirmation bias, disconfirmation bias, sophisticated argument, motivated skepticism, and other attitude effects. I spent the next thirty minutes hammering on that theme, reintroducing it from as many different perspectives as I could.
I wanted to get my audience interested in the subject. Well, a simple description of conjunction fallacy and representativeness would suffice for that. But suppose they did get interested. Then what? The literature on bias is mostly cognitive psychology for cognitive psychology's sake. I had to give my audience their dire warnings during that one lecture, or they probably wouldn't hear them at all.
Whether I do it on paper, or in speech, I now try to never mention calibration and overconfidence unless I have first talked about disconfirmation bias, motivated skepticism, sophisticated arguers, and dysrationalia in the mentally agile. First, do no harm!
April 07, 2007
Debiasing as Non-Self-Destruction
Nick Bostrom asks:
One sign that science is not all bogus is that it enables us to do things, like go the moon. What practical things does debiassing enable us to do, other than refraining from buying lottery tickets?
It seems to me that how to be smart varies widely between professions. A hedge-fund trader, a research biologist, and a corporate CEO must learn different skill sets in order to be actively excellent - an apprenticeship in one would not serve for the other.
Yet such concepts as "be willing to admit you lost", or "policy debates should not appear one-sided", or "plan to overcome your flaws instead of just confessing them", seem like they could apply to many professions. And all this advice is not so much about how to be extraordinarily clever, as, rather, how to not be stupid. Each profession has its own way to be clever, but their ways of not being stupid have much more in common. And while victors may prefer to attribute victory to their own virtue, my small knowledge of history suggests that far more battles have been lost by stupidity than won by genius.
Debiasing is mostly not about how to be extraordinarily clever, but about how to not be stupid. Its great successes are disasters that do not materialize, defeats that never happen, mistakes that no one sees because they are not made. Often you can't even be sure that something would have gone wrong if you had not tried to debias yourself. You don't always see the bullet that doesn't hit you.
The great victories of debiasing are exactly the lottery tickets we didn't buy - the hopes and dreams we kept in the real world, instead of diverting them into infinitesimal probabilities. The triumphs of debiasing are cults not joined; optimistic assumptions rejected during planning; time not wasted on blind alleys. It is the art of non-self-destruction.
Admittedly, none of this is spectacular enough to make the evening news. It's not a moon landing - though the moon landing did surely require thousands of things to not go wrong.
So how can we know that our debiasing efforts are genuinely useful? Well, this is the worst sort of anecdotal evidence - but people do sometimes ignore my advice, and then, sometimes, catastrophe ensues of just the sort I told them to expect. That is a very weak kind of confirmation, and I would like to see controlled studies... but most of the studies I've read consist of taking a few undergraduates who are in it for the course credit, merely telling them about the bias, and then waiting to see if they improve. What we need is longitudinal studies of life outcomes, and I can think of few people I would name as candidates for the experimental group.
The fact is, most people who take a halfhearted potshot at debiasing themselves do not get huge amounts of mileage out of it. This is one of those things you have to work at for quite a while before you get good at it, especially since there's currently no source of systematic training, or even a decent manual. If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. But until then, mistakes avoided are often just replaced by other mistakes. It takes time for your mind to become significantly quieter. Indeed, a little knowledge of cognitive bias often does more harm than good.
As for public proof, I can see at least three ways that it could come about. First, there might be founded an Order of Bayescraft for people who are serious about it, and the graduates of these dojos might prove systematically more successful even after controlling for measures of fluid intelligence. Second, you could wait for some individual or group, working on an important domain-specific problem but also known for their commitment to debiasing, to produce a spectacularly huge public success. Third, there might be found techniques that can be taught easily and that have readily measureable results; and then simple controlled experiments could serve as public proof, at least for people who attend to Science.
April 08, 2007
"Inductive Bias"
(Part two in a series on "statistical bias", "inductive bias", and "cognitive bias".)
Suppose that you see a swan for the first time, and it is white. It does not follow logically that the next swan you see must be white, but white seems like a better guess than any other color. A machine learning algorithm of the more rigid sort, if it sees a single white swan, may thereafter predict that any swan seen will be white. But this, of course, does not follow logically - though AIs of this sort are often misnamed "logical". For a purely logical reasoner to label the next swan white as a deductive conclusion, it would need an additional assumption: "All swans are the same color." This is a wonderful assumption to make if all swans are, in reality, the same color; otherwise, not so good. Tom Mitchell's Machine Learning defines the inductive bias of a machine learning algorithm as the assumptions that must be added to the observed data to transform the algorithm's outputs into logical deductions.
A more general view of inductive bias would identify it with a Bayesian's prior over sequences of observations...
Consider the case of an urn filled with red and white balls, from which we are to sample without replacement. I might have prior information that the urn contains 5 red balls and 5 white balls. Or, I might have prior information that a random number was selected from a uniform distribution between 0 and 1, and this number was then used as a fixed probability to independently generate a series of 10 balls. In either case, I will estimate a 50% probability that the first ball is red, a 50% probability that the second ball is red, etc., which you might foolishly think indicated the same prior belief. But, while the marginal probabilities on each round are equivalent, the probabilities over sequences are different. In the first case, if I see 3 red balls initially, I will estimate a probability of 2/7 that the next ball will be red. In the second case, if I see 3 red balls initially, I will estimate a 4/5 chance that the next ball will be red (by Laplace's Law of Succession, thus named because it was proved by Thomas Bayes). In both cases we refine our future guesses based on past data, but in opposite directions, which demonstrates the importance of prior information.
Suppose that your prior information about the urn is that a monkey tosses balls into the urn, selecting red balls with 1/4 probability and white balls with 3/4 probability, each ball selected independently. The urn contains 10 balls, and we sample without replacement. (E. T. Jaynes called this the "binomial monkey prior".) Now suppose that on the first three rounds, you see three red balls. What is the probability of seeing a red ball on the fourth round?
First, we calculate the prior probability that the monkey tossed 0 red balls and 10 white balls into the urn; then the prior probability that the monkey tossed 1 red ball and 9 white balls into the urn; and so on. Then we take our evidence (three red balls, sampled without replacement) and calculate the likelihood of seeing that evidence, conditioned on each of the possible urn contents. Then we update and normalize the posterior probability of the possible remaining urn contents. Then we average over the probability of drawing a red ball from each possible urn, weighted by that urn's posterior probability. And the answer is... (scribbles frantically for quite some time)... 1/4!
Of course it's 1/4. We specified that each ball was independently tossed into the urn, with a known 1/4 probability of being red. Imagine that the monkey is tossing the balls to you, one by one; if it tosses you a red ball on one round, that doesn't change the probability that it tosses you a red ball on the next round. When we withdraw one ball from the urn, it doesn't tell us anything about the other balls in the urn.
If you start out with a maximum-entropy prior, then you never learn anything, ever, no matter how much evidence you observe. You do not even learn anything wrong - you always remain as ignorant as you began.
The more inductive bias you have, the faster you learn to predict the future, but only if your inductive bias does in fact concentrate more probability into sequences of observations that actually occur. If your inductive bias concentrates probability into sequences that don't occur, this diverts probability mass from sequences that do occur, and you will learn more slowly, or not learn at all, or even - if you are unlucky enough - learn in the wrong direction.
Inductive biases can be probabilistically correct or probabilistically incorrect, and if they are correct, it is good to have as much of them as possible, and if they are incorrect, you are left worse off than if you had no inductive bias at all. Which is to say that inductive biases are like any other kind of belief; the true ones are good for you, the bad ones are worse than nothing. In contrast, statistical bias is always bad, period - you can trade it off against other ills, but it's never a good thing for itself. Statistical bias is a systematic direction in errors; inductive bias is a systematic direction in belief revisions.
As the example of maximum entropy demonstrates, without a direction to your belief revisions, you end up not revising your beliefs at all. No future prediction based on past experience follows as a matter of strict logical deduction. Which is to say: All learning is induction, and all induction takes place through inductive bias.
Why is inductive bias called "bias"? Because it has systematic qualities, like a statistical bias? Because it is a form of pre-evidential judgment, which resembles the word "prejudice", which resembles the political concept of bias? Damned if I know, really - I'm not the one who decided to call it that. Words are only words; that's why humanity invented mathematics.
April 09, 2007
Futuristic Predictions as Consumable Goods
The Wikipedia entry on Friedman Units tracks over 30 different cases between 2003 and 2007 in which someone labeled the "next six months" as the "critical period in Iraq". Apparently one of the worst offenders is journalist Thomas Friedman after whom the unit was named (8 different predictions in 4 years). In similar news, some of my colleagues in Artificial Intelligence (you know who you are) have been predicting the spectacular success of their projects in "3-5 years" for as long as I've known them, that is, since at least 2000.
Why do futurists make the same mistaken predictions over and over? The same reason politicians abandon campaign promises and switch principles as expediency demands. Predictions, like promises, are sold today and consumed today. They produce a few chewy bites of delicious optimism or delicious horror, and then they're gone. If the tastiest prediction is allegedly about a time interval "3-5 years in the future" (for AI projects) or "6 months in the future" (for Iraq), then futurists will produce tasty predictions of that kind. They have no reason to change the formulation any more than Hershey has to change the composition of its chocolate bars. People won't remember the prediction in 6 months or 3-5 years, any more than chocolate sits around in your stomach for a year and keeps you full.
The futurists probably aren't even doing it deliberately; they themselves have long since digested their own predictions. Can you remember what you had for breakfast on April 9th, 2006? I bet you can't, and I bet you also can't remember what you predicted for "one year from now".
April 11, 2007
Marginally Zero-Sum Efforts
Bostrom recently noted the problem of the commons in labeling efforts "important"; each managerial player has an incentive to label their project world-shakingly important, even though this devalues the priority label as used at other times or other projects, creating positive feedback in inflated labels.
This reminds me of how my grandfather, a pioneer in quantitative genetics, regularly bemoans the need to write more and more grant proposals to maintain a constant level of funding. It's not that the funding is drying up in his field. But suppose there's money for 20 grants, and 21 scientists in need of grants - or one scientist who'd like to run two projects, or receive more funding for one project... One scientist doesn't get his first grant proposal funded, so he writes another one. His second grant proposal does get funded, which uses up a grant that could have gone to another scientist, who now also has his first grant proposal denied, and has to write and send off a second grant proposal too...
The problem here is that, while some initial level of effort is beneficial, all effort beyond that is marginally zero-sum; there's a marginal return to the individual on additional efforts, but no marginal return to the group. If there are 20 grants, then ultimately only 20 grant proposals are going to be funded. No matter how many grant proposals anyone writes, the total funding available remains the same. Everyone would be better off if everyone agreed to write only one grant proposal. But in this case, there wouldn't be much competition for any given grant, and the rewards for writing another two or three grant proposals would be huge... until everyone else started doing the same thing.
There's no obvious limit to this process; the 21 scientists could write 1,000 grant proposals apiece, and still get only 20 grants between them. They'd all be better off if they only wrote one grant proposal apiece; but anyone who cuts back unilaterally will be snowed under.
In a way, this is even worse than the classic problem of the commons. A common grazing field eventually gets eaten down to bedrock and the farmers find something else to do with their herds. When professional efforts are marginally zero-sum, but yield positive returns to the individual, the resulting cycle of busy-work can expand to the limits of individual endurance.
I've often suspected that a similar effect governs bureaucracies (both government and corporate); the longer you stay at your desk each day, the more you are perceived as a hard worker and get promoted. But there's only a limited number of promotions to go around... and only a limited amount of genuinely important work to do.
Social approbation is the usual method for dealing with non-positive-sum actions. Theft has positive returns to the individual, but not positive returns to society, so we put thieves in jail. But in this case, the social dilemma is that neither writing grant proposals, nor showing up at your office desk, is inherently an evil deed. Some grant proposals do need to get written. It's not inherently a zero-sum activity. It's just marginally zero-sum beyond a certain point.
April 11, 2007
Priors as Mathematical Objects
Followup to: "Inductive Bias"
What exactly is a "prior", as a mathematical object? Suppose you're looking at an urn filled with red and white balls. When you draw the very first ball, you haven't yet had a chance to gather much evidence, so you start out with a rather vague and fuzzy expectation of what might happen - you might say "fifty/fifty, even odds" for the chance of getting a red or white ball. But you're ready to revise that estimate for future balls as soon as you've drawn a few samples. So then this initial probability estimate, 0.5, is not repeat not a "prior".
An introduction to Bayes's Rule for confused students might refer to the population frequency of breast cancer as the "prior probability of breast cancer", and the revised probability after a mammography as the "posterior probability". But in the scriptures of Deep Bayesianism, such as Probability Theory: The Logic of Science, one finds a quite different concept - that of prior information, which includes e.g. our beliefs about the sensitivity and specificity of mammography exams. Our belief about the population frequency of breast cancer is only one small element of our prior information.
In my earlier post on inductive bias, I discussed three possible beliefs we might have about an urn of red and white balls, which will be sampled without replacement:
- Case 1: The urn contains 5 red balls and 5 white balls;
- Case 2: A random number was generated between 0 and 1, and each ball was selected to be red (or white) at this probability;
- Case 3: A monkey threw balls into the urn, each with a 50% chance of being red or white.
In each case, if you ask me - before I draw any balls - to estimate my marginal probability that the fourth ball drawn will be red, I will respond "50%". And yet, once I begin observing balls drawn from the urn, I reason from the evidence in three different ways:
- Case 1: Each red ball drawn makes it less likely that future balls will be red, because I believe there are fewer red balls left in the urn.
- Case 2: Each red ball drawn makes it more plausible that future balls will be red, because I will reason that the random number was probably higher, and that the urn is hence more likely to contain mostly red balls.
- Case 3: Observing a red or white ball has no effect on my future estimates, because each ball was independently selected to be red or white at a fixed, known probability.
Suppose I write a Python program to reproduce my reasoning in each of these scenarios. The program will take in a record of balls observed so far, and output an estimate of the probability that the next ball drawn will be red. It turns out that the only necessary information is the count of red balls seen and white balls seen, which we will respectively call R and W. So each program accepts inputs R and W, and outputs the probability that the next ball drawn is red:
- Case 1: return (5 - R)/(10 - R - W) # Number of red balls remaining / total balls remaining
- Case 2: return (R + 1)/(R + W + 2) # Laplace's Law of Succession
- Case 3: return 0.5
These programs are correct so far as they go. But unfortunately, probability theory does not operate on Python programs. Probability theory is an algebra of uncertainty, a calculus of credibility, and Python programs are not allowed in the formulas. It is like trying to add 3 to a toaster oven.
To use these programs in the probability calculus, we must figure out how to convert a Python program into a more convenient mathematical object - say, a probability distribution.
Suppose I want to know the combined probability that the sequence observed will be RWWRR, according to program 2 above. Program 2 does not have a direct faculty for returning the joint or combined probability of a sequence, but it is easy to extract anyway. First, I ask what probability program 2 assigns to observing R, given that no balls have been observed. Program 2 replies "1/2". Then I ask the probability that the next ball is R, given that one red ball has been observed; program 2 replies "2/3". The second ball is actually white, so the joint probability so far is 1/2 * 1/3 = 1/6. Next I ask for the probability that the third ball is red, given that the previous observation is RW; this is summarized as "one red and one white ball", and the answer is 1/2. The third ball is white, so the joint probability for RWW is 1/12. For the fourth ball, given the previous observation RWW, the probability of redness is 2/5, and the joint probability goes to 1/30. We can write this as p(RWWR|RWW) = 2/5, which means that if the sequence so far is RWW, the probability assigned by program 2 to the sequence continuing with R and forming RWWR equals 2/5. And then p(RWWRR|RWWR) = 1/2, and the combined probability is 1/60.
We can do this with every possible sequence of ten balls, and end up with a table of 1024 entries. This table of 1024 entries constitutes a probability distribution over sequences of observations of length 10, and it says everything the Python program had to say (about 10 or fewer observations, anyway). Suppose I have only this probability table, and I want to know the probability that the third ball is red, given that the first two balls drawn were white. I need only sum over the probability of all entries beginning with WWR, and divide by the probability of all entries beginning with WW.
We have thus transformed a program that computes the probability of future events given past experiences, into a probability distribution over sequences of observations.
You wouldn't want to do this in real life, because the Python program is ever so much more compact than a table with 1024 entries. The point is not that we can turn an efficient and compact computer program into a bigger and less efficient giant lookup table; the point is that we can view an inductive learner as a mathematical object, a distribution over sequences, which readily fits into standard probability calculus. We can take a computer program that reasons from experience and think about it using probability theory.
Why might this be convenient? Say that I'm not sure which of these three scenarios best describes the urn - I think it's about equally likely that each of the three cases holds true. How should I reason from my actual observations of the urn? If you think about the problem from the perspective of constructing a computer program that imitates my inferences, it looks complicated - we have to juggle the relative probabilities of each hypothesis, and also the probabilities within each hypothesis. If you think about it from the perspective of probability theory, the obvious thing to do is to add up all three distributions with weightings of 1/3 apiece, yielding a new distribution (which is in fact correct). Then the task is just to turn this new distribution into a computer program, which turns out not to be difficult.
So that is what a prior really is - a mathematical object that represents all of your starting information plus the way you learn from experience.
April 13, 2007
Lotteries: A Waste of Hope
The classic criticism of the lottery is that the people who play are the ones who can least afford to lose; that the lottery is a sink of money, draining wealth from those who most need it. Some lottery advocates, and even some commentors on this blog, have tried to defend lottery-ticket buying as a rational purchase of fantasy - paying a dollar for a day's worth of pleasant anticipation, imagining yourself as a millionaire.
But consider exactly what this implies. It would mean that you're occupying your valuable brain with a fantasy whose real probability is nearly zero - a tiny line of likelihood which you, yourself, can do nothing to realize. The lottery balls will decide your future. The fantasy is of wealth that arrives without effort - without conscientiousness, learning, charisma, or even patience.
Which makes the lottery another kind of sink: a sink of emotional energy. It encourages people to invest their dreams, their hopes for a better future, into an infinitesimal probability. If not for the lottery, maybe they would fantasize about going to technical school, or opening their own business, or getting a promotion at work - things they might be able to actually do, hopes that would make them want to become stronger. Their dreaming brains might, in the 20th visualization of the pleasant fantasy, notice a way to really do it. Isn't that what dreams and brains are for? But how can such reality-limited fare compete with the artificially sweetened prospect of instant wealth - not after herding a dot-com startup through to IPO, but on Tuesday?
Seriously, why can't we just say that buying lottery tickets is stupid? Human beings are stupid, from time to time - it shouldn't be so surprising a hypothesis.
Unsurprisingly, the human brain doesn't do 64-bit floating-point arithmetic, and it can't devalue the emotional force of a pleasant anticipation by a factor of 0.00000001 without dropping the line of reasoning entirely. Unsurprisingly, many people don't realize that a numerical calculation of expected utility ought to override or replace their imprecise financial instincts, and instead treat the calculation as merely one argument to be balanced against their pleasant anticipations - an emotionally weak argument, since it's made up of mere squiggles on paper, instead of visions of fabulous wealth.
This seems sufficient to explain the popularity of lotteries. Why do so many arguers feel impelled to defend this classic form of self-destruction?
The process of overcoming bias requires (1) first noticing the bias, (2) analyzing the bias in detail, (3) deciding that the bias is bad, (4) figuring out a workaround, and then (5) implementing it. It's unfortunate how many people get through steps 1 and 2 and then bog down in step 3, which by rights should be the easiest of the five. Biases are lemons, not lemonade, and we shouldn't try to make lemonade out of them - just burn those lemons down.
April 13, 2007
New Improved Lottery
People are still suggesting that the lottery is not a waste of hope, but a service which enables purchase of fantasy - "daydreaming about becoming a millionaire for much less money than daydreaming about hollywood stars in movies". One commenter wrote: "There is a big difference between zero chance of becoming wealthy, and epsilon. Buying a ticket allows your dream of riches to bridge that gap."
Actually, one of the points I was trying to make is that between zero chance of becoming wealthy, and epsilon chance, there is an order-of-epsilon difference. If you doubt this, let epsilon equal one over googolplex.
Anyway: If we pretend that the lottery sells epsilon hope, this suggests a design for a New Improved Lottery. The New Improved Lottery pays out every five years on average, at a random time - determined, say, by the decay of a not-very-radioactive element. You buy in once, for a single dollar, and get not just a few days of epsilon chance of becoming rich, but a few years of epsilon. Not only that, your wealth could strike at any time! At any minute, the phone could ring to inform you that you, yes, you are a millionaire!
Think of how much better this would be than an ordinary lottery drawing, which only takes place at defined times, a few times per week. Let's say the boss comes in and demands you rework a proposal, or restock inventory, or something similarly annoying. Instead of getting to work, you could turn to the phone and stare, hoping for that call - because there would be epsilon chance that, at that exact moment, you yes you would be awarded the Grand Prize! And even if it doesn't happen this minute, why, there's no need to be disappointed - it might happen the next minute!
Think of how many more fantasies this New Improved Lottery would enable. You could shop at the store, adding expensive items to your shopping cart - if your cellphone doesn't ring with news of a lottery win, you could always put the items back, right?
Maybe the New Improved Lottery could even show a constantly fluctuating probability distribution over the likelihood of a win occurring, and the likelihood of particular numbers being selected, with the overall expectation working out to the aforesaid Poisson distribution. Think of how much fun that would be! Oh, goodness, right this minute the chance of a win occurring is nearly ten times higher than usual! And look, the number 42 that I selected for the Mega Ball has nearly twice the usual chance of winning! You could feed it to a display on people's cellphones, so they could just flip open the cellphone and see their chances of winning. Think of how exciting that would be! Much more exciting than trying to balance your checkbook! Much more exciting than doing your homework! This new dream would be so much tastier that it would compete with, not only hopes of going to technical school, but even hopes of getting home from work early. People could just stay glued to the screen all day long, why, they wouldn't need to dream about anything else!
Yep, offering people tempting daydreams that will not actually happen sure is a valuable service, all right. People are willing to pay, it must be valuable. The alternative is that consumers are making mistakes, and we all know that can't happen.
And yet current governments, with their vile monopoly on lotteries, don't offer this simple and obvious service. Why? Because they want to overcharge people. They want them to spend money every week. They want them to spend a hundred dollars for the thrill of believing their chance of winning is a hundred times as large, instead of being able to stare at a cellphone screen waiting for the likelihood to spike. So if you believe that the lottery is a service, it is clearly an enormously overpriced service - charged to the poorest members of society - and it is your solemn duty as a citizen to demand the New Improved Lottery instead.
April 15, 2007
Your Rationality is My Business
Some responses to Lotteries: A Waste of Hope chided me for daring to criticize others' decisions; if someone else chooses to buy lottery tickets, who am I to disagree? This is a special case of a more general question: What business is it of mine, if someone else chooses to believe what is pleasant rather than what is true? Can't we each choose for ourselves whether to care about the truth?
An obvious snappy comeback is: "Why do you care whether I care whether someone else cares about the truth?" It is somewhat inconsistent for your utility function to contain a negative term for anyone else's utility function having a term for someone else's utility function. But that is only a snappy comeback, not an answer.
So here then is my answer: I believe that it is right and proper for me, as a human being, to have an interest in the future, and what human civilization becomes in the future. One of those interests is the human pursuit of truth, which has strengthened slowly over the generations (for there was not always Science). I wish to strengthen that pursuit further, in this generation. That is a wish of mine, for the Future. For we are all of us players upon that vast gameboard, whether we accept the responsibility or not.
And that makes your rationality my business.
Is this a dangerous idea? Yes, and not just pleasantly edgy "dangerous". People have been burned to death because some priest decided that they didn't think the way they should. Deciding to burn people to death because they "don't think properly" - that's a revolting kind of reasoning, isn't it? You wouldn't want people to think that way, why, it's disgusting. People who think like that, well, we'll have to do something about them...
I agree! Here's my proposal: Let's argue against bad ideas but not set their bearers on fire.
The syllogism we desire to avoid runs: "I think Susie said a bad thing, therefore, Susie should be set on fire." Some try to avoid the syllogism by labeling it improper to think that Susie said a bad thing. No one should judge anyone, ever; anyone who judges is committing a terrible sin, and should be publicly pilloried for it.
As for myself, I deny the therefore. My syllogism runs, "I think Susie said something wrong, therefore, I will argue against what she said, but I will not set her on fire, or try to stop her from talking by violence or regulation..."
We are all of us players upon that vast gameboard; and one of my interests for the Future is to make the game fair. The counterintuitive idea underlying science is that factual disagreements should be fought out with experiments and mathematics, not violence and edicts. This incredible notion can be extended beyond science, to a fair fight for the whole Future. You should have to win by convincing people, and should not be allowed to burn them. This is one of the principles of Rationality, to which I have pledged my allegiance.
People who advocate relativism or selfishness do not appear to me to be truly relativistic or selfish. If they were really relativistic, they would not judge. If they were really selfish, they would get on with making money instead of arguing passionately with others. Rather, they have chosen the side of Relativism, whose goal upon that vast gameboard is to prevent the players - all the players - from making certain kinds of judgments. Or they have chosen the side of Selfishness, whose goal is to make all players selfish. And then they play the game, fairly or unfairly according to their wisdom.
If there are any true Relativists or Selfishes, we do not hear them - they remain silent, non-players.
I cannot help but care how you think, because - as I cannot help but see the universe - each time a human being turns away from the truth, the unfolding story of humankind becomes a little darker. In many cases, it is a small darkness only. (Someone doesn't always end up getting hurt.) Lying to yourself, in the privacy of your own thoughts, does not shadow humanity's history so much as telling public lies or setting people on fire. Yet there is a part of me which cannot help but mourn. And so long as I don't try to set you on fire - only argue with your ideas - I believe that it is right and proper to me, as a human, that I care about my fellow humans. That, also, is a position I defend into the Future.
April 15, 2007
Consolidated Nature of Morality Thread
My intended next OB post will, in passing, distinguish between moral judgments and factual beliefs. Several times before, this has sparked a debate about the nature of morality. (E.g., Believing in Todd.) Such debates often repeat themselves, reinvent the wheel each time, start all over from previous arguments. To avoid this, I suggest consolidating the debate. Whenever someone feels tempted to start a debate about the nature of morality in the comments thread of another post, the comment should be made to this post, instead, with an appropriate link to the article commented upon. Otherwise it does tend to take over discussions like kudzu. (This isn't the first blog/list where I've seen it happen.)
I'll start the ball rolling with ten points to ponder about the nature of morality...
- It certainly looks like there is an important
distinction between a statement like "The total loss of human life
caused by World War II was roughly 72
million people" and "We ought to avoid a repeat of World War
II." Anyone who argues that these statements are of the same
fundamental kind must explain away the apparent structural
differences between them. What are the exact structural
differences?
- We experience some of our morals and preferences as being
voluntary choices, others as involuntary perceptions. I
choose to play on the
side of Rationality, but I don't think I could choose to
believe that death is good any more than I could choose to believe
the sky is green. What psychological factors account for
these differences in my perceptions of my own preferences?
- At a relatively young age, children begin to believe that while
the teacher can make it all right to stand on your chair by giving
permission, the teacher cannot make it all right to steal from
someone else's backpack. (I can't recall the exact citation
on this.) Do young children in a religious environment
believe that God can make it all right to steal from someone's
backpack?
- Both individual human beings and civilizations appear to change
at least some of their moral beliefs over the course of time.
Some of these changes are experienced as "decisions", others are
experienced as "discoveries". Is there a systematic direction
to at least some of these changes? How does this systematic
direction arise causally?
- To paraphrase Alfred Tarski, the statement "My car is painted
green" is true if and only if my car
is painted green. Similarly, someone might try to get away
with asserting that the statement "Human deaths are bad" is true if
and only if human deaths are bad. Is this valid?
- Suppose I involuntarily administered to you a potion which
would cause you to believe that human deaths were good.
Afterward, would you believe truly that human deaths were
good, or would you believe falsely that human deaths were
good?
- Although the statement "My car is painted green" is presently
false, I can make it true at a future time by painting my car
green. However, I can think of no analogous action I could
take which would make it right to kill people. Does this make
the moral statement stronger, weaker, or is there no sense in
making the comparison?
- There does not appear to be any "place" in the environment
where the referents of moral statements are stored, analogous to
the place where my car is stored. Does this necessarily
indicate that moral statements are empty of content, or could they
correspond to something else? Is the statement 2 + 2 = 4
true? Could it be made untrue? Is it falsifiable?
Where is its content?
- The phrase "is/ought" gap refers to the notion that no
ought statement can be logically derived from any number
of is statements, without at least one ought
statement in the mix. For example, suppose I have a remote
control with two buttons, and the red button kills an innocent
prisoner, and the green button sets them free. I cannot
derive the ought-statement, "I ought not to press the red
button", without both the is-statement "If I press the red
button, an innocent will die" and the ought-statement "I
ought not to kill innocents." Should we distinguish
mixed ought-statements like "I ought not to press the red
button" from pure ought-statements like "I ought not to
kill innocents"? If so, is there really any such thing as a
"pure" ought-statement, or do they all have is-statements
mixed into them somewhere?
- The statement "This painting is beautiful" could be rendered
untrue by flinging a bucket of mud on the painting.
Similarly, in the remote-control example above, the statement "It
is wrong to press the red button" can be rendered untrue by
rewiring the remote. Are there pure aesthetic
judgments? Are there pure preferences?
April 26, 2007
Feeling Rational
A popular belief about "rationality" is that rationality opposes all emotion - that all our sadness and all our joy are automatically anti-logical by virtue of being feelings. Yet strangely enough, I can't find any theorem of probability theory which proves that I should appear ice-cold and expressionless.
So is rationality orthogonal to feeling? No; our emotions arise from our models of reality. If I believe that my dead brother has been discovered alive, I will be happy; if I wake up and realize it was a dream, I will be sad. P. C. Hodgell said: "That which can be destroyed by the truth should be." My dreaming self's happiness was opposed by truth. My sadness on waking is rational; there is no truth which destroys it.
Rationality begins by asking how-the-world-is, but spreads virally to any other thought which depends on how we think the world is. By talking about your beliefs about "how-the-world-is", I mean anything you believe is out there in reality, anything that either does or does not exist, any member of the class "things that can make other things happen". If you believe that there is a goblin in your closet that ties your shoe's laces together, then this is a belief about how-the-world-is. Your shoes are real - you can pick them up. If there's something out there which can reach out and tie your shoelaces together, it must be real too, part of the vast web of causes and effects we call the "universe".
Feeling angry at the goblin who tied your shoelaces involves a state of mind that is not just about how-the-world-is. Suppose that, as a Buddhist or a lobotomy patient or just a very phlegmatic person, finding your shoelaces tied together didn't make you angry. This wouldn't affect what you expected to see in the world - you'd still expect to open up your closet and find your shoelaces tied together. Your anger or calm shouldn't affect your best guess here, because what happens in your closet does not depend on your emotional state of mind; though it may take some effort to think that clearly.
But the angry feeling is tangled up with a state of mind that is about how-the-world-is; you become angry because you think the goblin tied your shoelaces. The criterion of rationality spreads virally, from the initial question of whether or not a goblin tied your shoelaces, to the resulting anger.
Becoming more rational - arriving at better estimates of how-the-world-is - can diminish feelings or intensify them. Sometimes we run away from strong feelings by denying the facts, by flinching away from the view of the world that gave rise to the powerful emotion. If so, then as you study the skills of rationality and train yourself not to deny facts, your feelings will become stronger.
In my early days I was never quite certain whether it was all right to feel things strongly - whether it was allowed, whether it was proper. I do not think this confusion arose only from my youthful misunderstanding of rationality. I have observed similar troubles in people who do not even aspire to be rationalists; when they are happy, they wonder if they are really allowed to be happy, and when they are sad, they are never quite sure whether to run away from the emotion or not. Since the days of Socrates at least, and probably long before, the way to appear cultured and sophisticated has been to never let anyone see you care strongly about anything. It's embarrassing to feel - it's just not done in polite society. You should see the strange looks I get when people realize how much I care about rationality. It's not the unusual subject, I think, but that they're not used to seeing sane adults who visibly care about anything.
But I know, now, that there's nothing wrong with
feeling strongly. Ever since I adopted the rule of "That
which can be destroyed by the truth should be," I've also come to
realize "That which the truth nourishes should thrive." When
something good happens, I am happy, and there is no confusion in my
mind about whether it is rational for me to be happy. When
something terrible
happens, I do not flee my sadness by searching for fake
consolations and false silver linings. I visualize the past
and future of humankind, the tens of billions of deaths over our
history, the misery and fear, the search for answers, the trembling
hands reaching upward out of so much blood, what we could become
someday when we make the stars our cities, all that darkness and
all that light - I know that I can never truly understand it, and I
haven't the words to say. Despite all my philosophy I am
still embarrassed to confess strong emotions, and you're probably
uncomfortable hearing them. But I know, now, that it is
rational to feel.
April 27, 2007
Universal Fire
In L. Sprague de Camp's fantasy story The Incomplete Enchanter (which set the mold for the many imitations that followed), the hero, Harold Shea, is transported from our own universe into the universe of Norse mythology. This world is based on magic rather than technology; so naturally, when Our Hero tries to light a fire with a match brought along from Earth, the match fails to strike.
I realize it was only a fantasy story, but... how do I put this...
No.
In the late eighteenth century, Antoine-Laurent de Lavoisier discovered fire. "What?" you say. "Hasn't the use of fire been dated back for hundreds of thousands of years?" Well, yes, people used fire; it was hot, bright, sort of orangey-colored, and you could use it to cook things. But nobody knew how it worked. Greek and medieval alchemists thought that Fire was a basic thing, one of the Four Elements. In Lavoisier's time the alchemical paradigm had been gradually amended and greatly complicated, but fire was still held to be basic - in the form of "phlogiston", a rather mysterious substance which was said to explain fire, and also every other phenomenon in alchemy.
Lavoisier's great innovation was to weigh all the pieces of the chemical puzzle, both before and after the chemical reaction. It had previously been thought that some chemical transmutations changed the weight of the total material: If you subjected finely ground antimony to the focused sunlight of a burning glass, the antimony would be reduced to ashes after one hour, and the ashes would weigh one-tenth more than the original antimony - even though the burning had been accompanied by the loss of a thick white smoke. Lavoisier weighed all the components of such reactions, including the air in which the reaction took place, and discovered that matter was neither created nor destroyed. If the burnt ashes increased in weight, there was a corresponding decrease in the weight of the air.
Lavoisier also knew how to separate gases, and discovered that a burning candle diminished the amount of one kind of gas, vital air, and produced another gas, fixed air. Today we would call them oxygen and carbon dioxide. When the vital air was exhausted, the fire went out. One might guess, perhaps, that combustion transformed vital air into fixed air and fuel to ash, and that the ability of this transformation to continue was limited by the amount of vital air available.
Lavoisier's proposal directly contradicted the then-current phlogiston theory. That alone would have been shocking enough, but it also turned out...
To appreciate what comes next, you must put yourself into an eighteenth-century frame of mind. Forget the discovery of DNA, which occurred only in 1953. Unlearn the cell theory of biology, which was formulated in 1839. Imagine looking at your hand, flexing your fingers... and having absolutely no idea how it worked. The anatomy of muscle and bone was known, but no one had any notion of "what makes it go" - why a muscle moves and flexes, while clay molded into a similar shape just sits there. Imagine your own body being composed of mysterious, incomprehensible gloop. And then, imagine discovering...
...that humans, in the course of breathing, consumed vital air and breathed out fixed air. People also ran on combustion! Lavoisier measured the amount of heat that animals (and Lavoisier's assistant, Seguin) produced when exercising, the amount of vital air consumed, and the fixed air breathed out. When animals produced more heat, they consumed more vital air and exhaled more fixed air. People, like fire, consumed fuel and oxygen; people, like fire, produced heat and carbon dioxide. Deprive people of oxygen, or fuel, and the light goes out.
Matches catch fire because of phosphorus - "safety matches" have phosphorus on the ignition strip; strike-anywhere matches have phosphorus in the match heads. Phosphorus is highly reactive; pure phosphorus glows in the dark and may spontaneously combust. (Henning Brand, who purified phosphorus in 1669, announced that he had discovered Elemental Fire.) Phosphorus is thus also well-suited to its role in adenosine triphosphate, ATP, your body's chief method of storing chemical energy. ATP is sometimes called the "molecular currency". It invigorates your muscles and charges up your neurons. Almost every metabolic reaction in biology relies on ATP, and therefore on the chemical properties of phosphorus.
If a match stops working, so do you. You can't change just one thing.
The surface-level rules, "Matches catch fire when struck," and "Humans need air to breathe," are not obviously connected. It took centuries to discover the connection, and even then, it still seems like some distant fact learned in school, relevant only to a few specialists. It is all too easy to imagine a world where one surface rule holds, and the other doesn't; to suppress our credence in one belief, but not the other. But that is imagination, not reality. If your map breaks into four pieces for easy storage, it doesn't mean the territory is also broken into disconnected parts. Our minds store different surface-level rules in different compartments, but this does not reflect any division in the laws that govern Nature.
We can take the lesson further. Phosphorus derives its behavior from even deeper laws, electrodynamics and chromodynamics. "Phosphorus" is merely our word for electrons and quarks arranged a certain way. You cannot change the chemical properties of phosphorus without changing the laws governing electrons and quarks.
If you stepped into a world where matches failed to strike, you would cease to exist as organized matter.
Reality is laced together a lot more tightly than humans might like to believe.
April 29, 2007
Universal Law
Followup to: Universal Fire
Antoine-Laurent de Lavoisier discovered that breathing (respiration) and fire (combustion) operated on the same principle. It was one of the most startling unifications in the history of science, for it brought together the mundane realm of matter and the sacred realm of life, which humans had divided into separate magisteria.
The first great simplification was that of Isaac Newton, who unified the course of the planets with the trajectory of a falling apple. The shock of this discovery was greater by far than Lavoisier's. It wasn't just that Newton had dared to unify the Earthly realm of base matter with the obviously different and sacred celestial realm, once thought to be the abode of the gods. Newton's discovery gave rise to the notion of a universal law, one that is the same everywhere and everywhen, with literally zero exceptions.
Human beings live in a world of surface phenomena, and surface phenomena are divided into leaky categories with plenty of exceptions. A tiger does not behave like a buffalo. Most buffalo have four legs, but perhaps this one has three. Why would anyone think there would be laws that hold everywhere? It's just so obviously untrue.
The only time when it seems like we would want a law to hold everywhere is when we are talking about moral laws - tribal rules of behavior. Some tribe members may try to take more than their fair share of the buffalo meat - perhaps coming up with some clever excuse - so in the case of moral laws we do seem to have an instinct to universality. Yes, the rule about dividing the meat evenly applies to you, right now, whether you like it or not. But even here there are exceptions. If - for some bizarre reason - a more powerful tribe threatened to spear all of you unless Bob received twice as much meat on just this one occasion, you'd give Bob twice as much meat. The idea of a rule with literally no exceptions seems insanely rigid, the product of closed-minded thinking by fanatics so in the grip of their one big idea that they can't see the richness and complexity of the real universe.
This is the customary accusation made against scientists - the professional students of the richness and complexity of the real universe. Because when you actually look at the universe, it turns out to be, by human standards, insanely rigid in applying its rules. As far as we know, there has been not one single violation of conservation of momentum from the uttermost dawn of time up until now.
Sometimes - very rarely - we observe an apparent violation of our models of the fundamental laws. Though our scientific models may last for a generation or two, they are not stable over the course of centuries... but do not fancy that this makes the universe itself whimsical. That is mixing up the map with the territory. For when the dust subsides and the old theory is overthrown, it turns out that the universe always was acting according to the new generalization we have discovered, which once again is absolutely universal as far as humanity's knowledge extends. When it was discovered that Newtonian gravitation was a special case of General Relativity, it was seen that General Relativity had been governing the orbit of Mercury for decades before any human being knew about it; and it would later become apparent that General Relativity had been governing the collapse of stars for billions of years before humanity. It is only our model that was mistaken - the Law itself was always absolutely constant - or so our new model tells us.
I may repose only 80% confidence that the lightspeed limit will last out the next hundred thousand years, but this does not mean that I think the lightspeed limit holds only 80% of the time, with occasional exceptions. The proposition to which I assign 80% probability is that the lightspeed law is absolutely inviolable throughout the entirety of space and time.
One of the reasons the ancient Greeks didn't discover science is that they didn't realize you could generalize from experiments. The Greek philosophers were interested in "normal" phenomena. If you set up a contrived experiment, you would probably get a "monstrous" result, one that had no implications for how things really worked.
So that is how humans tend to dream, before they learn better; but what of the universe's own quiet dreams that it dreamed to itself before ever it dreamed of humans? If you would learn to think like reality, then here is the Tao:
Since the beginning
not one unusual thing
has ever happened.
May 02, 2007
Think Like Reality
Whenever I hear someone describe quantum physics as "weird" - whenever I hear someone bewailing the mysterious effects of observation on the observed, or the bizarre existence of nonlocal correlations, or the incredible impossibility of knowing position and momentum at the same time - then I think to myself: This person will never understand physics no matter how many books they read.
Reality has been around since long before you showed up. Don't go calling it nasty names like "bizarre" or "incredible". The universe was propagating complex amplitudes through configuration space for ten billion years before life ever emerged on Earth. Quantum physics is not "weird". You are weird. You have the absolutely bizarre idea that reality ought to consist of little billiard balls bopping around, when in fact reality is a perfectly normal cloud of complex amplitude in configuration space. This is your problem, not reality's, and you are the one who needs to change.
Human intuitions were produced by evolution and evolution is a hack. The same optimization process that built your retina backward and then routed the optic cable through your field of vision, also designed your visual system to process persistent objects bouncing around in 3 spatial dimensions because that's what it took to chase down tigers. But "tigers" are leaky surface generalizations - tigers came into existence gradually over evolutionary time, and they are not all absolutely similar to each other. When you go down to the fundamental level, the level on which the laws are stable, global, and exception-free, there aren't any tigers. In fact there aren't any persistent objects bouncing around in 3 spatial dimensions. Deal with it.
Calling reality "weird" keeps you inside a viewpoint already proven erroneous. Probability theory tells us that surprise is the measure of a poor hypothesis; if a model is consistently stupid - consistently hits on events the model assigns tiny probabilities - then it's time to discard that model. A good model makes reality look normal, not weird; a good model assigns high probability to that which is actually the case. Intuition is only a model by another name: poor intuitions are shocked by reality, good intuitions make reality feel natural. You want to reshape your intuitions so that the universe looks normal. You want to think like reality.
This end state cannot be forced. It is pointless to pretend that quantum physics feels natural to you when in fact it feels strange. This is merely denying your confusion, not becoming less confused. But it will also hinder you to keep thinking How bizarre! Spending emotional energy on incredulity wastes time you could be using to update. It repeatedly throws you back into the frame of the old, wrong viewpoint. It feeds your sense of righteous indignation at reality daring to contradict you.
The principle extends beyond physics. Have you ever caught yourself saying something like, "I just don't understand how a PhD physicist can believe in astrology?" Well, if you literally don't understand, this indicates a problem with your model of human psychology. Perhaps you are indignant - you wish to express strong moral disapproval. But if you literally don't understand, then your indignation is stopping you from coming to terms with reality. It shouldn't be hard to imagine how a PhD physicist ends up believing in astrology. People compartmentalize, enough said.
I now try to avoid using the English idiom "I just don't understand how..." to express indignation. If I genuinely don't understand how, then my model is being surprised by the facts, and I should discard it and find a better model.
Surprise exists in the map, not in the territory. There are no surprising facts, only models that are surprised by facts. Likewise for facts called such nasty names as "bizarre", "incredible", "unbelievable", "unexpected", "strange", "anomalous", or "weird". When you find yourself tempted by such labels, it may be wise to check if the alleged fact is really factual. But if the fact checks out, then the problem isn't the fact, it's you.
May 03, 2007
Beware the Unsurprised
In Think Like Reality, I put forth the astonishing and controversial proposition that when human intuitions disagree with a fact, we need to either disprove the "fact" in question, or try to reshape the intuition. (Well, it wouldn't have been so controversial, but like a fool I picked quantum mechanics to illustrate the point. Never use quantum mechanics as an example of anything.) Probability theory says that a model which is consistently surprised on the data is probably not a very good model.
Matt Shulman pointed out in personal conversation that, in practice, we may want to be wary of people who don't appear surprised by surprising-seeming data. Some people affect to be unsurprised because it is a fakeable signal of competence. Well, a lot of things that good rationalists will do - such as appearing skeptical and appearing to take other people's opinions into account - are also fakeable signals of competence. But, in practice, Matt's point is still well-taken.
People may also appear unsurprised (Matt points out) if their models are so vague that they don't understand the implications one way or the other. (Rob Spear: "It doesn't matter to the general public whether reality has 11, 42, or 97.5 dimensions... The primary good that most modern physics provides to the people is basically light entertainment.") Or they may appear unsurprised if they fail to emotionally connect to the implications - "Oh, sure, an asteroid is going to hit Earth... but personally I don't think humanity really deserves to survive anyway... are you taking Sally to her doctor's appointment tomorrow?"
Or Cialdini on the bystander effect:
We can learn from the way the other witnesses are reacting whether the event is or is not an emergency. What is easy to forget, though, is that everybody else observing the event is likely to be looking for social evidence, too. Because we all prefer to appear poised and unflustered among others, we are likely to search for that evidence placidly, with brief, camouflaged glances at those around us. Therefore everyone is likely to see everyone else looking unruffled and failing to act.
So appearing unsurprised, or pretending to yourself that you weren't surprised, is both personally and socially detrimental. By saying that a consistently surprised model is a poor model, I didn't intend to make it more difficult for people to admit their surprise! Even rationalists are surprised sometimes - the important thing is to throw away the model, reshape your intuitions, and otherwise update yourself so that it doesn't happen again.
Think Like Reality wasn't arguing that we should never admit surprise, but that, having been surprised, we shouldn't get all indignant at reality for surprising us - because that just keeps us in the mistaken frame of mind that was surprised in the first place; instead, we should try to adjust our intuitions so that reality doesn't seem surprising the next time. That doesn't mean rationalizing the events in hindsight using your current model - hindsight bias is detrimental to this process because it leads you to underestimate how surprised you were, and hence adjust your model less than it needs to be adjusted.
May 06, 2007
The Third Alternative
"Believing in Santa Claus gives children a sense of wonder and encourages them to behave well in hope of receiving presents. If Santa-belief is destroyed by truth, the children will lose their sense of wonder and stop behaving nicely. Therefore, even though Santa-belief is false-to-fact, it is a Noble Lie whose net benefit should be preserved for utilitarian reasons."
Classically, this is known as a false dilemma, the fallacy of the excluded middle, or the package-deal fallacy. Even if we accept the underlying factual and moral premises of the above argument, it does not carry through. Even supposing that the Santa policy (encourage children to believe in Santa Claus) is better than the null policy (do nothing), it does not follow that Santa-ism is the best of all possible alternatives. Other policies could also supply children with a sense of wonder, such as taking them to watch a Space Shuttle launch or supplying them with science fiction novels. Likewise (if I recall correctly), offering children bribes for good behavior encourages the children to behave well only when adults are watching, while praise without bribes leads to unconditional good behavior.
Noble Lies are generally package-deal fallacies; and the response to a package-deal fallacy is that if we really need the supposed gain, we can construct a Third Alternative for getting it.
How can we obtain Third Alternatives? The first step in obtaining a Third Alternative is deciding to look for one, and the last step is the decision to accept it. This sounds obvious, and yet most people fail on these two steps, rather than within the search process. Where do false dilemmas come from? Some arise honestly, because superior alternatives are cognitively hard to see. But one factory for false dilemmas is justifying a questionable policy by pointing to a supposed benefit over the null action. In this case, the justifier does not want a Third Alternative; finding a Third Alternative would destroy the justification. The last thing a Santa-ist wants to hear is that praise works better than bribes, or that spaceships can be as inspiring as flying reindeer.
The best is the enemy of the good. If the goal is really to help people, then a superior alternative is cause for celebration - once we find this better strategy, we can help people more effectively. But if the goal is to justify a particular strategy by claiming that it helps people, a Third Alternative is an enemy argument, a competitor.
Modern cognitive psychology views decision-making as a search for alternatives. In real life, it's not enough to compare options, you have to generate the options in the first place. On many problems, the number of alternatives is huge, so you need a stopping criterion for the search. When you're looking to buy a house, you can't compare every house in the city; at some point you have to stop looking and decide.
But what about when our conscious motives for the search - the criteria we can admit to ourselves - don't square with subconscious influences? When we are carrying out an allegedly altruistic search, a search for an altruistic policy, and we find a strategy that benefits others but disadvantages ourselves - well, we don't stop looking there; we go on looking. Telling ourselves that we're looking for a strategy that brings greater altruistic benefit, of course. But suppose we find a policy that has some defensible benefit, and also just happens to be personally convenient? Then we stop the search at once! In fact, we'll probably resist any suggestion that we start looking again - pleading lack of time, perhaps. (And yet somehow, we always have cognitive resources for coming up with justifications for our current policy.)
Beware when you find yourself arguing that a policy is defensible rather than optimal; or that it has some benefit compared to the null action, rather than the best benefit of any action.
False dilemmas are often presented to justify unethical policies that are, by some vast coincidence, very convenient. Lying, for example, is often much more convenient than telling the truth; and believing whatever you started out with is more convenient than updating. Hence the popularity of arguments for Noble Lies; it serves as a defense of a pre-existing belief - one does not find Noble Liars who calculate an optimal new Noble Lie; they keep whatever lie they started with. Better stop that search fast!
To do better, ask yourself straight out: If I saw that there was a superior alternative to my current policy, would I be glad in the depths of my heart, or would I feel a tiny flash of reluctance before I let go? If the answers are "no" and "yes", beware that you may not have searched for a Third Alternative.
Which leads into another good question to ask yourself straight out: Did I spend five minutes with my eyes closed, brainstorming wild and creative options, trying to think of a better alternative? It has to be five minutes by the clock, because otherwise you blink - close your eyes and open them again - and say, "Why, yes, I searched for alternatives, but there weren't any." Blinking makes a good black hole down which to dump your duties. An actual, physical clock is recommended.
And those wild and creative options - were you careful not to think of a good one? Was there a secret effort from the corner of your mind to ensure that every option considered would be obviously bad?
It's amazing how many Noble Liars and their ilk are eager to embrace ethical violations - with all due bewailing of their agonies of conscience - when they haven't spent even five minutes by the clock looking for an alternative. There are some mental searches that we secretly wish would fail; and when the prospect of success is uncomfortable, people take the earliest possible excuse to give up.
May 08, 2007
Third Alternatives for Afterlife-ism
One of the most commonly proposed Noble Lies is belief in an afterlife. Surely, goes the argument, the crushing certainty of absolute annihilation in a few decades is too much for any human being to bear. People need hope - if they don't believe in an afterlife, they won't be able to live.
Surely this must be the strongest of all arguments for Noble Lies. You can find Third Alternatives to many dilemmas, but can you find one to Death?
Well, did you close your eyes and think creatively about the problem for five minutes? No excuses, please; just answer "Yes" or "No". Did you, or did you not, brainstorm the problem for five minutes by the clock before giving up?
The assumed task is to find a source of hope against looming death. So at the very least I would cite medical nanotechnology, the argument from actuarial escape velocity, cryonics, or meddling with the forbidden ultimate technology. But do you think that anyone who actually argued for afterlife as a Noble Lie would be glad to hear about these Third Alternatives? No, because the point was not really to find the best strategy for supplying hope, but rather to excuse a fixed previous belief from criticism.
You can argue against the feasibility of one of the above Third Alternatives, or even argue against the feasibility of all of them, but that's not the point. Any one of those Third Alternatives stretches credulity less than a soul - that is (a) an imperishable dualistic stuff floating alongside the brain which (b) malfunctions exactly as the brain is neurologically damaged and yet (c) survives the brain's entire death. Even if we suppose the above Third Alternatives to be false-in-fact, they are packaged with far fewer associated absurdities, and put far less of a strain on the Standard Model.
Thus on the presentation of any one of these Third Alternatives, afterlife-ism stands immediately convicted because it cannot be the best strategy even as a Noble Lie. The old Noble Lie is dominated in the payoff table. If you decided to lie (to others or yourself) to soften the horror of personal extinction, then you'd nudge the balance of evidence a little on actuarial escape velocity - not spin up a soul from whole cloth.
(A truly fanatic rationalist - like me - would refuse to judge between these two lies, regarding them both as equal transgressions of the deontological commandments Thou Shalt Not Nudge Thy Probability Assignments and Thou Shalt Not Pursue Hope As An Emotion, Only Actual Positive Outcomes. Which is still no argument in favor of afterlife-ism; when a negative utility drops off my radar screen and becomes incomparable, I generally don't choose that policy.)
May 13, 2007
Scope Insensitivity
Once upon a time, three groups of subjects were asked how much they would pay to save 2000 / 20000 / 200000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88 [1]. This is scope insensitivity or scope neglect: the number of birds saved - the scope of the altruistic action - had little effect on willingness to pay.
Similar experiments showed that Toronto residents would pay little more to clean up all polluted lakes in Ontario than polluted lakes in a particular region of Ontario [2], or that residents of four western US states would pay only 28% more to protect all 57 wilderness areas in those states than to protect a single area [3].
People visualize "a single exhausted bird, its feathers soaked in black oil, unable to escape" [4]. This image, or prototype, calls forth some level of emotional arousal that is primarily responsible for willingness-to-pay - and the image is the same in all cases. As for scope, it gets tossed out the window - no human can visualize 2000 birds at once, let alone 200000. The usual finding is that exponential increases in scope create linear increases in willingness-to-pay - perhaps corresponding to the linear time for our eyes to glaze over the zeroes; this small amount of affect is added, not multiplied, with the prototype affect. This hypothesis is known as "valuation by prototype".
An alternative hypothesis is "purchase of moral satisfaction". People spend enough money to create a warm glow in themselves, a sense of having done their duty. The level of spending needed to purchase a warm glow depends on personality and financial situation, but it certainly has nothing to do with the number of birds.
We are insensitive to scope even when human lives are at stake: Increasing the alleged risk of chlorinated drinking water from 0.004 to 2.43 annual deaths per 1000 - a factor of 600 - increased willingness-to-pay from $3.78 to $15.23 [5]. Baron and Greene found no effect from varying lives saved by a factor of 10 [6].
A paper entitled Insensitivity to the value of human life: A study of psychophysical numbing collected evidence that our perception of human deaths follows Weber's Law - obeys a logarithmic scale where the "just noticeable difference" is a constant fraction of the whole. A proposed health program to save the lives of Rwandan refugees garnered far higher support when it promised to save 4,500 lives in a camp of 11,000 refugees, rather than 4,500 in a camp of 250,000. A potential disease cure had to promise to save far more lives in order to be judged worthy of funding, if the disease was originally stated to have killed 290,000 rather than 160,000 or 15,000 people per year. [7]
The moral: If you want to be an effective altruist, you have to think it through with the part of your brain that processes those unexciting inky zeroes on paper, not just the part that gets real worked up about that poor struggling oil-soaked bird.
[1] Desvousges, W. Johnson, R. Dunford, R. Boyle, K. J. Hudson, S. and Wilson K. N. (1992). Measuring non-use damages using contingent valuation: experimental evaluation accuracy. Research Triangle Institute Monograph 92-1.
[2] Kahneman, D. 1986. Comments on the contingent valuation method. Pp. 185-194 in Valuing environmental goods: a state of the arts assessment of the contingent valuation method, eds. R. G. Cummings, D. S. Brookshire and W. D. Schulze. Totowa, NJ: Roweman and Allanheld.
[3] McFadden, D. and Leonard, G. 1995. Issues in the contingent valuation of environmental goods: methodologies for data collection and analysis. In Contingent valuation: a critical assessment, ed. J. A. Hausman. Amsterdam: North Holland.
[4] Kahneman, D., Ritov, I. and Schkade, D. A. 1999. Economic Preferences or Attitude Expressions?: An Analysis of Dollar Responses to Public Issues, Journal of Risk and Uncertainty, 19: 203-235.
[5] Carson, R. T. and Mitchell, R. C. 1995. Sequencing and Nesting in Contingent Valuation Surveys. Journal of Environmental Economics and Management, 28(2): 155-73.
[6] Baron, J. and Greene, J. 1996. Determinants of insensitivity to quantity in valuation of public goods: contribution, warm glow, budget constraints, availability, and prominence. Journal of Experimental Psychology: Applied, 2: 107-125.
[7] Fetherstonhaugh, D., Slovic, P., Johnson, S. and Friedrich, J. 1997. Insensitivity to the value of human life: A study of psychophysical numbing. Journal of Risk and Uncertainty, 14: 238-300.
May 18, 2007
One Life Against the World
Followup to: Scope Insensitivity
"Whoever saves a single life, it is as if he had saved the whole world."
-- The Talmud, Sanhedrin 4:5
It's a beautiful thought, isn't it? Feel that warm glow.
I can testify that helping one person feels just as good as helping the whole world. Once upon a time, when I was burned out for the day and wasting time on the Internet - it's a bit complicated, but essentially, I managed to turn someone's whole life around by leaving an anonymous blog comment. I wasn't expecting it to have an effect that large, but it did. When I discovered what I had accomplished, it gave me a tremendous high. The euphoria lasted through that day and into the night, only wearing off somewhat the next morning. It felt just as good (this is the scary part) as the euphoria of a major scientific insight, which had previously been my best referent for what it might feel like to do drugs.
Saving one life probably does feel just as good as being the first person to realize what makes the stars shine. It probably does feel just as good as saving the entire world.
But if you ever have a choice, dear reader, between saving a single life and saving the whole world - then save the world. Please. Because beyond that warm glow is one heck of a gigantic difference.
For some people, the notion that saving the world is significantly better than saving one human life will be obvious, like saying that six billion dollars is worth more than one dollar, or that six cubic kilometers of gold weighs more than one cubic meter of gold. (And never mind the expected value of posterity.) Why might it not be obvious? Well, suppose there's a qualitative duty to save what lives you can - then someone who saves the world, and someone who saves one human life, are just fulfilling the same duty. Or suppose that we follow the Greek conception of personal virtue, rather than consequentialism; someone who saves the world is virtuous, but not six billion times as virtuous as someone who saves one human life. Or perhaps the value of one human life is already too great to comprehend - so that the passing grief we experience at funerals is an infinitesimal underestimate of what is lost - and thus passing to the entire world changes little.
I agree that one human life is of unimaginably high value. I also hold that two human lives are twice as unimaginably valuable. Or to put it another way: Whoever saves one life, if it is as if they had saved the whole world; whoever saves ten lives, it is as if they had saved ten worlds. Whoever actually saves the whole world - not to be confused with pretend rhetorical saving the world - it is as if they had saved an intergalactic civilization.
Two deaf children are sleeping on the railroad tracks, the train speeding down; you see this, but you are too far away to save the child. I'm nearby, within reach, so I leap forward and drag one child off the railroad tracks - and then stop, calmly sipping a Diet Pepsi as the train bears down on the second child. "Quick!" you scream to me. "Do something!" But (I call back) I already saved one child from the train tracks, and thus I am "unimaginably" far ahead on points. Whether I save the second child, or not, I will still be credited with an "unimaginably" good deed. Thus, I have no further motive to act. Doesn't sound right, does it?
Why should it be any different if a philanthropist spends $10
million on curing a rare but spectacularly fatal disease which
afflicts only a hundred people planetwide, when the same money has
an equal probability of producing a cure for a less spectacular
disease that kills 10% of 100,000 people? I don't think it
is different. When human lives are at stake, we have a
duty to maximize, not satisfice; and this duty has the
same strength as the original duty to save lives. Whoever knowingly
chooses to save one life, when they could have saved two - to say
nothing of a thousand lives, or a world - they have damned
themselves as thoroughly as any murderer.
Addendum: It's not cognitively easy to spend money to save lives, since cliche methods that instantly leap to mind don't work or are counterproductive. (I will post later on why this tends to be so.) Stuart Armstrong also points out that if we are to disdain the philanthropist who spends life-saving money inefficiently, we should be consistent and disdain more those who could spend money to save lives but don't.
June 22, 2007
Risk-Free Bonds Aren't
I've always been annoyed by the term "risk-free bonds rate", meaning the return on US Treasury bills. Just because US bonds have not defaulted within their trading experience, people assume this is impossible? A list of major governments in 1900 would probably put the Ottoman Empire or Austria-Hungary well ahead of the relatively young United States. Citing the good track record of the US alone, and not all governments of equal apparent stability at the start of the same time period, is purest survivorship bias.
The United States is a democracy; if enough people vote for representatives who decide not to pay off the bonds, they won't get paid. Do you want to look at recent history, let alone ancient history, and tell me this is impossible? The Internet could enable coordinated populist voting that would sweep new candidates into office, in defiance of prevous political machines. Then the US economy melts under the burden of consumer debt, which causes China to stop buying US bonds and dump its dollar reserves. Then Al Qaeda finally smuggles a nuke into Washington, D.C. Then the next global pandemic hits. And these are just "good stories" - the probability of the US defaulting on its bonds for any reason, is necessarily higher than the probability of it happening for the particular reasons I've just described. I'm not saying these are high probabilities, but they are probabilities. Treasury bills are nowhere near "risk free".
I may be prejudiced here, because I anticipate particular Black Swans (AI, nanotech, biotech) that I see as having a high chance of striking over the lifetime of a 30-year Treasury bond. But even if you don't share those particular assumptions, do you expect the United States to still be around in 300 years? If not, do you know exactly when it will go bust? Then why isn't the risk of losing your capital on a 30-year Treasury bond at least, say, 10%?
Nassim Nicholas Taleb's latest, The Black Swan, is about the impact of unknown unknowns - sudden blowups, processes that seem to behave normally for long periods and then melt down, variables in which most of the movement may occur on a tiny fraction of the moves. Taleb inveighs against the dangers of induction, the ludic fallacy, hindsight, survivorship bias. And then on page 205, Taleb suggests:
Instead of putting your money in "medium risk" investments (how do you know it is medium risk? by listening to tenure-seeking "experts"?), you need to put a portion, say 85 to 90 percent, in extremely safe instruments, like Treasury bills - as safe a class of instruments as you can manage to find on this planet. The remaining 10 to 15 percent you put in extremely speculative bets, as leveraged as possible (like options), preferably venture capital-style portfolios. That way you do not depend on errors of risk management; no Black Swan can hurt you at all, beyond your "floor", the nest egg that you have in maximally safe instruments.
Does Taleb know something I don't, or has he forgotten to apply his own principles in the heat of the moment? (That's a serious question, by the way, if Taleb happens to be reading this. I'm not an experienced trader, and Taleb undoubtedly knows more than I do about how to use Black Swan thinking in trading. But we all know how hard it is to remember to apply our finely honed skepticism in the face of handy popular phrases like "risk-free bonds rate".) Regardless, I think that if you advise your readers to invest 90% of their money in "extremely safe" instruments, you should certainly also warn that it had better not all go into the same instrument - no, not even Treasury bills or gold bullion. There is always risk management, and you are always exposed to error. The safest instruments you can find on this planet aren't very safe.
June 24, 2007
Correspondence Bias
The correspondence bias is the tendency to draw inferences about a person's unique and enduring dispositions from behaviors that can be entirely explained by the situations in which they occur.
-- Gilbert and Malone
We tend to see far too direct a correspondence between others' actions and personalities. When we see someone else kick a vending machine for no visible reason, we assume they are "an angry person". But when you yourself kick the vending machine, it's because the bus was late, the train was early, your report is overdue, and now the damned vending machine has eaten your lunch money for the second day in a row. Surely, you think to yourself, anyone would kick the vending machine, in that situation.
We attribute our own actions to our situations, seeing our behaviors as perfectly normal responses to experience. But when someone else kicks a vending machine, we don't see their past history trailing behind them in the air. We just see the kick, for no reason we know about, and we think this must be a naturally angry person - since they lashed out without any provocation.
Yet consider the prior probabilities. There are more late buses in the world, than mutants born with unnaturally high anger levels that cause them to sometimes spontaneously kick vending machines. Now the average human is, in fact, a mutant. If I recall correctly, an average individual has 2-10 somatically expressed mutations. But any given DNA location is very unlikely to be affected. Similarly, any given aspect of someone's disposition is probably not very far from average. To suggest otherwise is to shoulder a burden of improbability.
Even when people are informed explicitly of situational causes, they don't seem to properly discount the observed behavior. When subjects are told that a pro-abortion or anti-abortion speaker was randomly assigned to give a speech on that position, subjects still think the speakers harbor leanings in the direction randomly assigned. (Jones and Harris 1967, "The attribution of attitudes.)
It seems quite intuitive to explain rain by water spirits; explain fire by a fire-stuff (phlogiston) escaping from burning matter; explain the soporific effect of a medication by saying that it contains a "dormitive potency". Reality usually involves more complicated mechanisms: an evaporation and condensation cycle underlying rain, oxidizing combustion underlying fire, chemical interactions with the nervous system for soporifics. But mechanisms sound more complicated than essences; they are harder to think of, less available. So when someone kicks a vending machine, we think they have an innate vending-machine-kicking-tendency.
Unless the "someone" who kicks the machine is us - in which case we're behaving perfectly normally, given our situations; surely anyone else would do the same. Indeed, we overestimate how likely others are to respond the same way we do - the "false consensus effect". Drinking students considerably overestimate the fraction of fellow students who drink, but nondrinkers considerably underestimate the fraction. The "fundamental attribution error" refers to our tendency to overattribute others' behaviors to their dispositions, while reversing this tendency for ourselves.
To understand why people act the way they do, we must first realize that everyone sees themselves as behaving normally. Don't ask what strange, mutant disposition they were born with, which directly corresponds to their surface behavior. Rather, ask what situations people see themselves as being in. Yes, people do have dispositions - but there are not enough heritable quirks of disposition to directly account for all the surface behaviors you see.
Suppose I gave you a control with two buttons, a red button and a green button. The red button destroys the world, and the green button stops the red button from being pressed. Which button would you press? The green one. Anyone who gives a different answer is probably overcomplicating the question.
And yet people sometimes ask me why I want to save the world. Like I must have had a traumatic childhood or something. Really, it seems like a pretty obvious decision... if you see the situation in those terms.
I may have non-average views which call for explanation - why do I believe such things, when most people don't? - but given those beliefs, my reaction doesn't seem to call forth an exceptional explanation. Perhaps I am a victim of false consensus; perhaps I overestimate how many people would press the green button if they saw the situation in those terms. But y'know, I'd still bet there'd be at least a substantial minority.
Most people see themselves as perfectly normal, from the inside. Even people you hate, people who do terrible things, are not exceptional mutants. No mutations are required, alas. When you understand this, you are ready to stop being surprised by human events.
June 26, 2007
Are Your Enemies Innately Evil?
Followup to: Correspondence Bias
As previously discussed, we see far too direct a correspondence between others' actions and their inherent dispositions. We see unusual dispositions that exactly match the unusual behavior, rather than asking after real situations or imagined situations that could explain the behavior. We hypothesize mutants.
When someone actually offends us - commits an action of which we (rightly or wrongly) disapprove - then, I observe, the correspondence bias redoubles. There seems to be a very strong tendency to blame evil deeds on the Enemy's mutant, evil disposition. Not as a moral point, but as a strict question of prior probability, we should ask what the Enemy might believe about their situation which would reduce the seeming bizarrity of their behavior. This would allow us to hypothesize a less exceptional disposition, and thereby shoulder a lesser burden of improbability.
On September 11th, 2001, nineteen Muslim males hijacked four jet airliners in a deliberately suicidal effort to hurt the United States of America. Now why do you suppose they might have done that? Because they saw the USA as a beacon of freedom to the world, but were born with a mutant disposition that made them hate freedom?
Realistically, most people don't construct their life stories with themselves as the villains. Everyone is the hero of their own story. The Enemy's story, as seen by the Enemy, is not going to make the Enemy look bad. If you try to construe motivations that would make the Enemy look bad, you'll end up flat wrong about what actually goes on in the Enemy's mind.
But politics is the mind-killer. Debate is war; arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the opposing side; otherwise it's like stabbing your soldiers in the back.
If the Enemy did have an evil disposition, that would be an argument in favor of your side. And any argument that favors your side must be supported, no matter how silly - otherwise you're letting up the pressure somewhere on the battlefront. Everyone strives to outshine their neighbor in patriotic denunciation, and no one dares to contradict. Soon the Enemy has horns, bat wings, flaming breath, and fangs that drip corrosive venom. If you deny any aspect of this on merely factual grounds, you are arguing the Enemy's side; you are a traitor. Very few people will understand that you aren't defending the Enemy, just defending the truth.
If it took a mutant to do monstrous things, the history of the human species would look very different. Mutants would be rare.
Or maybe the fear is that understanding will lead to forgiveness. It's easier to shoot down evil mutants. It is a more inspiring battle cry to scream, "Die, vicious scum!" instead of "Die, people who could have been just like me but grew up in a different environment!" You might feel guilty killing people who weren't pure darkness.
This looks to me like the deep-seated yearning for a one-sided policy debate in which the best policy has no drawbacks. If an army is crossing the border or a lunatic is coming at you with a knife, the policy alternatives are (a) defend yourself (b) lie down and die. If you defend yourself, you may have to kill. If you kill someone who could, in another world, have been your friend, that is a tragedy. And it is a tragedy. The other option, lying down and dying, is also a tragedy. Why must there be a non-tragic option? Who says that the best policy available must have no downside? If someone has to die, it may as well be the initiator of force, to discourage future violence and thereby minimize the total sum of death.
If the Enemy has an average disposition, and is acting from beliefs about their situation that would make violence a typically human response, then that doesn't mean their beliefs are factually accurate. It doesn't mean they're justified. It means you'll have to shoot down someone who is the hero of their own story, and in their novel the protagonist will die on page 80. That is a tragedy, but it is better than the alternative tragedy. It is the choice that every police officer makes, every day, to keep our neat little worlds from dissolving into chaos.
When you accurately estimate the Enemy's psychology - when you know what is really in the Enemy's mind - that knowledge won't feel like landing a delicious punch on the opposing side. It won't give you a warm feeling of righteous indignation. It won't make you feel good about yourself. If your estimate makes you feel unbearably sad, you may be seeing the world as it really is. More rarely, an accurate estimate may send shivers of serious horror down your spine, as when dealing with true psychopaths, or neurologically intact people with beliefs that have utterly destroyed their sanity (Scientologists or Jesus Camp).
So let's come right out and say it - the 9/11 hijackers weren't evil mutants. They did not hate freedom. They, too, were the heroes of their own stories, and they died for what they believed was right - truth, justice, and the Islamic way. If the hijackers saw themselves that way, it doesn't mean their beliefs were true. If the hijackers saw themselves that way, it doesn't mean that we have to agree that what they did was justified. If the hijackers saw themselves that way, it doesn't mean that the passengers of United Flight 93 should have stood aside and let it happen. It does mean that in another world, if they had been raised in a different environment, those hijackers might have been police officers. And that is indeed a tragedy. Welcome to Earth.
July 12, 2007
Two More Things to Unlearn from School
In Three Things to Unlearn from School, Ben Casnocha cites Bill Bullard's list of three bad habits of thought: Attaching importance to personal opinions, solving given problems, and earning the approval of others. Bullard's proposed alternatives don't look very good to me, but Bullard has surely identified some important problems.
I can think of other school-inculcated bad habits of thought, too many to list, but I'll name two of my least favorite.
I suspect the most dangerous habit of thought taught in schools is that even if you don't really understand something, you should parrot it back anyway. One of the most fundamental life skills is realizing when you are confused, and school actively destroys this ability - teaches students that they "understand" when they can successfully answer questions on an exam, which is very very very far from absorbing the knowledge and making it a part of you. Students learn the habit that eating consists of putting food into mouth; the exams can't test for chewing or swallowing, and so they starve.
Much of this problem may come from needing to take three 4-credit courses per quarter, with a textbook chapter plus homework to be done every week - the courses are timed for frantic memorization, it's not possible to deeply chew over and leisurely digest knowledge in the same period. College students aren't allowed to be confused; if they started saying, "Wait, do I really understand this? Maybe I'd better spend a few days looking up related papers, or consult another textbook," they'd fail all the courses they took that quarter. A month later they would understand the material far better and remember it much longer - but one month after finals is too late; it counts for nothing in the lunatic university utility function.
Many students who have gone through this process no longer even realize when something confuses them, or notice gaps in their understanding. They have been trained out of pausing to think.
I recall reading, though I can't remember where, that physicists in some country were more likely to become extreme religious fanatics. This confused me, until the author suggested that physics students are presented with a received truth that is actually correct, from which they learn the habit of trusting authority.
It may be dangerous to present people with a giant mass of authoritative knowledge, especially if it is actually true. It may damage their skepticism.
So what could you do? Teach students the history of physics, how each idea was replaced in turn by a new correct one? "Here's the old idea, here's the new idea, here's the experiment - the new idea wins!" Repeat this lesson ten times and what is the habit of thought learned? "New ideas always win; every new idea in physics turns out to be correct." You still haven't taught any critical thinking, because you only showed them history as seen with perfect hindsight. You've taught them the habit that distinguishing true ideas from false ones is perfectly clear-cut and straightforward, so if a shiny new idea has anything to recommend it, it's probably true.
Maybe it would be possible to teach the history of physics from a historically realistic point of view, without benefit of hindsight: show students the different alternatives that were considered historically plausible, re-enact the historical disagreements and debates.
Maybe you could avoid handing students knowledge on a silver platter: show students different versions of physics equations that looked plausible, and ask them to figure out which was the correct one, or invent experiments that would distinguish between alternatives. This wouldn't be as challenging as needing to notice anomalies without hints and invent alternatives from scratch, but it would be a vast improvement over memorizing a received authority.
Then, perhaps, you could teach the habit of thought: "The ideas of received authority are often imperfect but it takes a great effort to find a new idea that is better. Most possible changes are for the worse, even though every improvement is necessarily a change."
July 28, 2007
Making Beliefs Pay Rent (in Anticipated Experiences)
Thus begins the ancient parable:
If a tree falls in a forest and no one hears it, does it make a sound? One says, "Yes it does, for it makes vibrations in the air." Another says, "No it does not, for there is no auditory processing in any brain."
Suppose that, after the tree falls, the two walk into the forest together. Will one expect to see the tree fallen to the right, and the other expect to see the tree fallen to the left? Suppose that before the tree falls, the two leave a sound recorder next to the tree. Would one, playing back the recorder, expect to hear something different from the other? Suppose they attach an electroencephalograph to any brain in the world; would one expect to see a different trace than the other? Though the two argue, one saying "No," and the other saying "Yes," they do not anticipate any different experiences. The two think they have different models of the world, but they have no difference with respect to what they expect will happen to them.
It's tempting to try to eliminate this mistake class by insisting that the only legitimate kind of belief is an anticipation of sensory experience. But the world does, in fact, contain much that is not sensed directly. We don't see the atoms underlying the brick, but the atoms are in fact there. There is a floor beneath your feet, but you don't experience the floor directly; you see the light reflected from the floor, or rather, you see what your retina and visual cortex have processed of that light. To infer the floor from seeing the floor is to step back into the unseen causes of experience. It may seem like a very short and direct step, but it is still a step.
You stand on top of a tall building, next to a grandfather clock with an hour, minute, and ticking second hand. In your hand is a bowling ball, and you drop it off the roof. On which tick of the clock will you hear the crash of the bowling ball hitting the ground?
To answer precisely, you must use beliefs like Earth's gravity is 9.8 meters per second per second, and This building is around 120 meters tall. These beliefs are not wordless anticipations of a sensory experience; they are verbal-ish, propositional. It probably does not exaggerate much to describe these two beliefs as sentences made out of words. But these two beliefs have an inferential consequence that is a direct sensory anticipation - if the clock's second hand is currently on the 12 numeral, you anticipate seeing it move to the 1 numeral before you hear the crash of the bowling ball. To anticipate sensory experiences as precisely as possible, we must process beliefs that are not anticipations of sensory experience.
It is a great strength of Homo sapiens that we can, better than any other species in the world, learn to model the unseen. It is also one of our great weak points. Humans often believe in things that are not only unseen but unreal.
The same brain that builds a network of inferred causes behind sensory experience, can also build a network of causes that is not connected to sensory experience, or poorly connected. Alchemists believed that phlogiston caused fire - we could oversimply their minds by drawing a little node labeled "Phlogiston", and an arrow from this node to their sensory experience of a crackling campfire - but this belief yielded no advance predictions; the link from phlogiston to experience was always configured after the experience, rather than constraining the experience in advance. Or suppose your postmodern English professor teaches you that the famous writer Wulky Wilkinsen is actually a "post-utopian". What does this mean you should expect from his books? Nothing. The belief, if you can call it that, doesn't connect to sensory experience at all. But you had better remember the propositional assertion that "Wulky Wilkinsen" has the "post-utopian" attribute, so you can regurgitate it on the upcoming quiz. Likewise if "post-utopians" show "colonial alienation"; if the quiz asks whether Wulky Wilkinsen shows colonial alienation, you'd better answer yes. The beliefs are connected to each other, though still not connected to any anticipated experience.
We can build up whole networks of beliefs that are connected only to each other - call these "floating" beliefs. It is a uniquely human flaw among animal species, a perversion of Homo sapiens's ability to build more general and flexible belief networks.
The rationalist virtue of empiricism consists of constantly asking which experiences our beliefs predict - or better yet, prohibit. Do you believe that phlogiston is the cause of fire? Then what do you expect to see happen, because of that? Do you believe that Wulky Wilkinsen is a post-utopian? Then what do you expect to see because of that? No, not "colonial alienation"; what experience will happen to you? Do you believe that if a tree falls in the forest, and no one hears it, it still makes a sound? Then what experience must therefore befall you?
It is even better to ask: what experience must not happen to you? Do you believe that elan vital explains the mysterious aliveness of living beings? Then what does this belief not allow to happen - what would definitely falsify this belief? A null answer means that your belief does not constrain experience; it permits anything to happen to you. It floats.
When you argue a seemingly factual question, always keep in mind which difference of anticipation you are arguing about. If you can't find the difference of anticipation, you're probably arguing about labels in your belief network - or even worse, floating beliefs, barnacles on your network. If you don't know what experiences are implied by Wulky Wilkinsen being a post-utopian, you can go on arguing forever. (You can also publish papers forever.)
Above all, don't ask what to believe - ask what to anticipate. Every question of belief should flow from a question of anticipation, and that question of anticipation should be the center of the inquiry. Every guess of belief should begin by flowing to a specific guess of anticipation, and should continue to pay rent in future anticipations. If a belief turns deadbeat, evict it.
July 29, 2007
Belief in Belief
Followup to: Making Beliefs Pay Rent (in Anticipated Experiences)
Carl Sagan once told a parable of a man who comes to us and claims: "There is a dragon in my garage." Fascinating! We reply that we wish to see this dragon - let us set out at once for the garage! "But wait," the claimant says to us, "it is an invisible dragon."
Now as Sagan points out, this doesn't make the hypothesis unfalsifiable. Perhaps we go to the claimant's garage, and although we see no dragon, we hear heavy breathing from no visible source; footprints mysteriously appear on the ground; and instruments show that something in the garage is consuming oxygen and breathing out carbon dioxide.
But now suppose that we say to the claimant, "Okay, we'll visit the garage and see if we can hear heavy breathing," and the claimant quickly says no, it's an inaudible dragon. We propose to measure carbon dioxide in the air, and the claimant says the dragon does not breathe. We propose to toss a bag of flour into the air to see if it outlines an invisible dragon, and the claimant immediately says, "The dragon is permeable to flour."
Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification. But I tell this parable to make a different point: The claimant must have an accurate model of the situation somewhere in his mind, because he can anticipate, in advance, exactly which experimental results he'll need to excuse.
Some philosophers have been much confused by such scenarios, asking, "Does the claimant really believe there's a dragon present, or not?" As if the human brain only had enough disk space to represent one belief at a time! Real minds are more tangled than that. As discussed in yesterday's post, there are different types of belief; not all beliefs are direct anticipations. The claimant clearly does not anticipate seeing anything unusual upon opening the garage door; otherwise he wouldn't make advance excuses. It may also be that the claimant's pool of propositional beliefs contains There is a dragon in my garage. It may seem, to a rationalist, that these two beliefs should collide and conflict even though they are of different types. Yet it is a physical fact that you can write "The sky is green!" next to a picture of a blue sky without the paper bursting into flames.
The rationalist virtue of empiricism is supposed to prevent us from this class of mistake. We're supposed to constantly ask our beliefs which experiences they predict, make them pay rent in anticipation. But the dragon-claimant's problem runs deeper, and cannot be cured with such simple advice. It's not exactly difficult to connect belief in a dragon to anticipated experience of the garage. If you believe there's a dragon in your garage, then you can expect to open up the door and see a dragon. If you don't see a dragon, then that means there's no dragon in your garage. This is pretty straightforward. You can even try it with your own garage.
No, this invisibility business is a symptom of something much worse.
Depending on how your childhood went, you may remember a time period when you first began to doubt Santa Claus's existence, but you still believed that you were supposed to believe in Santa Claus, so you tried to deny the doubts. As Daniel Dennett observes, where it is difficult to believe a thing, it is often much easier to believe that you ought to believe it. What does it mean to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green? The statement is confusing; it's not even clear what it would mean to believe it - what exactly would be believed, if you believed. You can much more easily believe that it is proper, that it is good and virtuous and beneficial, to believe that the Ultimate Cosmic Sky is both perfectly blue and perfectly green. Dennett calls this "belief in belief".
And here things become complicated, as human minds are wont to do - I think even Dennett oversimplifies how this psychology works in practice. For one thing, if you believe in belief, you cannot admit to yourself that you only believe in belief, because it is virtuous to believe, not to believe in belief, and so if you only believe in belief, instead of believing, you are not virtuous. Nobody will admit to themselves, "I don't believe the Ultimate Cosmic Sky is blue and green, but I believe I ought to believe it" - not unless they are unusually capable of acknowledging their own lack of virtue. People don't believe in belief in belief, they just believe in belief.
(Those who find this confusing may find it helpful to study mathematical logic, which trains one to make very sharp distinctions between the proposition P, a proof of P, and a proof that P is provable. There are similarly sharp distinctions between P, wanting P, believing P, wanting to believe P, and believing that you believe P.)
There's different kinds of belief in belief. You may believe in belief explicitly; you may recite in your deliberate stream of consciousness the verbal sentence "It is virtuous to believe that the Ultimate Cosmic Sky is perfectly blue and perfectly green." (While also believing that you believe this, unless you are unusually capable of acknowledging your own lack of virtue.) But there's also less explicit forms of belief in belief. Maybe the dragon-claimant fears the public ridicule that he imagines will result if he publicly confesses he was wrong (although, in fact, a rationalist would congratulate him, and others are more likely to ridicule him if he goes on claiming there's a dragon in his garage). Maybe the dragon-claimant flinches away from the prospect of admitting to himself that there is no dragon, because it conflicts with his self-image as the glorious discoverer of the dragon, who saw in his garage what all others had failed to see.
If all our thoughts were deliberate verbal sentences like philosophers manipulate, the human mind would be a great deal easier for humans to understand. Fleeting mental images, unspoken flinches, desires acted upon without acknowledgement - these account for as much of ourselves as words.
While I disagree with Dennett on some details and complications, I still think that Dennett's notion of belief in belief is the key insight necessary to understand the dragon-claimant. But we need a wider concept of belief, not limited to verbal sentences. "Belief" should include unspoken anticipation-controllers. "Belief in belief" should include unspoken cognitive-behavior-guiders. It is not psychologically realistic to say "The dragon-claimant does not believe there is a dragon in his garage; he believes it is beneficial to believe there is a dragon in his garage." But it is realistic to say the dragon-claimant anticipates as if there is no dragon in his garage, and makes excuses as if he believed in the belief.
You can possess an ordinary mental picture of your garage, with no dragons in it, which correctly predicts your experiences on opening the door, and never once think the verbal phrase There is no dragon in my garage. I even bet it's happened to you - that when you open your garage door or bedroom door or whatever, and expect to see no dragons, no such verbal phrase runs through your mind.
And to flinch away from giving up your belief in the dragon - or flinch away from giving up your self-image as a person who believes in the dragon - it is not necessary to explicitly think I want to believe there's a dragon in my garage. It is only necessary to flinch away from the prospect of admitting you don't believe.
To correctly anticipate, in advance, which experimental results shall need to be excused, the dragon-claimant must (a) possess an accurate anticipation-controlling model somewhere in his mind, and (b) act cognitively to protect either (b1) his free-floating propositional belief in the dragon or (b2) his self-image of believing in the dragon.
If someone believes in their belief in the dragon, and also believes in the dragon, the problem is much less severe. They will be willing to stick their neck out on experimental predictions, and perhaps even agree to give up the belief if the experimental prediction is wrong - although belief in belief can still interfere with this, if the belief itself is not absolutely confident. When someone makes up excuses in advance, it would seem to require that belief, and belief in belief, have become unsynchronized.
July 31, 2007
Bayesian Judo
You can have some fun with people whose anticipations get out of sync with what they believe they believe.
I was once at a dinner party, trying to explain to a man what I did for a living, when he said: "I don't believe Artificial Intelligence is possible because only God can make a soul."
At this point I must have been divinely inspired, because I instantly responded: "You mean if I can make an Artificial Intelligence, it proves your religion is false?"
He said, "What?"
I said, "Well, if your religion predicts that I can't possibly make an Artificial Intelligence, then, if I make an Artificial Intelligence, it means your religion is false. Either your religion allows that it might be possible for me to build an AI; or, if I build an AI, that disproves your religion."
There was a pause, as the one realized he had just made his hypothesis vulnerable to falsification, and then he said, "Well, I didn't mean that you couldn't make an intelligence, just that it couldn't be emotional in the same way we are."
I said, "So if I make an Artificial Intelligence that, without being deliberately preprogrammed with any sort of script, starts talking about an emotional life that sounds like ours, that means your religion is wrong."
He said, "Well, um, I guess we may have to agree to disagree on this."
I said: "No, we can't, actually. There's a theorem of rationality called Aumann's Agreement Theorem which shows that no two rationalists can agree to disagree. If two people disagree with each other, at least one of them must be doing something wrong."
We went back and forth on this briefly. Finally, he said, "Well, I guess I was really trying to say that I don't think you can make something eternal."
I said, "Well, I don't think so either! I'm glad we were able to reach agreement on this, as Aumann's Agreement Theorem requires." I stretched out my hand, and he shook it, and then he wandered away.
A woman who had stood nearby, listening to the conversation, said to me gravely, "That was beautiful."
"Thank you very much," I said.
August 02, 2007
Professing and Cheering
I once attended a panel on the topic, "Are science and religion compatible?" One of the women on the panel, a pagan, held forth interminably upon how she believed that the Earth had been created when a giant primordial cow was born into the primordial abyss, who licked a primordial god into existence, whose descendants killed a primordial giant and used its corpse to create the Earth, etc. The tale was long, and detailed, and more absurd than the Earth being supported on the back of a giant turtle. And the speaker clearly knew enough science to know this.
I still find myself struggling for words to describe what I saw as this woman spoke. She spoke with... pride? Self-satisfaction? A deliberate flaunting of herself?
The woman went on describing her creation myth for what seemed like forever, but was probably only five minutes. That strange pride/satisfaction/flaunting clearly had something to do with her knowing that her beliefs were scientifically outrageous. And it wasn't that she hated science; as a panelist she professed that religion and science were compatible. She even talked about how it was quite understandable that the Vikings talked about a primordial abyss, given the land in which they lived - explained away her own religion! - and yet nonetheless insisted this was what she "believed", said with peculiar satisfaction.
I'm not sure that Daniel Dennett's concept of "belief in belief" stretches to cover this event. It was weirder than that. She didn't recite her creation myth with the fanatical faith of someone who needs to reassure herself. She didn't act like she expected us, the audience, to be convinced - or like she needed our belief to validate her.
Dennett, in addition to suggesting belief in belief, has also suggested that much of what is called "religious belief" should really be studied as "religious profession". Suppose an alien anthropologist studied a group of postmodernist English students who all seemingly believed that Wulky Wilkensen was a post-utopian author. The appropriate question may not be "Why do the students all believe this strange belief?" but "Why do they all write this strange sentence on quizzes?" Even if a sentence is essentially meaningless, you can still know when you are supposed to chant the response aloud.
I think Dennett may be slightly too cynical in suggesting that religious profession is just saying the belief aloud - most people are honest enough that, if they say a religious statement aloud, they will also feel obligated to say the verbal sentence into their own stream of consciousness.
But even the concept of "religious profession" doesn't seem to cover the pagan woman's claim to believe in the primordial cow. If you had to profess a religious belief to satisfy a priest, or satisfy a co-religionist - heck, to satisfy your own self-image as a religious person - you would have to pretend to believe much more convincingly than this woman was doing. As she recited her tale of the primordial cow, with that same strange flaunting pride, she wasn't even trying to be persuasive - wasn't even trying to convince us that she took her own religion seriously. I think that's the part that so took me aback. I know people who believe they believe ridiculous things, but when they profess them, they'll spend much more effort to convince themselves that they take their beliefs seriously.
It finally occurred to me that this woman wasn't trying to convince us or even convince herself. Her recitation of the creation story wasn't about the creation of the world at all. Rather, by launching into a five-minute diatribe about the primordial cow, she was cheering for paganism, like holding up a banner at a football game. A banner saying "GO BLUES" isn't a statement of fact, or an attempt to persuade; it doesn't have to be convincing - it's a cheer.
That strange flaunting pride... it was like she was marching naked in a gay pride parade. (Incidentally, I'd have no objection if she had marched naked in a gay pride parade. Lesbianism is not something that truth can destroy.) It wasn't just a cheer, like marching, but an outrageous cheer, like marching naked - believing that she couldn't be arrested or criticized, because she was doing it for her pride parade.
That's why it mattered to her that what she was saying was beyond ridiculous. If she'd tried to make it sound more plausible, it would have been like putting on clothes.
August 02, 2007
Belief as Attire
I have so far distinguished between belief as anticipation-controller, belief in belief, professing and cheering. Of these, we might call anticipation-controlling beliefs "proper beliefs" and the other forms "improper belief". A proper belief can be wrong or irrational, e.g., someone who genuinely anticipates that prayer will cure her sick baby, but the other forms are arguably "not belief at all".
Yet another form of improper belief is belief as group-identification - as a way of belonging. Robin Hanson uses the excellent metaphor of wearing unusual clothing, a group uniform like a priest's vestments or a Jewish skullcap, and so I will call this "belief as attire".
In terms of humanly realistic psychology, the Muslims who flew planes into the World Trade Center undoubtedly saw themselves as heroes defending truth, justice, and the Islamic Way from hideous alien monsters a la the movie Independence Day. Only a very inexperienced nerd, the sort of nerd who has no idea how non-nerds see the world, would say this out loud in an Alabama bar. It is not an American thing to say. The American thing to say is that the terrorists "hate our freedom" and that flying a plane into a building is a "cowardly act". You cannot say the phrases "heroic self-sacrifice" and "suicide bomber" in the same sentence, even for the sake of accurately describing how the Enemy sees the world. The very concept of the courage and altruism of a suicide bomber is Enemy attire - you can tell, because the Enemy talks about it. The cowardice and sociopathy of a suicide bomber is American attire. There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.
Belief-as-attire may help explain how people can be passionate about improper beliefs. Mere belief in belief, or religious professing, would have some trouble creating genuine, deep, powerful emotional effects. Or so I suspect; I confess I'm not an expert here. But my impression is this: People who've stopped anticipating-as-if their religion is true, will go to great lengths to convince themselves they are passionate, and this desperation can be mistaken for passion. But it's not the same fire they had as a child.
On the other hand, it is very easy for a human being to genuinely, passionately, gut-level belong to a group, to cheer for their favorite sports team. (This is the foundation on which rests the swindle of "Republicans vs. Democrats" and analogous false dilemmas in other countries, but that's a topic for another post.) Identifying with a tribe is a very strong emotional force. People will die for it. And once you get people to identify with a tribe, the beliefs which are attire of that tribe will be spoken with the full passion of belonging to that tribe.
August 03, 2007
Religion's Claim to be Non-Disprovable
The earliest account I know of a scientific experiment is, ironically, the story of Elijah and the priests of Baal.
The people of Israel are wavering between Jehovah and Baal, so Elijah announces that he will conduct an experiment to settle it - quite a novel concept in those days! The priests of Baal will place their bull on an altar, and Elijah will place Jehovah's bull on an altar, but neither will be allowed to start the fire; whichever God is real will call down fire on His sacrifice. The priests of Baal serve as control group for Elijah - the same wooden fuel, the same bull, and the same priests making invocations, but to a false god. Then Elijah pours water on his altar - ruining the experimental symmetry, but this was back in the early days - to signify deliberate acceptance of the burden of proof, like needing a 0.05 significance level. The fire comes down on Elijah's altar, which is the experimental observation. The watching people of Israel shout "The Lord is God!" - peer review.
And then the people haul the 450 priests of Baal down to the river Kishon and slit their throats. This is stern, but necessary. You must firmly discard the falsified hypothesis, and do so swiftly, before it can generate excuses to protect itself. If the priests of Baal are allowed to survive, they will start babbling about how religion is a separate magisterium which can be neither proven nor disproven.
Back in the old days, people actually believed their religions instead of just believing in them. The biblical archaeologists who went in search of Noah's Ark did not think they were wasting their time; they anticipated they might become famous. Only after failing to find confirming evidence - and finding disconfirming evidence in its place - did religionists execute what William Bartley called the retreat to commitment, "I believe because I believe."
Back in the old days, there was no concept of religion being a separate magisterium. The Old Testament is a stream-of-consciousness culture dump: history, law, moral parables, and yes, models of how the universe works. In not one single passage of the Old Testament will you find anyone talking about a transcendent wonder at the complexity of the universe. But you will find plenty of scientific claims, like the universe being created in six days (which is a metaphor for the Big Bang), or rabbits chewing their cud and grasshoppers having four legs. (Which is a metaphor for...)
Back in the old days, saying the local religion "could not be proven" would have gotten you burned at the stake. One of the core beliefs of Orthodox Judaism is that God appeared at Mount Sinai and said in a thundering voice, "Yeah, it's all true." From a Bayesian perspective that's some darned unambiguous evidence of a superhumanly powerful entity. (Albeit it doesn't prove that the entity is God per se, or that the entity is benevolent - it could be alien teenagers.) The vast majority of religions in human history - excepting only those invented extremely recently - tell stories of events that would constitute completely unmistakable evidence if they'd actually happened. The orthogonality of religion and factual questions is a recent and strictly Western concept. The people who wrote the original scriptures didn't even know the difference.
The Roman Empire inherited philosophy from the ancient Greeks; imposed law and order within its provinces; kept bureaucratic records; and enforced religious tolerance. The New Testament, created during the time of the Roman Empire, bears some traces of modernity as a result. You couldn't invent a story about God completely obliterating the city of Rome (a la Sodom and Gomorrah), because the Roman historians would call you on it, and you couldn't just stone them.
In contrast, the people who invented the Old Testament stories could make up pretty much anything they liked. Early Egyptologists were genuinely shocked to find no trace whatsoever of Hebrew tribes having ever been in Egypt - they weren't expecting to find a record of the Ten Plagues, but they expected to find something. As it turned out, they did find something. They found out that, during the supposed time of the Exodus, Egypt ruled much of Canaan. That's one huge historical error, but if there are no libraries, nobody can call you on it.
The Roman Empire did have libraries. Thus, the New Testament doesn't claim big, showy, large-scale geopolitical miracles as the Old Testament routinely did. Instead the New Testament claims smaller miracles which nonetheless fit into the same framework of evidence. A boy falls down and froths at the mouth; the cause is an unclean spirit; an unclean spirit could reasonably be expected to flee from a true prophet, but not to flee from a charlatan; Jesus casts out the unclean spirit; therefore Jesus is a true prophet and not a charlatan. This is perfectly ordinary Bayesian reasoning, if you grant the basic premise that epilepsy is caused by demons (and that the end of an epileptic fit proves the demon fled).
Not only did religion used to make claims about factual and scientific matters, religion used to make claims about everything. Religion laid down a code of law - before legislative bodies; religion laid down history - before historians and archaeologists; religion laid down the sexual morals - before Women's Lib; religion described the forms of government - before constitutions; and religion answered scientific questions from biological taxonomy to the formation of stars. The Old Testament doesn't talk about a sense of wonder at the complexity of the universe - it was busy laying down the death penalty for women who wore men's clothing, which was solid and satisfying religious content of that era. The modern concept of religion as purely ethical derives from every other area having been taken over by better institutions. Ethics is what's left.
Or rather, people think ethics is what's left. Take a culture dump from 2,500 years ago. Over time, humanity will progress immensely, and pieces of the ancient culture dump will become ever more glaringly obsolete. Ethics has not been immune to human progress - for example, we now frown upon such Bible-approved practices as keeping slaves. Why do people think that ethics is still fair game?
Intrinsically, there's nothing small about the ethical problem with slaughtering thousands of innocent first-born male children to convince an unelected Pharaoh to release slaves who logically could have been teleported out of the country. It should be more glaring than the comparatively trivial scientific error of saying that grasshoppers have four legs. And yet, if you say the Earth is flat, people will look at you like you're crazy. But if you say the Bible is your source of ethics, women will not slap you. Most people's concept of rationality is determined by what they think they can get away with; they think they can get away with endorsing Bible ethics; and so it only requires a manageable effort of self-deception for them to overlook the Bible's moral problems. Everyone has agreed not to notice the elephant in the living room, and this state of affairs can sustain itself for a time.
Maybe someday, humanity will advance further, and anyone who endorses the Bible as a source of ethics will be treated the same way as Trent Lott endorsing Strom Thurmond's presidential campaign. And then it will be said that religion's "true core" has always been genealogy or something.
The idea that religion is a separate magisterium which cannot be proven or disproven is a Big Lie - a lie which is repeated over and over again, so that people will say it without thinking; yet which is, on critical examination, simply false. It is a wild distortion of how religion happened historically, of how all scriptures present their beliefs, of what children are told to persuade them, and of what the majority of religious people on Earth still believe. You have to admire its sheer brazenness, on a par with Oceania has always been at war with Eastasia. The prosecutor whips out the bloody axe, and the defendant, momentarily shocked, thinks quickly and says: "But you can't disprove my innocence by mere evidence - it's a separate magisterium!"
And if that doesn't work, grab a piece of paper and scribble yourself a Get-Out-of-Jail-Free card.
August 04, 2007
The Importance of Saying "Oops"
I just finished reading a history of Enron's downfall, The Smartest Guys in the Room, which hereby wins my award for "Least Appropriate Book Title".
An unsurprising feature of Enron's slow rot and abrupt collapse was that the executive players never admitted to having made a large mistake. When catastrophe #247 grew to such an extent that it required an actual policy change, they would say "Too bad that didn't work out - it was such a good idea - how are we going to hide the problem on our balance sheet?" As opposed to, "It now seems obvious in retrospect that it was a mistake from the beginning." As opposed to, "I've been stupid." There was never a watershed moment, a moment of humbling realization, of acknowledging a fundamental problem. After the bankruptcy, Jeff Skilling, the former COO and brief CEO of Enron, declined his own lawyers' advice to take the Fifth Amendment; he testified before Congress that Enron had been a great company.
Not every change is an improvement, but every improvement is necessarily a change. If we only admit small local errors, we will only make small local changes. The motivation for a big change comes from acknowledging a big mistake.
As a child I was raised on equal parts science and science fiction, and from Heinlein to Feynman I learned the tropes of Traditional Rationality: Theories must be bold and expose themselves to falsification; be willing to commit the heroic sacrifice of giving up your own ideas when confronted with contrary evidence; play nice in your arguments; try not to deceive yourself; and other fuzzy verbalisms.
A traditional rationalist upbringing tries to produce arguers who will concede to contrary evidence eventually - there should be some mountain of evidence sufficient to move you. This is not trivial; it distinguishes science from religion. But there is less focus on speed, on giving up the fight as quickly as possible, integrating evidence efficiently so that it only takes a minimum of contrary evidence to destroy your cherished belief.
I was raised in Traditional Rationality, and thought myself quite the rationalist. I switched to Bayescraft (Laplace/Jaynes/Tversky/Kahneman) in the aftermath of... well, it's a long story. Roughly, I switched because I realized that Traditional Rationality's fuzzy verbal tropes had been insufficient to prevent me from making a large mistake.
After I had finally and fully admitted my mistake, I looked back
upon the path that had led me to my Awful Realization. And I
saw that I had made a series of small concessions, minimal
concessions, grudgingly conceding each millimeter of ground,
realizing as little as possible of my mistake on each occasion,
admitting failure only in small tolerable nibbles. I could
have moved so much faster, I realized, if I had simply screamed
"OOPS!"
And I thought: I must raise the level of my game.
There is a powerful advantage to admitting you have made a large mistake. It's painful. It can also change your whole life.
It is important to have the watershed moment, the moment of humbling realization. To acknowledge a fundamental problem, not divide it into palatable bite-size mistakes.
Do not indulge in drama and become proud of admitting errors. It is surely superior to get it right the first time. But if you do make an error, better by far to see it all at once. Even hedonically, it is better to take one large loss than many small ones. The alternative is stretching out the battle with yourself over years. The alternative is Enron.
Since then I have watched others making their own series of minimal concessions, grudgingly conceding each millimeter of ground; never confessing a global mistake where a local one will do; always learning as little as possible from each error. What they could fix in one fell swoop voluntarily, they transform into tiny local patches they must be argued into. Never do they say, after confessing one mistake, I've been a fool. They do their best to minimize their embarrassment by saying I was right in principle, or It could have worked, or I still want to embrace the true essence of whatever-I'm-attached-to. Defending their pride in this passing moment, they ensure they will again make the same mistake, and again need to defend their pride.
Better to swallow the entire bitter pill in one terrible gulp.
August 05, 2007
Focus Your Uncertainty
Will bond yields go up, or down, or remain the same? If you're a TV pundit and your job is to explain the outcome after the fact, then there's no reason to worry. No matter which of the three possibilities comes true, you'll be able to explain why the outcome perfectly fits your pet market theory . There's no reason to think of these three possibilities as somehow opposed to one another, as exclusive, because you'll get full marks for punditry no matter which outcome occurs.
But wait! Suppose you're a novice TV pundit, and you aren't experienced enough to make up plausible explanations on the spot. You need to prepare remarks in advance for tomorrow's broadcast, and you have limited time to prepare. In this case, it would be helpful to know which outcome will actually occur - whether bond yields will go up, down, or remain the same - because then you would only need to prepare one set of excuses.
Alas, no one can possibly foresee the future. What are you to do? You certainly can't use "probabilities". We all know from school that "probabilities" are little numbers that appear next to a word problem, and there aren't any little numbers here. Worse, you feel uncertain. You don't remember feeling uncertain while you were manipulating the little numbers in word problems. College classes teaching math are nice clean places, therefore math itself can't apply to life situations that aren't nice and clean. You wouldn't want to inappropriately transfer thinking skills from one context to another. Clearly, this is not a matter for "probabilities".
Nonetheless, you only have 100 minutes to prepare your excuses. You can't spend the entire 100 minutes on "up", and also spend all 100 minutes on "down", and also spend all 100 minutes on "same". You've got to prioritize somehow.
If you needed to justify your time expenditure to a review committee, you would have to spend equal time on each possibility. Since there are no little numbers written down, you'd have no documentation to justify spending different amounts of time. You can hear the reviewers now: And why, Mr. Finkledinger, did you spend exactly 42 minutes on excuse #3? Why not 41 minutes, or 43? Admit it - you're not being objective! You're playing subjective favorites!
But, you realize with a small flash of relief, there's no review committee to scold you. This is good, because there's a major Federal Reserve announcement tomorrow, and it seems unlikely that bond prices will remain the same. You don't want to spend 33 precious minutes on an excuse you don't anticipate needing.
Your mind keeps drifting to the explanations you use on television, of why each event plausibly fits your market theory. But it rapidly becomes clear that plausibility can't help you here - all three events are plausible. Fittability to your pet market theory doesn't tell you how to divide your time. There's an uncrossable gap between your 100 minutes of time, which are conserved; versus your ability to explain how an outcome fits your theory, which is unlimited.
And yet... even in your uncertain state of mind, it seems that you anticipate the three events differently; that you expect to need some excuses more than others. And - this is the fascinating part - when you think of something that makes it seem more likely that bond prices will go up, then you feel less likely to need an excuse for bond prices going down or remaining the same.
It even seems like there's a relation between how much you anticipate each of the three outcomes, and how much time you want to spend preparing each excuse. Of course the relation can't actually be quantified. You have 100 minutes to prepare your speech, but there isn't 100 of anything to divide up in this anticipation business. (Although you do work out that, if some particular outcome occurs, then your utility function is logarithmic in time spent preparing the excuse.)
Still... your mind keeps coming back to the idea that anticipation is limited, unlike excusability, but like time to prepare excuses. Maybe anticipation should be treated as a conserved resource, like money. Your first impulse is to try to get more anticipation, but you soon realize that, even if you get more anticiptaion, you won't have any more time to prepare your excuses. No, your only course is to allocate your limited supply of anticipation as best you can.
You're pretty sure you weren't taught anything like that in your statistics courses. They didn't tell you what to do when you felt so terribly uncertain. They didn't tell you what to do when there were no little numbers handed to you. Why, even if you tried to use numbers, you might end up using any sort of numbers at all - there's no hint what kind of math to use, if you should be using math! Maybe you'd end up using pairs of numbers, right and left numbers, which you'd call DS for Dexter-Sinister... or who knows what else? (Though you do have only 100 minutes to spend preparing excuses.)
If only there were an art of focusing your uncertainty - of squeezing as much anticipation as possible into whichever outcome will actually happen!
But what could we call an art like that? And what would the rules be like?
August 06, 2007
The Proper Use of Doubt
Once, when I was holding forth upon the Way, I remarked upon how most organized belief systems exist to flee from doubt. A listener replied to me that the Jesuits must be immune from this criticism, because they practice organized doubt: their novices, he said, are told to doubt Christianity; doubt the existence of God; doubt if their calling is real; doubt that they are suitable for perpetual vows of chastity and poverty. And I said: Ah, but they're supposed to overcome these doubts, right? He said: No, they are to doubt that perhaps their doubts may grow and become stronger.
Googling failed to confirm or refute these allegations. (If anyone in the audience can help, I'd be much obliged.) But I find this scenario fascinating, worthy of discussion, regardless of whether it is true or false of Jesuits. If the Jesuits practiced deliberate doubt, as described above, would they therefore be virtuous as rationalists?
I think I have to concede that the Jesuits, in the (possibly hypothetical) scenario above, would not properly be described as "fleeing from doubt". But the (possibly hypothetical) conduct still strikes me as highly suspicious. To a truly virtuous rationalist, doubt should not be scary. The conduct described above sounds to me like a program of desensitization for something very scary, like exposing an arachnophobe to spiders under carefully controlled conditions.
But even so, they are encouraging their novices to doubt - right? Does it matter if their reasons are flawed? Is this not still a worthy deed unto a rationalist?
All curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. But if you obtain an answer, if you satisfy your curiosity, then the glorious mystery will no longer be mysterious.
In the same way, every doubt exists in order to annihilate some particular belief. If a doubt fails to destroy its target, the doubt has died unfulfilled - but that is still a resolution, an ending, albeit a sadder one. A doubt that neither destroys itself nor destroys its target might as well have never existed at all. It is the resolution of doubts, not the mere act of doubting, which drives the ratchet of rationality forward.
Every improvement is a change, but not every change is an improvement. Every rationalist doubts, but not all doubts are rational. Wearing doubts doesn't make you a rationalist any more than wearing a white medical lab coat makes you a doctor.
A rational doubt comes into existence for a specific reason - you have some specific justification to suspect the belief is wrong. This reason in turn, implies an avenue of investigation which will either destroy the targeted belief, or destroy the doubt. This holds even for highly abstract doubts, like "I wonder if there might be a simpler hypothesis which also explains this data." In this case you investigate by trying to think of simpler hypotheses. As this search continues longer and longer without fruit, you will think it less and less likely that the next increment of computation will be the one to succeed. Eventually the cost of searching will exceed the expected benefit, and you'll stop searching. At which point you can no longer claim to be usefully doubting. A doubt that is not investigated might as well not exist. Every doubt exists to destroy itself, one way or the other. An unresolved doubt is a null-op; it does not turn the wheel, neither forward nor back.
If you really believe a religion (not just believe in it), then why would you tell your novices to consider doubts that must die unfulfilled? It would be like telling physics students to painstakingly doubt that the 20th-century revolution might have been a mistake, and that Newtonian mechanics was correct all along. If you don't really doubt something, why would you pretend that you do?
Because we all want to be seen as rational - and doubting is widely believed to be a virtue of a rationalist. But it is not widely understood that you need a particular reason to doubt, or that an unresolved doubt is a null-op. Instead people think it's about modesty, a submissive demeanor, maintaining the tribal status hierarchy - almost exactly the same problem as with humility, on which I have previously written. Making a great public display of doubt to convince yourself that you are a rationalist, will do around as much good as wearing a lab coat.
To avoid professing doubts, remember:
- A rational doubt exists to destroy its target belief, and if it does not destroy its target it dies unfulfilled.
- A rational doubt arises from some specific reason the belief might be wrong.
- An unresolved doubt is a null-op.
- An uninvestigated doubt might as well not exist.
- You should not be proud of mere doubting, although you can justly be proud when you have just finished tearing a cherished belief to shreds.
- Though it may take courage to face your doubts, never forget that to an ideal mind doubt would not be scary in the first place.
August 07, 2007
The Virtue of Narrowness
What is true of one apple may not be true of another apple;
thus more can be said about a single apple than about all the
apples in the world.
-- Twelve Virtues of
Rationality
Within their own professions, people grasp the importance of narrowness; a car mechanic knows the difference between a carburetor and a radiator, and would not think of them both as "car parts". A hunter-gatherer knows the difference between a lion and a panther. A janitor does not wipe the floor with window cleaner, even if the bottles look similar to one who has not mastered the art.
Outside their own professions, people often commit the misstep of trying to broaden a word as widely as possible, to cover as much territory as possible. Is it not more glorious, more wise, more impressive, to talk about all the apples in the world? How much loftier it must be to explain human thought in general, without being distracted by smaller questions, such as how humans invent techniques for solving a Rubik's Cube. Indeed, it scarcely seems necessary to consider specific questions at all; isn't a general theory a worthy enough accomplishment on its own?
It is the way of the curious to lift up one pebble from among a million pebbles on the shore, and see something new about it, something interesting, something different. You call these pebbles "diamonds", and ask what might be special about them - what inner qualities they might have in common, beyond the glitter you first noticed. And then someone else comes along and says: "Why not call this pebble a diamond too? And this one, and this one?" They are enthusiastic, and they mean well. For it seems undemocratic and exclusionary and elitist and unholistic to call some pebbles "diamonds", and others not. It seems... narrow-minded... if you'll pardon the phrase. Hardly open, hardly embracing, hardly communal.
You might think it poetic, to give one word many meanings, and thereby spread shades of connotation all around. But even poets, if they are good poets, must learn to see the world precisely. It is not enough to compare love to a flower. Hot jealous unconsummated love is not the same as the love of a couple married for decades. If you need a flower to symbolize jealous love, you must go into the garden, and look, and make subtle distinctions - find a flower with a heady scent, and a bright color, and thorns. Even if your intent is to shade meanings and cast connotations, you must keep precise track of exactly which meanings you shade and connote.
It is a necessary part of the rationalist's art - or even the poet's art! - to focus narrowly on unusual pebbles which possess some special quality. And look at the details which those pebbles - and those pebbles alone! - share among each other. This is not a sin.
It is perfectly all right for modern evolutionary biologists to explain just the patterns of living creatures, and not the "evolution" of stars or the "evolution" of technology. Alas, some unfortunate souls use the same word "evolution" to cover the naturally selected patterns of replicating life, and the strictly accidental structure of stars, and the intelligently configured structure of technology. And as we all know, if people use the same word, it must all be the same thing. You should automatically generalize anything you think you know about biological evolution to technology. Anyone who tells you otherwise must be a mere pointless pedant. It couldn't possibly be that your abysmal ignorance of modern evolutionary theory is so total that you can't tell the difference between a carburetor and a radiator. That's unthinkable. No, the other guy - you know, the one who's studied the math - is just too dumb to see the connections.
And what could be more virtuous than seeing connections? Surely the wisest of all human beings are the New Age gurus who say "Everything is connected to everything else." If you ever say this aloud, you should pause, so that everyone can absorb the sheer shock of this Deep Wisdom.
There is a trivial mapping between a graph and its complement. A fully connected graph, with an edge between every two vertices, conveys the same amount of information as a graph with no edges at all. The important graphs are the ones where some things are not connected to some other things.
When the unenlightened ones try to be profound, they draw endless verbal comparisons between this topic, and that topic, which is like this, which is like that; until their graph is fully connected and also totally useless. The remedy is specific knowledge and in-depth study. When you understand things in detail, you can see how they are not alike, and start enthusiastically subtracting edges off your graph.
Likewise, the important categories are the ones that do not contain everything in the universe. Good hypotheses can only explain some possible outcomes, and not others.
It was perfectly all right for Isaac Newton to explain just gravity, just the way things fall down - and how planets orbit the Sun, and how the Moon generates the tides - but not the role of money in human society or how the heart pumps blood. Sneering at narrowness is rather reminiscent of ancient Greeks who thought that going out and actually looking at things was manual labor, and manual labor was for slaves.
As Plato put it (in The Republic, Book VII):
"If anyone should throw back his head and learn something by staring at the varied patterns on a ceiling, apparently you would think that he was contemplating with his reason, when he was only staring with his eyes... I cannot but believe that no study makes the soul look on high except that which is concerned with real being and the unseen. Whether he gape and stare upwards, or shut his mouth and stare downwards, if it be things of the senses that he tries to learn something about, I declare he never could learn, for none of these things admit of knowledge: I say his soul is looking down, not up, even if he is floating on his back on land or on sea!"
Many today make a similar mistake, and think that narrow concepts are as lowly and unlofty and unphilosophical as, say, going out and looking at things - an endeavor only suited to the underclass. But rationalists - and also poets - need narrow words to express precise thoughts; they need categories which include only some things, and exclude others. There's nothing wrong with focusing your mind, narrowing your categories, excluding possibilities, and sharpening your propositions. Really, there isn't! If you make your words too broad, you end up with something that isn't true and doesn't even make good poetry.
And DON'T EVEN GET ME STARTED on people who think Wikipedia is an "Artificial Intelligence", the invention of LSD was a "Singularity" or that corporations are "superintelligent"!
August 08, 2007
You Can Face Reality
Owning up to it doesn't make it worse.
Not being open about it doesn't make it go away.
And because it's true, it is what is there to be interacted with.
Anything untrue isn't there to be lived.
People can stand what is true,
for they are already enduring it.
-- Eugene Gendlin
(Hat tip to Stephen Omohundro.)
August 09, 2007
The Apocalypse Bet
A problem with betting on engineered superplagues, physics disasters, nanotechnological warfare, or intelligence explosions of both Friendly and unFriendly type, is that all these events are likely to disrupt settlement of trades (to put it mildly). It's not easy to sell a bet that pays off only if the prediction market ceases to exist.
And yet everyone still wants to know the year, month, and day of the Singularity. Even I want to know, I'm just professionally aware that the knowledge is not available.
This morning, I saw that someone had launched yet another poll on "when the Singularity will occur". Just a raw poll, mind you, not a prediction market. I was thinking of how completely and utterly worthless this poll was, and how a prediction market might be slightly less than completely worthless, when it occurred to me how to structure the bet - bet that "settlement of trades will be disrupted / the resources gambled will become worthless, no later than time T".
Suppose you think that gold will become worthless on April 27th, 2020 at between four and four-thirty in the morning. I, on the other hand, think this event will not occur until 2030. We can sign a contract in which I pay you one ounce of gold per year from 2010 to 2020, and then you pay me two ounces of gold per year from 2020 to 2030. If gold becomes worthless when you say, you will have profited; if gold becomes worthlesss when I say, I will have profited. We can have a prediction market on a generic apocalypse, in which participants who believe in an earlier apocalypse are paid by believers in a later apocalypse, until they pass the date of their prediction, at which time the flow reverses with interest. I don't see any way to distinguish between apocalypses, but we can ask the participants why they were willing to bet, and probably receive a decent answer.
I would be quite interested in seeing what such a market had to say. And if the predicted date was hovering around 2080, I would pick up as much of that free money as I dared.
August 10, 2007
Your Strength as a Rationalist
(The following happened to me in an IRC chatroom, long enough ago that I was still hanging around in IRC chatrooms. Time has fuzzed the memory and my report may be imprecise.)
So there I was, in an IRC chatroom, when someone reports that a friend of his needs medical advice. His friend says that he's been having sudden chest pains, so he called an ambulance, and the ambulance showed up, but the paramedics told him it was nothing, and left, and now the chest pains are getting worse. What should his friend do?
I was confused by this story. I remembered reading about homeless people in New York who would call ambulances just to be taken someplace warm, and how the paramedics always had to take them to the emergency room, even on the 27th iteration. Because if they didn't, the ambulance company could be sued for lots and lots of money. Likewise, emergency rooms are legally obligated to treat anyone, regardless of ability to pay. (And the hospital absorbs the costs, which are enormous, so hospitals are closing their emergency rooms... It makes you wonder what's the point of having economists if we're just going to ignore them.) So I didn't quite understand how the described events could have happened. Anyone reporting sudden chest pains should have been hauled off by an ambulance instantly.
And this is where I fell down as a rationalist. I remembered several occasions where my doctor would completely fail to panic at the report of symptoms that seemed, to me, very alarming. And the Medical Establishment was always right. Every single time. I had chest pains myself, at one point, and the doctor patiently explained to me that I was describing chest muscle pain, not a heart attack. So I said into the IRC channel, "Well, if the paramedics told your friend it was nothing, it must really be nothing - they'd have hauled him off if there was the tiniest chance of serious trouble."
Thus I managed to explain the story within my existing model, though the fit still felt a little forced...
Later on, the fellow comes back into the IRC chatroom and says his friend made the whole thing up. Evidently this was not one of his more reliable friends.
I should have realized, perhaps, that an unknown acquaintance of an acquaintance in an IRC channel might be less reliable than a published journal article. Alas, belief is easier than disbelief; we believe instinctively, but disbelief requires a conscious effort.
So instead, by dint of mighty straining, I forced my model of reality to explain an anomaly that never actually happened. And I knew how embarrassing this was. I knew that the usefulness of a model is not what it can explain, but what it can't. A hypothesis that forbids nothing, permits everything, and thereby fails to constrain anticipation.
Your strength as a rationalist is your ability to be more confused by fiction than by reality. If you are equally good at explaining any outcome, you have zero knowledge.
We are all weak, from time to time; the sad part is that I could have been stronger. I had all the information I needed to arrive at the correct answer, I even noticed the problem, and then I ignored it. My feeling of confusion was a Clue, and I threw my Clue away.
I should have paid more attention to that sensation of still feels a little forced. It's one of the most important feelings a truthseeker can have, a part of your strength as a rationalist. It is a design flaw in human cognition that this sensation manifests as a quiet strain in the back of your mind, instead of a wailing alarm siren and a glowing neon sign reading "EITHER YOUR MODEL IS FALSE OR THIS STORY IS WRONG."
August 11, 2007
I Defy the Data!
One of the great weaknesses of Science is this mistaken idea that if an experiment contradicts the dominant theory, we should throw out the theory instead of the experiment.
Experiments can go awry. They can contain design flaws. They can be deliberately corrupted. They can be unconsciously corrupted. They can be selectively reported. Most of all, 1 time in 20 they can be "statistically significant" by sheer coincidence, and there are a lot of experiments out there.
Unfortunately, Science has this notion that you can never go
against an honestly obtained experimental result. So, when
someone obtains an experimental result that contradicts the
standard model, researchers are faced with a dilemma for resolving
their cognitive dissonance: they either have to
immediately throw away the standard model, or else
attack the experiment - accuse the researchers of
dishonesty, or flawed design, or conflict of interest...
Someone once presented me with a new study on the effects of intercessory prayer (that is, people praying for patients who are not told about the prayer), which showed 50% of the prayed-for patients achieving success at in-vitro fertilization, versus 25% of the control group. I liked this claim. It had a nice large effect size. Claims of blatant impossible effects are much more pleasant to deal with than claims of small impossible effects that are "statistically significant".
So I cheerfully said: "I defy the data."
My original phrasing was actually "I deny the data". Nonetheless I said it outright, without apology, and with deliberate insolence. I am keeping my theory; your experiment is wrong.
If an experimental result contradicts the Standard Model, this is an important fact. It needs to be openly acknowledged. An experiment that makes traditionalists want to discard the data - or even an experiment that makes traditionalists very skeptical of the data - should be a high priority for replication. An experiment worth defying should command attention!
But it is not socially acceptable to say, "The hell with your experimental falsification, I'm keeping my theory." So the data has to be defied covertly - by character assassination of the researchers, by sly innuendos, by dire hints of controversy. The data has to be dismissed, excused away, swept under a rug, silently into the dark, because you can't admit you're defying the data. This is not a good way of focusing attention on an anomalous result. This is not a good way to ensure funding for replication attempts.
It would be much better if science had a standard procedure for saying, "I defy the data!" It would be clearly understood that this was a bold act, and someone else in the audience might stand up and say, "Wait a minute, is that data really worth defying?" If a major figure in the field said "I defy the data!", this would be sufficient justification on grant proposals for why the result urgently needed replication. Scientists could say, "I'm holding my breath, waiting for replication," rather than having to take sides immediately in the character-assassination controversy.
Maybe you could even get the media to report that the experiment has been "published but defied". Then the replication, or failure to replicate, would be news. The replicators could get their names in the newspaper, and the negative result could be published in a major journal. If you want replications done, you'll have to offer some incentive.
I would also suggest that when an experiment is defied, the replication must pre-declare a minimum effect size, and attain significance of p<0.01. In extreme cases where claims have been made and shot down before, p<0.001.
Oh, and the prayer study? Soon enough we heard that it had been retracted and was probably fraudulent. But I didn't say fraud. I didn't speculate on how the results might have been obtained. That would have been dismissive. I just stuck my neck out, and nakedly, boldly, without excuses, defied the data.
Addendum: I should have spelled this out explicitly: You can defy the data on one experiment. You can't defy the data on multiple experiments. At that point you either have to relinquish the theory or dismiss the data - point to a design flaw, or refer to an even larger body of experiments that failed to replicate the result, or accuse the researchers of a deliberate hoax, et cetera. But you should not turn around and argue that the theory and the experiment are actually compatible. Why didn't you think of that before you defied the data? Defying the data admits that the data is not compatible with your theory; it sticks your neck way out, so your head can be easily chopped off.
August 12, 2007
Absence of Evidence Is Evidence of Absence
From Robyn Dawes's Rational Choice in an Uncertain World:
Post-hoc fitting of evidence to hypothesis was involved in a most grievous chapter in United States history: the internment of Japanese-Americans at the beginning of the Second World War. When California governor Earl Warren testified before a congressional hearing in San Francisco on February 21, 1942, a questioner pointed out that there had been no sabotage or any other type of espionage by the Japanese-Americans up to that time. Warren responded, "I take the view that this lack [of subversive activity] is the most ominous sign in our whole situation. It convinces me more than perhaps any other factor that the sabotage we are to get, the Fifth Column activities are to get, are timed just like Pearl Harbor was timed... I believe we are just being lulled into a false sense of security."
Consider Warren's argument from a Bayesian perspective. When we see evidence, hypotheses that assigned a higher likelihood to that evidence, gain probability at the expense of hypotheses that assigned a lower likelihood to the evidence. This is a phenomenon of relative likelihoods and relative probabilities. You can assign a high likelihood to the evidence and still lose probability mass to some other hypothesis, if that other hypothesis assigns a likelihood that is even higher.
Warren seems to be arguing that, given that we see no sabotage, this confirms that a Fifth Column exists. You could argue that a Fifth Column might delay its sabotage. But the likelihood is still higher that the absence of a Fifth Column would perform an absence of sabotage.
Let E stand for the observation of sabotage, H1 for the hypothesis of a Japanese-American Fifth Column, and H2 for the hypothesis that no Fifth Column exists. Whatever the likelihood that a Fifth Column would do no sabotage, the probability P(E|H1), it cannot be as large as the likelihood that no Fifth Column does no sabotage, the probability P(E|H2). So observing a lack of sabotage increases the probability that no Fifth Column exists.
A lack of sabotage doesn't prove that no Fifth Column exists. Absence of proof is not proof of absence. In logic, A->B, "A implies B", is not equivalent to ~A->~B, "not-A implies not-B".
But in probability theory, absence of evidence is always evidence of absence. If E is a binary event and P(H|E) > P(H), "seeing E increases the probability of H"; then P(H|~E) < P(H), "failure to observe E decreases the probability of H". P(H) is a weighted mix of P(H|E) and P(H|~E), and necessarily lies between the two. If any of this sounds at all confusing, see An Intuitive Explanation of Bayesian Reasoning.
Under the vast majority of real-life circumstances, a cause may not reliably produce signs of itself, but the absence of the cause is even less likely to produce the signs. The absence of an observation may be strong evidence of absence or very weak evidence of absence, depending on how likely the cause is to produce the observation. The absence of an observation that is only weakly permitted (even if the alternative hypothesis does not allow it at all), is very weak evidence of absence (though it is evidence nonetheless). This is the fallacy of "gaps in the fossil record" - fossils form only rarely; it is futile to trumpet the absence of a weakly permitted observation when many strong positive observations have already been recorded. But if there are no positive observations at all, it is time to worry; hence the Fermi Paradox.
Your strength as a
rationalist is your ability to be more confused by fiction than
by reality; if you are equally good at explaining any outcome you
have zero knowledge. The strength of a model is not what it
can explain, but what it can't, for only
prohibitions constrain anticipation.
If you don't notice when your model makes the evidence unlikely,
you might as well have no model, and also you might as well have no
evidence; no brain and no eyes.
August 13, 2007
Conservation of Expected Evidence
Followup to: Absence of Evidence Is
Evidence of Absence.
Friedrich Spee von Langenfeld, a priest who heard the confessions of condemned witches, wrote in 1631 the Cautio Criminalis ('prudence in criminal cases') in which he bitingly described the decision tree for condemning accused witches: If the witch had led an evil and improper life, she was guilty; if she had led a good and proper life, this too was a proof, for witches dissemble and try to appear especially virtuous. After the woman was put in prison: if she was afraid, this proved her guilt; if she was not afraid, this proved her guilt, for witches characteristically pretend innocence and wear a bold front. Or on hearing of a denunciation of witchcraft against her, she might seek flight or remain; if she ran, that proved her guilt; if she remained, the devil had detained her so she could not get away.
Spee acted as confessor to many witches; he was thus in a position to observe every branch of the accusation tree, that no matter what the accused witch said or did, it was held a proof against her. In any individual case, you would only hear one branch of the dilemma. It is for this reason that scientists write down their experimental predictions in advance.
But you can't have it both ways - as a matter of probability theory, not mere fairness. The rule that "absence of evidence is evidence of absence" is a special case of a more general law, which I would name Conservation of Expected Evidence: The expectation of the posterior probability, after viewing the evidence, must equal the prior probability.
P(H) = P(H)
P(H) = P(H,E) + P(H,~E)
P(H) = P(H|E)*P(E) + P(H|~E)*P(~E)
Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence.
If you expect a strong probability of seeing weak evidence in one direction, it must be balanced by a weak expectation of seeing strong evidence in the other direction. If you're very confident in your theory, and therefore anticipate seeing an outcome that matches your hypothesis, this can only provide a very small increment to your belief (it is already close to 1); but the unexpected failure of your prediction would (and must) deal your confidence a huge blow. On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence - before you've actually seen it - should not shift your prior beliefs. (Again, if this is not intuitively obvious, see An Intuitive Explanation of Bayesian Reasoning.)
So if you claim that "no sabotage" is evidence for the existence of a Japanese-American Fifth Column, you must conversely hold that seeing sabotage would argue against a Fifth Column. If you claim that "a good and proper life" is evidence that a woman is a witch, then an evil and improper life must be evidence that she is not a witch. If you argue that God, to test humanity's faith, refuses to reveal His existence, then the miracles described in the Bible must argue against the existence of God.
Doesn't quite sound right, does it? Pay attention to that feeling of this seems a little forced, that quiet strain in the back of your mind. It's important.
For a true Bayesian, it is impossible to seek evidence that confirms a theory. There is no possible plan you can devise, no clever strategy, no cunning device, by which you can legitimately expect your confidence in a fixed proposition to be higher (on average) than before. You can only ever seek evidence to test a theory, not to confirm it.
This realization can take quite a load off your mind. You need not worry about how to interpret every possible experimental result to confirm your theory. You needn't bother planning how to make any given iota of evidence confirm your theory, because you know that for every expectation of evidence, there is an equal and oppositive expectation of counterevidence. If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree. It is a zero-sum game. No matter how you connive, no matter how you argue, no matter how you strategize, you can't possibly expect the resulting game plan to shift your beliefs (on average) in a particular direction.
You might as well sit back and relax while you wait for the evidence to come in.
...human psychology is so screwed up.
August 14, 2007
Update Yourself Incrementally
Politics is the mind-killer. Debate is war, arguments are soldiers. There is the temptation to search for ways to interpret every possible experimental result to confirm your theory, like securing a citadel against every possible line of attack. This you cannot do. It is mathematically impossible. For every expectation of evidence, there is an equal and opposite expectation of counterevidence.
But it's okay if your cherished belief isn't perfectly defended. If the hypothesis is that the coin comes up heads 95% of the time, then one time in twenty you will see what looks like contrary evidence. This is okay. It's normal. It's even expected, so long as you've got nineteen supporting observations for every contrary one. A probabilistic model can take a hit or two, and still survive, so long as the hits don't keep on coming in.
Yet it is widely believed, especially in the court of public opinion, that a true theory can have no failures and a false theory no successes.
You find people holding up a single piece of what they conceive to be evidence, and claiming that their theory can 'explain' it, as though this were all the support that any theory needed. Apparently a false theory can have no supporting evidence; it is impossible for a false theory to fit even a single event. Thus, a single piece of confirming evidence is all that any theory needs.
It is only slightly less foolish to hold up a single piece of probabilistic counterevidence as disproof, as though it were impossible for a correct theory to have even a slight argument against it. But this is how humans have argued for ages and ages, trying to defeat all enemy arguments, while denying the enemy even a single shred of support. People want their debates to be one-sided; they are accustomed to a world in which their preferred theories have not one iota of antisupport. Thus, allowing a single item of probabilistic counterevidence would be the end of the world.
I just know someone in the audience out there is going to say, "But you can't concede even a single point if you want to win debates in the real world! If you concede that any counterarguments exist, the Enemy will harp on them over and over - you can't let the Enemy do that! You'll lose! What could be more viscerally terrifying than that?"
Whatever. Rationality is not for winning debates, it is for deciding which side to join. If you've already decided which side to argue for, the work of rationality is done within you, whether well or poorly. But how can you, yourself, decide which side to argue? If choosing the wrong side is viscerally terrifying, even just a little viscerally terrifying, you'd best integrate all the evidence.
Rationality is not a walk, but a dance. On each step in that dance your foot should come down in exactly the correct spot, neither to the left nor to the right. Shifting belief upward with each iota of confirming evidence. Shifting belief downward with each iota of contrary evidence. Yes, down. Even with a correct model, if it is not an exact model, you will sometimes need to revise your belief down.
If an iota or two of evidence happens to countersupport your belief, that's okay. It happens, sometimes, with probabilistic evidence for non-exact theories. (If an exact theory fails, you are in trouble!) Just shift your belief downward a little - the probability, the odds ratio, or even a nonverbal weight of credence in your mind. Just shift downward a little, and wait for more evidence. If the theory is true, supporting evidence will come in shortly, and the probability will climb again. If the theory is false, you don't really want it anyway.
The problem with using black-and-white, binary, qualitative reasoning is that any single observation either destroys the theory or it does not. When not even a single contrary observation is allowed, it creates cognitive dissonance and has to be argued away. And this rules out incremental progress; it rules out correct integration of all the evidence. Reasoning probabilistically, we realize that on average, a correct theory will generate a greater weight of support than countersupport. And so you can, without fear, say to yourself: "This is gently contrary evidence, I will shift my belief downward". Yes, down. It does not destroy your cherished theory. That is qualitative reasoning; think quantitatively.
For every expectation of evidence, there is an equal and opposite expectation of counterevidence. On every occasion, you must, on average, anticipate revising your beliefs downward as much as you anticipate revising them upward. If you think you already know what evidence will come in, then you must already be fairly sure of your theory - probability close to 1 - which doesn't leave much room for the probability to go further upward. And however unlikely it seems that you will encounter disconfirming evidence, the resulting downward shift must be large enough to precisely balance the anticipated gain on the other side. The weighted mean of your expected posterior probability must equal your prior probability.
How silly is it, then, to be terrified of revising your probability downward, if you're bothering to investigate a matter at all? On average, you must anticipate as much downward shift as upward shift from every individual observation.
It may perhaps happen that an iota of antisupport comes in again, and again and again, while new support is slow to trickle in. You may find your belief drifting downward and further downward. Until, finally, you realize from which quarter the winds of evidence are blowing against you. In that moment of realization, there is no point in constructing excuses. In that moment of realization, you have already relinquished your cherished belief. Yay! Time to celebrate! Pop a champagne bottle or send out for pizza! You can't become stronger by keeping the beliefs you started with, after all.
August 15, 2007
One Argument Against An Army
Followup to: Update Yourself Incrementally
Yesterday I talked about a style of reasoning in which not a single contrary argument is allowed, with the result that every non-supporting observation has to be argued away. Today I suggest that when people encounter a contrary argument, they prevent themselves from downshifting their confidence by rehearsing already-known support.
Suppose the country of Freedonia is debating whether its neighbor, Sylvania, is responsible for a recent rash of meteor strikes on its cities. There are several pieces of evidence suggesting this: the meteors struck cities close to the Sylvanian border; there was unusual activity in the Sylvanian stock markets before the strikes; and the Sylvanian ambassador Trentino was heard muttering about "heavenly vengeance".
Someone comes to you and says: "I don't think Sylvania is
responsible for the meteor strikes. They have trade with us
of billions of dinars annually." "Well," you reply, "the
meteors struck cities close to Sylvania, there was suspicious
activity in their stock market, and their ambassador spoke of
heavenly vengeance afterward." Since these three arguments
outweigh the first, you keep your belief that Sylvania is
responsible - you believe rather than disbelieve, qualitatively.
Clearly, the balance of evidence weighs against Sylvania.
Then another comes to you and says: "I don't think Sylvania is responsible for the meteor strikes. Directing an asteroid strike is really hard. Sylvania doesn't even have a space program." You reply, "But the meteors struck cities close to Sylvania, and their investors knew it, and the ambassador came right out and admitted it!" Again, these three arguments outweigh the first (by three arguments against one argument), so you keep your belief that Sylvania is responsible.
Indeed, your convictions are strengthened. On two separate occasions now, you have evaluated the balance of evidence, and both times the balance was tilted against Sylvania by a ratio of 3-to-1.
You encounter further arguments by the pro-Sylvania traitors - again, and again, and a hundred times again - but each time the new argument is handily defeated by 3-to-1. And on every occasion, you feel yourself becoming more confident that Sylvania was indeed responsible, shifting your prior according to the felt balance of evidence.
The problem, of course, is that by rehearsing arguments you already knew, you are double-counting the evidence This would be a grave sin even if you double-counted all the evidence. (Imagine a scientist who does an experiment with 50 subjects and fails to obtain statistically significant results, so he counts all the data twice.)
But to selectively double-count only some evidence is sheer farce. I remember seeing a cartoon as a child, where a villain was dividing up loot using the following algorithm: "One for you, one for me. One for you, one-two for me. One for you, one-two-three for me."
As I emphasized yesterday, even if a cherished belief is true, a rationalist may sometimes need to downshift the probability while integrating all the evidence. Yes, the balance of support may still favor your cherished belief. But you still have to shift the probability down - yes, down - from whatever it was before you heard the contrary evidence. It does no good to rehearse supporting arguments, because you have already taken those into account.
And yet it does appear to me that when people are confronted by a new counterargument, they search for a justification not to downshift their confidence, and of course they find supporting arguments they already know. I have to keep constant vigilance not to do this myself! It feels as natural as parrying a sword-strike with a handy shield.
With the right kind of wrong reasoning, a handful of support - or even a single argument - can stand off an army of contradictions.
August 16, 2007
Hindsight bias
Hindsight bias is when people who know the answer vastly overestimate its predictability or obviousness, compared to the estimates of subjects who must guess without advance knowledge. Hindsight bias is sometimes called the I-knew-it-all-along effect.
Fischhoff and Beyth (1975) presented students with historical accounts of unfamiliar incidents, such as a conflict between the Gurkhas and the British in 1814. Given the account as background knowledge, five groups of students were asked what they would have predicted as the probability for each of four outcomes: British victory, Gurkha victory, stalemate with a peace settlement, or stalemate with no peace settlement. Four experimental groups were respectively told that these four outcomes were the historical outcome. The fifth, control group was not told any historical outcome. In every case, a group told an outcome assigned substantially higher probability to that outcome, than did any other group or the control group.
Hindsight bias matters in legal cases, where a judge or jury must determine whether a defendant was legally negligent in failing to foresee a hazard (Sanchiro 2003). In an experiment based on an actual legal case, Kamin and Rachlinski (1995) asked two groups to estimate the probability of flood damage caused by blockage of a city-owned drawbridge. The control group was told only the background information known to the city when it decided not to hire a bridge watcher. The experimental group was given this information, plus the fact that a flood had actually occurred. Instructions stated the city was negligent if the foreseeable probability of flooding was greater than 10%. 76% of the control group concluded the flood was so unlikely that no precautions were necessary; 57% of the experimental group concluded the flood was so likely that failure to take precautions was legally negligent. A third experimental group was told the outcome andalso explicitly instructed to avoid hindsight bias, which made no difference: 56% concluded the city was legally negligent.
Viewing history through the lens of hindsight, we vastly underestimate the cost of effective safety precautions. In 1986, the Challenger exploded for reasons traced to an O-ring losing flexibility at low temperature. There were warning signs of a problem with the O-rings. But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning sign which seemed as severe as the O-ring problem, without benefit of hindsight. It could have been done, but it would have required a general policy much more expensive than just fixing the O-Rings.
Shortly after September 11th 2001, I thought to myself, and now someone will turn up minor intelligence warnings of something-or-other, and then the hindsight will begin. Yes, I'm sure they had some minor warnings of an al Qaeda plot, but they probably also had minor warnings of mafia activity, nuclear material for sale, and an invasion from Mars.
Because we don't see the cost of a general policy, we learn overly specific lessons. After September 11th, the FAA prohibited box-cutters on airplanes - as if the problem had been the failure to take this particular "obvious" precaution. We don't learn the general lesson: the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.
The test of a model is how much probability it assigns to the observed outcome. Hindsight bias systematically distorts this test; we think our model assigned much more probability than it actually did. Instructing the jury doesn't help. You have to write down your predictions in advance. Or as Fischhoff (1982) put it:
When we attempt to understand past events, we implicitly test the hypotheses or rules we use both to interpret and to anticipate the world around us. If, in hindsight, we systematically underestimate the surprises that the past held and holds for us, we are subjecting those hypotheses to inordinately weak tests and, presumably, finding little reason to change them.
Fischhoff, B. 1982. For those condemned to study the past: Heuristics and biases in hindsight. In Kahneman et. al. 1982: 332–351.
Fischhoff, B., and Beyth, R. 1975. I knew it would happen: Remembered probabilities of once-future things. Organizational Behavior and Human Performance, 13: 1-16.
Kamin, K. and Rachlinski, J. 1995. Ex Post ≠Ex Ante: Determining Liability in Hindsight. Law and Human Behavior, 19(1): 89-104.
Sanchiro, C. 2003. Finding Error. Mich. St. L. Rev. 1189.
August 17, 2007
Hindsight Devalues Science
This excerpt from Meyers's Exploring Social Psychology is worth reading in entirety. Cullen Murphy, editor of The Atlantic, said that the social sciences turn up "no ideas or conclusions that can't be found in [any] encyclopedia of quotations... Day after day social scientists go out into the world. Day after day they discover that people's behavior is pretty much what you'd expect."
Of course, the "expectation" is all hindsight. (Hindsight bias: Subjects who know the actual answer to a question assign much higher probabilities they "would have" guessed for that answer, compared to subjects who must guess without knowing the answer.)
The historian Arthur Schlesinger, Jr. dismissed scientific studies of WWII soldiers' experiences as "ponderous demonstrations" of common sense. For example:
- Better educated soldiers suffered more adjustment problems than less educated soldiers. (Intellectuals were less prepared for battle stresses than street-smart people.)
- Southern soldiers coped better with the hot South Sea Island climate than Northern soldiers. (Southerners are more accustomed to hot weather.)
- White privates were more eager to be promoted to noncommissioned officers than Black privates. (Years of oppression take a toll on achievement motivation.)
- Southern Blacks preferred Southern to Northern White officers (because Southern officers were more experienced and skilled in interacting with Blacks).
- As long as the fighting continued, soldiers were more eager to return home than after the war ended. (During the fighting, soldiers knew they were in mortal danger.)
How many of these findings do you think you could have predicted in advance? 3 out of 5? 4 out of 5? Are there any cases where you would have predicted the opposite - where your model takes a hit? Take a moment to think before continuing...
In this demonstration (from Paul Lazarsfeld by way of Meyers), all of the findings above are the opposite of what was actually found. How many times did you think your model took a hit? How many times did you admit you would have been wrong? That's how good your model really was. The measure of your strength as a rationalist is your ability to be more confused by fiction than by reality.
Unless, of course, I reversed the results again. What do you think?
Do your thought processes at this point, where you really don't know the answer, feel different from the thought processes you used to rationalize either side of the "known" answer?
Daphna Baratz exposed college students to pairs of supposed findings, one true ("In prosperous times people spend a larger portion of their income than during a recession") and one the truth's opposite. In both sides of the pair, students rated the supposed finding as what they "would have predicted". Perfectly standard hindsight bias.
Which leads people to think they have no need for science, because they "could have predicted" that.
(Just as you would expect, right?)
Hindsight will lead us to systematically undervalue the surprisingness of scientific findings, especially the discoveries we understand - the ones that seem real to us, the ones we can retrofit into our models of the world. If you understand neurology or physics and read news in that topic, then you probably underestimate the surprisingness of findings in those fields too. This unfairly devalues the contribution of the researchers; and worse, will prevent you from noticing when you are seeing evidence that doesn't fit what you really would have expected.
We need to make a conscious effort to be shocked enough.
August 19, 2007
Scientific Evidence, Legal Evidence, Rational Evidence
Suppose that your good friend, the police commissioner, tells you in strictest confidence that the crime kingpin of your city is Wulky Wilkinsen. As a rationalist, are you licensed to believe this statement? Put it this way: if you go ahead and mess around with Wulky's teenage daughter, I'd call you foolhardy. Since it is prudent to act as if Wulky has a substantially higher-than-default probability of being a crime boss, the police commissioner's statement must have been strong Bayesian evidence.
Our legal system will not imprison Wulky on the basis of the police commissioner's statement. It is not admissible as legal evidence. Maybe if you locked up every person accused of being a crime boss by a police commissioner, you'd initially catch a lot of crime bosses, plus some people that a police commissioner didn't like. Power tends to corrupt: over time, you'd catch fewer and fewer real crime bosses (who would go to greater lengths to ensure anonymity) and more and more innocent victims (unrestrained power attracts corruption like honey attracts flies).
This does not mean that the police commissioner's statement is not rational evidence. It still has a lopsided likelihood ratio, and you'd still be a fool to mess with Wulky's teenager daughter. But on a social level, in pursuit of a social goal, we deliberately define "legal evidence" to include only particular kinds of evidence, such as the police commissioner's own observations on the night of April 4th. All legal evidence should ideally be rational evidence, but not the other way around. We impose special, strong, additional standards before we anoint rational evidence as "legal evidence".
As I write this sentence at 8:33pm, Pacific time, on August 18th 2007, I am wearing white socks. As a rationalist, are you licensed to believe the previous statement? Yes. Could I testify to it in court? Yes. Is it a scientific statement? No, because there is no experiment you can perform yourself to verify it. Science is made up of generalizations which apply to many particular instances, so that you can run new real-world experiments which test the generalization, and thereby verify for yourself that the generalization is true, without having to trust anyone's authority. Science is the publicly reproducible knowledge of humankind.
Like a court system, science as a social process is made up of fallible humans. We want a protected pool of beliefs that are especially reliable. And we want social rules that encourage the generation of such knowledge. So we impose special, strong, additional standards before we canonize rational knowledge as "scientific knowledge", adding it to the protected belief pool.
Is a rationalist licensed to believe in the historical existence of Alexander the Great? Yes. We have a rough picture of ancient Greece, untrustworthy but better than maximum entropy. But we are dependent on authorities such as Plutarch; we cannot discard Plutarch and verify everything for ourselves. Historical knowledge is not scientific knowledge.
Is a rationalist licensed to believe that the Sun will rise on September 18th, 2007? Yes - not with absolute certainty, but that's the way to bet. (Pedants: interpret this as the Earth's rotation and orbit remaining roughly constant relative to the Sun.) Is this statement, as I write this essay on August 18th 2007, a scientific belief?
It may seem perverse to deny the adjective "scientific" to statements like "The Sun will rise on September 18th, 2007." If Science could not make predictions about future events - events which have not yet happened - then it would be useless; it could make no prediction in advance of experiment. The prediction that the Sun will rise is, definitely, an extrapolation from scientific generalizations. It is based upon models of the Solar System which you could test for yourself by experiment.
But imagine that you're constructing an experiment to verify prediction #27, in a new context, of an accepted theory Q. You may not have any concrete reason to suspect the belief is wrong; you just want to test it in a new context. It seems dangerous to say, before running the experiment, that there is a "scientific belief" about the result. There is a "conventional prediction" or "theory Q's prediction". But if you already know the "scientific belief" about the result, why bother to run the experiment?
You begin to see, I hope, why I identify Science with generalizations, rather than the history of any one experiment. A historical event happens once; generalizations apply over many events. History is not reproducible; scientific generalizations are.
Is my definition of "scientific knowledge" true? That is not a well-formed question. The special standards we impose upon science are pragmatic choices. Nowhere upon the stars or the mountains is it written that p<0.05 shall be the standard for scientific publication. Many now argue that 0.05 is too weak, and that it would be useful to lower it to 0.01 or 0.001.
Perhaps future generations, acting on the theory that science is the public, reproducible knowledge of humankind, will only label as "scientific" papers published in an open-access journal. If you charge for access to the knowledge, is it part of the knowledge of humankind? Can we trust a result if people must pay to criticize it? Is it really science?
The question "Is it really science?" is ill-formed. Is a $20,000/year closed-access journal really Bayesian evidence? As with the police commissioner's private assurance that Wulky is the kingpin, I think we must answer "Yes." But should the closed-access journal be further canonized as "science"? Should we allow it into the special, protected belief pool? For myself, I think science would be better served by the dictum that only open knowledge counts as the public, reproducible knowledge pool of humankind.
August 20, 2007
Is Molecular Nanotechnology "Scientific"?
Prerequisite / Read this first: Scientific Evidence, Legal Evidence, Rational Evidence
Consider the statement "It is physically possible to construct diamondoid nanomachines which repair biological cells." Some people will tell you that molecular nanotechnology is "pseudoscience" because it has not been verified by experiment - no one has ever seen a nanofactory, so how can believing in their possibility be scientific?
Drexler, I think, would reply that his extrapolations of diamondoid nanomachines are based on standard physics, which is to say, scientific generalizations. Therefore, if you say that nanomachines cannot work, you must be inventing new physics. Or to put it more sharply: If you say that a simulation of a molecular gear is inaccurate, if you claim that atoms thus configured would behave differently from depicted, then either you know a flaw in the simulation algorithm or you're inventing your own laws of physics.
My own sympathies, I confess, are with Drexler. And not just because you could apply the same argument of "I've never seen it, therefore it can't happen" to my own field of Artificial Intelligence.
What about the Wright Brothers' attempt to build a non-biological heavier-than-air powered flying machine? Was that "pseudoscience"? No one had ever seen one before. Wasn't "all flying machines crash" a generalization true over all previous observations? Wouldn't it be scientific to extend this generalization to predict future experiments?
"Flying machines crash" is a qualitative, imprecise, verbal, surface-level generalization. If you have a quantitative theory of aerodynamics which can calculate precisely how previous flying machines crashed, that same theory of aerodynamics would predict the Wright Flyer will fly (and how high, at what speed). Deep quantitative generalizations take strict precedence over verbal surface generalizations. Only deep laws possess the absolute universality and stability of physics. Perhaps there are no new quarks under the Sun, but on higher levels of organization, new things happen all the time.
"No one has ever seen a non-biological nanomachine" is a verbalish surface-level generalization, which can hardly overrule the precise physical models used to simulate a molecular gear.
And yet... I still would not say that "It's possible to construct a nanofactory" is a scientific belief. This belief will not become scientific until someone actually constructs a nanofactory. Just because something is the best extrapolation from present generalizations, doesn't make it true. We have not done an atom-by-atom calculation for the synthesis and behavior of an entire nanofactory; the argument for nanofactories is based on qualitative, abstract reasoning. Such reasoning, even from the best available current theories, sometimes goes wrong. Not always, but sometimes.
The argument for "it's possible to construct a nanofactory" is based on the protected belief pool of science. But it does not, itself, meet the special strong standards required to ceremonially add a belief to the protected belief pool.
Yet if, on a whim, you decide to make a strong positive assertion that nanomachines are impossible, you are being irrational. You are even being "unscientific". An ungrounded whimsical assertion that tomorrow the Sun will not rise is "unscientific", because you have needlessly contradicted the best extrapolation from current scientific knowledge.
In the nanotechnology debate, we see once again the severe folly of thinking that everything which is not science is pseudoscience - as if Nature is prohibited from containing any truths except those already verified by surface observations of scientific experiments. It is a fallacy of the excluded middle.
Of course you could try to criticize the feasibility of diamondoid nanotechnology from within the known laws of physics. That could be argued. It wouldn't have the just plain silly quality of "Nanotech is pseudoscience because no one's ever seen a nanotech." Drexler used qualitative, abstract reasoning from known science; perhaps his argument has a hidden flaw according to known science.
For now, "diamondoid nanosystems are possible" is merely a best guess. It is merely based on qualitative, abstract, approximate, potentially fallible reasoning from beliefs already in the protected belief pool of science. Such a guess is not reliable enough itself to be added to the protected belief pool. It is merely rational.
August 20, 2007
Fake Explanations
Once upon a time, there was an instructor who taught physics students. One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator. The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm. And the instructor said, Why do you think this happens? Some students guessed convection of air currents, and others guessed strange metals in the plate. They devised many creative explanations, none stooping so low as to say "I don't know" or "This seems impossible."
And the answer was that before the students entered the room, the instructor turned the plate around.
Consider the student who frantically stammers, "Eh, maybe because of the heat conduction and so?" I ask: is this answer a proper belief? The words are easily enough professed - said in a loud, emphatic voice. But do the words actually control anticipation?
Ponder that innocent little phrase, "because of", which comes before "heat conduction". Ponder some of the other things we could put after it. We could say, for example, "Because of phlogiston", or "Because of magic."
"Magic!" you cry. "That's not a scientific explanation!" Indeed, the phrases "because of heat conduction" and "because of magic" are readily recognized as belonging to different literary genres. "Heat conduction" is something that Spock might say on Star Trek, whereas "magic" would be said by Giles in Buffy the Vampire Slayer.
However, as Bayesians, we take no notice of literary genres. For us, the substance of a model is the control it exerts on anticipation. If you say "heat conduction", what experience does that lead you to anticipate? Under normal circumstances, it leads you to anticipate that, if you put your hand on the side of the plate near the radiator, that side will feel warmer than the opposite side. If "because of heat conduction" can also explain the radiator-adjacent side feeling cooler, then it can explain pretty much anything.
And as we all know by this point (I do hope), if you are equally good at explaining any outcome, you have zero knowledge. "Because of heat conduction", used in such fashion, is a disguised hypothesis of maximum entropy. It is anticipation-isomorphic to saying "magic". It feels like an explanation, but it's not.
Supposed that instead of guessing, we measured the heat of the metal plate at various points and various times. Seeing a metal plate next to the radiator, we would ordinarily expect the point temperatures to satisfy an equilibrium of the diffusion equation with respect to the boundary conditions imposed by the environment. You might not know the exact temperature of the first point measured, but after measuring the first points - I'm not physicist enough to know how many would be required - you could take an excellent guess at the rest.
A true master of the art of using numbers to constrain the anticipation of material phenomena - a "physicist" - would take some measurements and say, "This plate was in equilibrium with the environment two and a half minutes ago, turned around, and is now approaching equilibrium again."
The deeper error of the students is not simply that they failed to constrain anticipation. Their deeper error is that they thought they were doing physics. They said the phrase "because of", followed by the sort of words Spock might say on Star Trek, and thought they thereby entered the magisterium of science.
Not so. They simply moved their magic from one literary genre to another.
August 21, 2007
Guessing the Teacher's Password
Followup to: Fake Explanations
When I was young, I read popular physics books such as Richard Feynman's QED: The Strange Theory of Light and Matter. I knew that light was waves, sound was waves, matter was waves. I took pride in my scientific literacy, when I was nine years old.
When I was older, and I began to read the Feynman Lectures on Physics, I ran across a gem called "the wave equation". I could follow the equation's derivation, but, looking back, I couldn't see its truth at a glance. So I thought about the wave equation for three days, on and off, until I saw that it was embarrassingly obvious. And when I finally understood, I realized that the whole time I had accepted the honest assurance of physicists that light was waves, sound was waves, matter was waves, I had not had the vaguest idea of what the word "wave" meant to a physicist.
There is an instinctive tendency to think that if a physicist says "light is made of waves", and the teacher says "What is light made of?", and the student says "Waves!", the student has made a true statement. That's only fair, right? We accept "waves" as a correct answer from the physicist; wouldn't it be unfair to reject it from the student? Surely, the answer "Waves!" is either true or false, right?
Which is one more bad habit to unlearn from school. Words do not have intrinsic definitions. If I hear the syllables "bea-ver" and think of a large rodent, that is a fact about my own state of mind, not a fact about the syllables "bea-ver". The sequence of syllables "made of waves" (or "because of heat conduction") is not a hypothesis, it is a pattern of vibrations traveling through the air, or ink on paper. It can associate to a hypothesis in someone's mind, but it is not, of itself, right or wrong. But in school, the teacher hands you a gold star for saying "made of waves", which must be the correct answer because the teacher heard a physicist emit the same sound-vibrations. Since verbal behavior (spoken or written) is what gets the gold star, students begin to think that verbal behavior has a truth-value. After all, either light is made of waves, or it isn't, right?
And this leads into an even worse habit. Suppose the teacher presents you with a confusing problem involving a metal plate next to a radiator; the far side feels warmer than the side next to the radiator. The teacher asks "Why?" If you say "I don't know", you have no chance of getting a gold star - it won't even count as class participation. But, during the current semester, this teacher has used the phrases "because of heat convection", "because of heat conduction", and "because of radiant heat". One of these is probably what the teacher wants. You say, "Eh, maybe because of heat conduction?"
This is not a hypothesis about the metal plate. This is not even a proper belief. It is an attempt to guess the teacher's password.
Even visualizing the symbols of the diffusion equation (the math governing heat conduction) doesn't mean you've formed a hypothesis about the metal plate. This is not school; we are not testing your memory to see if you can write down the diffusion equation. This is Bayescraft; we are scoring your anticipations of experience. If you use the diffusion equation, by measuring a few points with a thermometer and then trying to predict what the thermometer will say on the next measurement, then it is definitely connected to experience. Even if the student just visualizes something flowing, and therefore holds a match near the cooler side of the plate to try to measure where the heat goes, then this mental image of flowing-ness connects to experience; it controls anticipation.
If you aren't using the diffusion equation - putting in
numbers and getting out results that control your anticipation of
particular experiences - then the connection between map and
territory is severed as though by a knife. What remains
is not a belief,
but a verbal behavior.
In the school system, it's all about verbal behavior, whether
written on paper or spoken aloud. Verbal behavior gets you a
gold star or a failing grade. Part of unlearning this bad
habit is becoming consciously aware of the difference between an
explanation and a password.
Does this seem too harsh? When you're faced by a confusing metal plate, can't "Heat conduction?" be a first step toward finding the answer? Maybe, but only if you don't fall into the trap of thinking that you are looking for a password. What if there is no teacher to tell you that you failed? Then you may think that "Light is wakalixes" is a good explanation, that "wakalixes" is the correct password. It happened to me when I was nine years old - not because I was stupid, but because this is what happens by default. This is how human beings think, unless they are trained not to fall into the trap. Humanity stayed stuck in holes like this for thousands of years.
Maybe, if we drill students that words don't count, only anticipation-controllers, the student will not get stuck on "Heat conduction? No? Maybe heat convection? That's not it either?" Maybe then, thinking the phrase "Heat conduction" will lead onto a genuinely helpful path, like:
- "Heat conduction?"
- But that's only a phrase - what does it mean?
- The diffusion equation?
- But those are only symbols - how do I apply them?
- What does applying the diffusion equation lead me to anticipate?
- It sure doesn't lead me to anticipate that the side of a metal plate farther away from a radiator would feel warmer.
- I notice that I am confused. Maybe the near side just feels cooler, because it's made of more insulative material and transfers less heat to my hand? I'll try measuring the temperature...
- Okay, that wasn't it. Can I try to verify whether the diffusion equation holds true of this metal plate, at all? Is heat flowing the way it usually does, or is something else going on?
- I could hold a match to the plate and try to measure how heat spreads over time...
If we are not strict about "Eh, maybe because of heat conduction?" being a fake explanation, the student will very probably get stuck on some wakalixes-password. This happens by default, it happened to the whole human species for thousands of years.
August 23, 2007
Science as Attire
Prerequisites:
Fake
Explanations, Belief
As Attire
The preview for the X-Men movie has a voice-over saying: "In every human being... there is the genetic code... for mutation." Apparently you can acquire all sorts of neat abilities by mutation. The mutant Storm, for example, has the ability to throw lightning bolts.
I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts. If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection. It would be worse than finding rabbit fossils in the pre-Cambrian. If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.
The X-Men comics use terms like "evolution", "mutation", and "genetic code", purely to place themselves in what they conceive to be the literary genre of science. The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.
I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits. They'll talk about "the next step in the evolution of humanity", as if natural selection got here by following a plan. Or even worse, they'll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they'll call that "evolution". If evolutionary biology could cover that, it could cover anything.
Probably an actual majority of the people who believe in evolution use the phrase "because of evolution" because they want to be part of the scientific in-crowd - belief as scientific attire, like wearing a lab coat. If the scientific in-crowd instead used the phrase "because of intelligent design", they would just as cheerfully use that instead - it would make no difference to their anticipation-controllers. Saying "because of evolution" instead of "because of intelligent design" does not, for them, prohibit Storm. Its only purpose, for them, is to identify with a tribe.
I encounter people who are quite willing to entertain the notion of dumber-than-human Artificial Intelligence, or even mildly smarter-than-human Artificial Intelligence. Introduce the notion of strongly superhuman Artificial Intelligence, and they'll suddenly decide it's "pseudoscience". It's not that they think they have a theory of intelligence which lets them calculate a theoretical upper bound on the power of an optimization process. Rather, they associate strongly superhuman AI to the literary genre of apocalyptic literature; whereas an AI running a small corporation associates to the literary genre of Wired magazine. They aren't speaking from within a model of cognition. They don't realize they need a model. They don't realize that science is about models. Their devastating critiques consist purely of comparisons to apocalyptic literature, rather than, say, known laws which prohibit such an outcome. They understand science only as a literary genre, or in-group to belong to. The attire doesn't look to them like a lab coat; this isn't the football team they're cheering for.
Is there anything in science that you are proud of believing, and yet you do not use the belief professionally? You had best ask yourself which future experiences your belief prohibits from happening to you. That is the sum of what you have assimilated and made a true part of yourself. Anything else is probably passwords or attire.
August 23, 2007
Fake Causality
Followup to: Fake Explanations, Guessing the Teacher's Password
Phlogiston was the 18 century's answer to the Elemental Fire of the Greek alchemists. Ignite wood, and let it burn. What is the orangey-bright "fire" stuff? Why does the wood transform into ash? To both questions, the 18th-century chemists answered, "phlogiston".
...and that was it, you see, that was their answer: "Phlogiston."
Phlogiston escaped from burning substances as visible fire. As the phlogiston escaped, the burning substances lost phlogiston and so became ash, the "true material". Flames in enclosed containers went out because the air became saturated with phlogiston, and so could not hold any more. Charcoal left little residue upon burning because it was nearly pure phlogiston.
Of course, one didn't use phlogiston theory to predict the outcome of a chemical transformation. You looked at the result first, then you used phlogiston theory to explain it. It's not that phlogiston theorists predicted a flame would extinguish in a closed container; rather they lit a flame in a container, watched it go out, and then said, "The air must have become saturated with phlogiston." You couldn't even use phlogiston theory to say what you ought not to see; it could explain everything.
This was an earlier age of science. For a long time, no one realized there was a problem. Fake explanations don't feel fake. That's what makes them dangerous.
Modern research suggests that humans think about cause and effect using something like the directed acyclic graphs (DAGs) of Bayes nets. Because it rained, the sidewalk is wet; because the sidewalk is wet, it is slippery:
[Rain] -> [Sidewalk wet] -> [Sidewalk slippery]
From this we can infer - or, in a Bayes net, rigorously calculate in probabilities - that when the sidewalk is slippery, it probably rained; but if we already know that the sidewalk is wet, learning that the sidewalk is slippery tells us nothing more about whether it rained.
Why is fire hot and bright when it burns?
["Phlogiston"] -> [Fire hot and bright]
It feels like an explanation. It's represented using the same cognitive data format. But the human mind does not automatically detect when a cause has an unconstraining arrow to its effect. Worse, thanks to hindsight bias, it may feel like the cause constrains the effect, when it was merely fitted to the effect.
Interestingly, our modern understanding of probabilistic reasoning about causality can describe precisely what the phlogiston theorists were doing wrong. One of the primary inspirations for Bayesian networks was noticing the problem of resonant updating between an effect and a cause. For example, let's say that I get a bit of unreliable information that the sidewalk is wet. This should make me think it's more likely to be raining. But, if it's more likely to be raining, doesn't that make it more likely that the sidewalk is wet? And wouldn't that make it more likely that the sidewalk is slippery? But if the sidewalk is slippery, it's probably wet; and then I should again raise my probability that it's raining...
Judea Pearl uses the metaphor of an algorithm for counting soldiers in a line. Suppose you're in the line, and you see two soldiers next to you, one in front and one in back. That's three soldiers. So you ask the soldier next to you, "How many soldiers do you see?" He looks around and says, "Three". So that's a total of six soldiers. This, obviously, is not how to do it.
A smarter way is to ask the soldier in front of you, "How many soldiers forward of you?" and the soldier in back, "How many soldiers backward of you?" The question "How many soldiers forward?" can be passed on as a message without confusion. If I'm at the front of the line, I pass the message "1 soldier forward", for myself. The person directly in back of me gets the message "1 soldier forward", and passes on the message "2 soldiers forward" to the soldier behind him. At the same time, each soldier is also getting the message "N soldiers backward" from the soldier behind them, and passing it on as "N+1 soldiers backward" to the soldier in front of them. How many soldiers in total? Add the two numbers you receive, plus one for yourself: that is the total number of soldiers in line.
The key idea is that every soldier must separately track the two messages, the forward-message and backward-message, and add them together only at the end. You never add any soldiers from the backward-message you receive to the forward-message you pass back. Indeed, the total number of soldiers is never passed as a message - no one ever says it aloud.
An analogous principle operates in rigorous probabilistic reasoning about causality. If you learn something about whether it's raining, from some source other than observing the sidewalk to be wet, this will send a forward-message from [rain] to [sidewalk wet] and raise our expectation of the sidewalk being wet. If you observe the sidewalk to be wet, this sends a backward-message to our belief that it is raining, and this message propagates from [rain] to all neighboring nodes except the [sidewalk wet] node. We count each piece of evidence exactly once; no update message ever "bounces" back and forth. The exact algorithm may be found in Judea Pearl's classic "Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference".
So what went wrong in phlogiston theory? When we observe that fire is hot, the [fire] node can send a backward-evidence to the ["phlogiston"] node, leading us to update our beliefs about phlogiston. But if so, we can't count this as a successful forward-prediction of phlogiston theory. The message should go in only one direction, and not bounce back.
Alas, human beings do not use a rigorous algorithm for updating belief networks. We learn about parent nodes from observing children, and predict child nodes from beliefs about parents. But we don't keep rigorously separate books for the backward-message and forward-message. We just remember that phlogiston is hot, which causes fire to be hot. So it seems like phlogiston theory predicts the hotness of fire. Or, worse, it just feels like phlogiston makes the fire hot.
Until you notice that no advance predictions are being made, the non-constraining causal node is not labeled "fake". It's represented the same way as any other node in your belief network. It feels like a fact, like all the other facts you know: Phlogiston makes the fire hot.
A properly designed AI would notice the problem instantly. This wouldn't even require special-purpose code, just correct bookkeeping of the belief network. (Sadly, we humans can't rewrite our own code, the way a properly designed AI could.)
Speaking of "hindsight bias" is just the nontechnical way of saying that humans do not rigorously separate forward and backward messages, allowing forward messages to be contaminated by backward ones.
Those who long ago went down the path of phlogiston were not trying to be fools. No scientist deliberately wants to get stuck in a blind alley. Are there any fake explanations in your mind? If there are, I guarantee they're not labeled "fake explanation", so polling your thoughts for the "fake" keyword will not turn them up.
Thanks to hindsight bias, it's also not
enough to check how well your theory "predicts" facts you already
know. You've got to predict for tomorrow, not
yesterday. It's the only way a messy human mind can be
guaranteed of sending a pure forward message.
August 24, 2007
Semantic Stopsigns
And the child asked:
Q: Where did this rock come from?
A: I chipped it off the big boulder, at the center of the
village.
Q: Where did the boulder come from?
A: It probably rolled off the huge mountain that towers over
our village.
Q: Where did the mountain come from?
A: The same place as all stone: it is the bones of Ymir, the
primordial giant.
Q: Where did the primordial giant, Ymir, come from?
A: From the great abyss, Ginnungagap.
Q: Where did the great abyss, Ginnungagap, come from?
A: Never ask that question.
Consider the seeming paradox of the First Cause. Science has traced events back to the Big Bang, but why did the Big Bang happen? It's all well and good to say that the zero of time begins at the Big Bang - that there is nothing before the Big Bang in the ordinary flow of minutes and hours. But saying this presumes our physical law, which itself appears highly structured; it calls out for explanation. Where did the physical laws come from? You could say that we're all a computer simulation, but then the computer simulation is running on some other world's laws of physics - where did those laws of physics come from?
At this point, some people say, "God!"
What could possibly make anyone, even a highly religious person, think this even helped answer the paradox of the First Cause? Why wouldn't you automatically ask, "Where did God come from?" Saying "God is uncaused" or "God created Himself" leaves us in exactly the same position as "Time began with the Big Bang." We just ask why the whole metasystem exists in the first place, or why some events but not others are allowed to be uncaused.
My purpose here is not to discuss the seeming paradox of the First Cause, but to ask why anyone would think "God!" could resolve the paradox. Saying "God!" is a way of belonging to a tribe, which gives people a motive to say it as often as possible - some people even say it for questions like "Why did this hurricane strike New Orleans?" Even so, you'd hope people would notice that on the particular puzzle of the First Cause, saying "God!" doesn't help. It doesn't make the paradox seem any less paradoxical even if true. How could anyone not notice this?
Jonathan Wallace suggested that "God!" functions as a semantic stopsign - that it isn't a propositional assertion, so much as a cognitive traffic signal: do not think past this point. Saying "God!" doesn't so much resolve the paradox, as put up a cognitive traffic signal to halt the obvious continuation of the question-and-answer chain.
Of course you'd never do that, being a good and proper atheist, right? But "God!" isn't the only semantic stopsign, just the obvious first example.
The transhuman technologies - molecular nanotechnology, advanced biotech, genetech, Artificial Intelligence, et cetera - pose tough policy questions. What kind of role, if any, should a government take in supervising a parent's choice of genes for their child? Could parents deliberately choose genes for schizophrenia? If enhancing a child's intelligence is expensive, should governments help ensure access, to prevent the emergence of a cognitive elite? You can propose various institutions to answer these policy questions - for example, that private charities should provide financial aid for intelligence enhancement - but the obvious next question is, "Will this institution be effective?" If we rely on product liability lawsuits to prevent corporations from building harmful nanotech, will that really work?
I know someone whose answer to every one of these questions is "Liberal democracy!" That's it. That's his answer. If you ask the obvious question of "How well have liberal democracies performed, historically, on problems this tricky?" or "What if liberal democracy does something stupid?" then you're an autocrat, or libertopian, or otherwise a very very bad person. No one is allowed to question democracy.
I once called this kind of thinking "the divine right of democracy". But it is more precise to say that "Democracy!" functioned for him as a semantic stopsign. If anyone had said to him "Turn it over to the Coca-Cola corporation!", he would have asked the obvious next questions: "Why? What will the Coca-Cola corporation do about it? Why should we trust them? Have they done well in the past on equally tricky problems?"
Or suppose that someone says "Mexican-Americans are plotting to remove all the oxygen in Earth's atmosphere." You'd probably ask, "Why would they do that? Don't Mexican-Americans have to breathe too? Do Mexican-Americans even function as a unified conspiracy?" If you don't ask these obvious next questions when someone says, "Corporations are plotting to remove Earth's oxygen," then "Corporations!" functions for you as a semantic stopsign.
Be careful here not to create a new generic counterargument against things you don't like - "Oh, it's just a stopsign!" No word is a stopsign of itself; the question is whether a word has that effect on a particular person. Having strong emotions about something doesn't qualify it as a stopsign. I'm not exactly fond of terrorists or fearful of private property; that doesn't mean "Terrorists!" or "Capitalism!" are cognitive traffic signals unto me. (The word "intelligence" did once have that effect on me, though no longer.) What distinguishes a semantic stopsign is failure to consider the obvious next question.
August 25, 2007
Mysterious Answers to Mysterious Questions
Imagine looking at your hand, and knowing nothing of cells, nothing of biochemistry, nothing of DNA. You've learned some anatomy from dissection, so you know your hand contains muscles; but you don't know why muscles move instead of lying there like clay. Your hand is just... stuff... and for some reason it moves under your direction. Is this not magic?
"The animal body does not act as a thermodynamic engine ... consciousness teaches every individual that they are, to some extent, subject to the direction of his will. It appears therefore that animated creatures have the power of immediately applying to certain moving particles of matter within their bodies, forces by which the motions of these particles are directed to produce derived mechanical effects... The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concurrence of atoms... Modern biologists were coming once more to the acceptance of something and that was a vital principle."
-- Lord Kelvin
This was the theory of vitalism; that the mysterious difference between living matter and non-living matter was explained by an elan vital or vis vitalis. Elan vital infused living matter and caused it to move as consciously directed. Elan vital participated in chemical transformations which no mere non-living particles could undergo - Wöhler's later synthesis of urea, a component of urine, was a major blow to the vitalistic theory because it showed that merechemistry could duplicate a product of biology.
Calling "elan vital" an explanation, even a fake explanation like phlogiston, is probably giving it too much credit. It functioned primarily as a curiosity-stopper. You said "Why?" and the answer was "Elan vital!"
When you say "Elan vital!", it feels like you know why your hand moves. You have a little causal diagram in your head that says ["Elan vital!"] -> [hand moves]. But actually you know nothing you didn't know before. You don't know, say, whether your hand will generate heat or absorb heat, unless you have observed the fact already; if not, you won't be able to predict it in advance. Your curiosity feels sated, but it hasn't been fed. Since you can say "Why? Elan vital!" to any possible observation, it is equally good at explaining all outcomes, a disguised hypothesis of maximum entropy, etcetera.
But the greater lesson lies in the vitalists' reverence for the elan vital, their eagerness to pronounce it a mystery beyond all science. Meeting the great dragon Unknown, the vitalists did not draw their swords to do battle, but bowed their necks in submission. They took pride in their ignorance, made biology into a sacred mystery, and thereby became loath to relinquish their ignorance when evidence came knocking.
The Secret of Life was infinitely beyond the reach of science! Not just a little beyond, mind you, but infinitely beyond! Lord Kelvin sure did get a tremendous emotional kick out of not knowing something.
But ignorance exists in the map, not in the territory. If I am ignorant about a phenomenon, that is a fact about my own state of mind, not a fact about the phenomenon itself. A phenomenon can seem mysterious to some particular person. There are no phenomena which are mysterious of themselves. To worship a phenomenon because it seems so wonderfully mysterious, is to worship your own ignorance.
Vitalism shared with phlogiston the error of encapsulating the mystery as a substance. Fire was mysterious, and the phlogiston theory encapsulated the mystery in a mysterious substance called "phlogiston". Life was a sacred mystery, and vitalism encapsulated the sacred mystery in a mysterious substance called "elan vital". Neither answer helped concentrate the model's probability density - make some outcomes easier to explain than others. The "explanation" just wrapped up the question as a small, hard, opaque black ball.
In a comedy written by Moliere, a physician explains the power of a soporific by saying that it contains a "dormitive potency". Same principle. It is a failure of human psychology that, faced with a mysterious phenomenon, we more readily postulate mysterious inherent substances than complex underlying processes.
But the deeper failure is supposing that an answer can be mysterious. If a phenomenon feels mysterious, that is a fact about our state of knowledge, not a fact about the phenomenon itself. The vitalists saw a mysterious gap in their knowledge, and postulated a mysterious stuff that plugged the gap. In doing so, they mixed up the map with the territory. All confusion and bewilderment exist in the mind, not in encapsulated substances.
This is the ultimate and fully general explanation for why, again and again in humanity's history, people are shocked to discover that an incredibly mysterious question has a non-mysterious answer. Mystery is a property of questions, not answers.
Therefore I call theories such as vitalism mysterious answers to mysterious questions.
These are the signs of mysterious answers to mysterious questions:
- First, the explanation acts as a curiosity-stopper rather than an anticipation-controller.
- Second, the hypothesis has no moving parts - the model is not a specific complex mechanism, but a blankly solid substance or force. The mysterious substance or mysterious force may be said to be here or there, to cause this or that; but the reason why the mysterious force behaves thus is wrapped in a blank unity.
- Third, those who proffer the explanation cherish their ignorance; they speak proudly of how the phenomenon defeats ordinary science or is unlike merely mundane phenomena.
- Fourth, even after the answer is given, the phenomenon is still a mystery and possesses the same quality of wonderful inexplicability that it had at the start.
August 26, 2007
The Futility of Emergence
Prerequisites: Belief in Belief, Fake Explanations, Fake Causality, Mysterious Answers to Mysterious Questions
The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a
limb, and name some current theory which I deem
analogously flawed?
I name emergence or emergent phenomena - usually defined as the study of systems whose high-level behaviors arise or "emerge" from the interaction of many low-level elements. (Wikipedia: "The way complex systems and patterns arise out of a multiplicity of relatively simple interactions".) Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a market crash and saying "It's not a quark!" Does that feel like an explanation? No? Then neither should saying "It's an emergent phenomenon!"
It's the noun "emergence" that I protest, rather than the verb "emerges from". There's nothing wrong with saying "X emerges from Y", where Y is some specific, detailed model with internal moving parts. "Arises from" is another legitimate phrase that means exactly the same thing: Gravity arises from the curvature of spacetime, according to the specific mathematical model of General Relativity. Chemistry arises from interactions between atoms, according to the specific model of quantum electrodynamics.
Now suppose I should say that gravity is explained by "arisence" or that chemistry is an "arising phenomenon", and claim that as my explanation.
The phrase "emerges from" is acceptable, just like "arises from" or "is caused by" are acceptable, if the phrase precedes some specific model to be judged on its own merits.
However, this is not the way "emergence" is commonly used. "Emergence" is commonly used as an explanation in its own right.
I have lost track of how many times I have heard people say, "Intelligence is an emergent phenomenon!" as if that explained intelligence. This usage fits all the checklist items for a mysterious answer to a mysterious question. What do you know, after you have said that intelligence is "emergent"? You can make no new predictions. You do not know anything about the behavior of real-world minds that you did not know before. It feels like you believe a new fact, but you don't anticipate any different outcomes. Your curiosity feels sated, but it has not been fed. The hypothesis has no moving parts - there's no detailed internal model to manipulate. Those who proffer the hypothesis of "emergence" confess their ignorance of the internals, and take pride in it; they contrast the science of "emergence" to other sciences merely mundane.
And even after the answer of "Why? Emergence!" is given, the phenomenon is still a mystery and possesses the same sacred impenetrability it had at the start.
A fun exercise is to eliminate the adjective "emergent" from any
sentence in which it appears, and see if the sentence says anything
different:
- Before: Human intelligence is an emergent product of neurons firing.
- After: Human intelligence is a product of neurons firing.
- Before: The behavior of the ant colony is the emergent outcome of the interactions of many individual ants.
- After: The behavior of the ant colony is the outcome of the interactions of many individual ants.
- Even better: A colony is made of ants. We can successfully predict some aspects of colony behavior using models that include only individual ants, without any global colony variables, showing that we understand how those colony behaviors arise from ant behaviors.
Another fun exercise is to replace the word "emergent" with the
old word, the
explanation that
people had to use before emergence was invented:
- Before: Life is an emergent phenomenon.
- After: Life is a magical phenomenon.
- Before: Human intelligence is an emergent product of neurons firing.
- After: Human intelligence is a magical product of neurons firing.
Does not each statement convey exactly the same amount of knowledge about the phenomenon's behavior? Does not each hypothesis fit exactly the same set of outcomes?
"Emergence" has become very popular, just as saying "magic" used to be very popular. "Emergence" has the same deep appeal to human psychology, for the same reason. "Emergence" is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Emergence is popular because it is the junk food of curiosity. You can explain anything using emergence, and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they've taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the literary genre of "science" but still the same species psychology.
August 27, 2007
Positive Bias: Look Into the Dark
I am teaching a class, and I write upon the blackboard three numbers: 2-4-6. "I am thinking of a rule," I say, "which governs sequences of three numbers. The sequence 2-4-6, as it so happens, obeys this rule. Each of you will find, on your desk, a pile of index cards. Write down a sequence of three numbers on a card, and I'll mark it "Yes" for fits the rule, or "No" for not fitting the rule. Then you can write down another set of three numbers and ask whether it fits again, and so on. When you're confident that you know the rule, write down the rule on a card. You can test as many triplets as you like."
Here's the record of one student's guesses:
4, 6, 2
No
4, 6, 8
Yes
10, 12, 14 Yes
At this point the student wrote down his guess at the rule. What do you think the rule is? Would you have wanted to test another triplet, and if so, what would it be? Take a moment to think before continuing.
The challenge above is based on a classic experiment due to Peter Wason, the 2-4-6 task. Although subjects given this task typically expressed high confidence in their guesses, only 21% of the subjects successfully guessed the experimenter's real rule, and replications since then have continued to show success rates of around 20%.
The study was called "On the failure to eliminate hypotheses in a conceptual task" (Quarterly Journal of Experimental Psychology, 12: 129-140, 1960). Subjects who attempt the 2-4-6 task usually try to generate positive examples, rather than negative examples - they apply the hypothetical rule to generate a representative instance, and see if it is labeled "Yes".
Thus, someone who forms the hypothesis "numbers increasing by two" will test the triplet 8-10-12, hear that it fits, and confidently announce the rule. Someone who forms the hypothesis X-2X-3X will test the triplet 3-6-9, discover that it fits, and then announce that rule.
In every case the actual rule is the same: the three numbers must be in ascending order.
But to discover this, you would have to generate triplets that shouldn't fit, such as 20-23-26, and see if they are labeled "No". Which people tend not to do, in this experiment. In some cases, subjects devise, "test", and announce rules far more complicated than the actual answer.
This cognitive phenomenon is usually lumped in with "confirmation bias". However, it seems to me that the phenomenon of trying to test positive rather than negative examples, ought to be distinguished from the phenomenon of trying to preserve the belief you started with. "Positive bias" is sometimes used as a synonym for "confirmation bias", and fits this particular flaw much better.
It once seemed that phlogiston theory could explain a flame going out in an enclosed box (the air became saturated with phlogiston and no more could be released), but phlogiston theory could just as well have explained the flame not going out. To notice this, you have to search for negative examples instead of positive examples, look into zero instead of one; which goes against the grain of what experiment has shown to be human instinct.
For by instinct, we human beings only live in half the world.
One may be lectured on positive bias for days, and yet overlook it in-the-moment. Positive bias is not something we do as a matter of logic, or even as a matter of emotional attachment. The 2-4-6 task is "cold", logical, not affectively "hot". And yet the mistake is sub-verbal, on the level of imagery, of instinctive reactions. Because the problem doesn't arise from following a deliberate rule that says "Only think about positive examples", it can't be solved just by knowing verbally that "We ought to think about both positive and negative examples." Which example automatically pops into your head? You have to learn, wordlessly, to zag instead of zig. You have to learn to flinch toward the zero, instead of away from it.
I have been writing for quite some time now on the notion that the strength of a hypothesis is what it can't explain, not what it can - if you are equally good at explaining any outcome, you have zero knowledge. So to spot an explanation that isn't helpful, it's not enough to think of what it does explain very well - you also have to search for results it couldn't explain, and this is the true strength of the theory.
So I said all this, and then yesterday, I challenged the usefulness of "emergence" as a concept. One commenter cited superconductivity and ferromagnetism as examples of emergence. I replied that non-superconductivity and non-ferromagnetism were also examples of emergence, which was the problem. But be it far from me to criticize the commenter! Despite having read extensively on "confirmation bias", I didn't spot the "gotcha" in the 2-4-6 task the first time I read about it. It's a subverbal blink-reaction that has to be retrained. I'm still working on it myself.
So much of a rationalist's skill is below the level of words. It makes for challenging work in trying to convey the Art through blog posts. People will agree with you, but then, in the next sentence, do something subdeliberative that goes in the opposite direction. Not that I'm complaining! A major reason I'm posting here is to observe what my words haven't conveyed.
Are you searching for positive examples of positive bias right now, or sparing a fraction of your search on what positive bias should lead you to not see? Did you look toward light or darkness?
August 29, 2007
Say Not "Complexity"
Once upon a time...
This is a story from when I first met Marcello, with whom I would later work for a year on AI theory; but at this point I had not yet accepted him as my apprentice. I knew that he competed at the national level in mathematical and computing olympiads, which sufficed to attract my attention for a closer look; but I didn't know yet if he could learn to think about AI.
I had asked Marcello to say how he thought an AI might discover how to solve a Rubik's Cube. Not in a preprogrammed way, which is trivial, but rather how the AI itself might figure out the laws of the Rubik universe and reason out how to exploit them. How would an AI invent for itself the concept of an "operator", or "macro", which is the key to solving the Rubik's Cube?
At some point in this discussion, Marcello said: "Well, I think the AI needs complexity to do X, and complexity to do Y -"
And I said, "Don't say 'complexity'."
Marcello said, "Why not?"
I said, "Complexity should never be a goal in itself. You may need to use a particular algorithm that adds some amount of complexity, but complexity for the sake of complexity just makes things harder." (I was thinking of all the people whom I had heard advocating that the Internet would "wake up" and become an AI when it became "sufficiently complex".)
And Marcello said, "But there's got to be some amount of complexity that does it."
I closed my eyes briefly, and tried to think of how to explain it all in words. To me, saying 'complexity' simply felt like the wrong move in the AI dance. No one can think fast enough to deliberate, in words, about each sentence of their stream of consciousness; for that would require an infinite recursion. We think in words, but our stream of consciousness is steered below the level of words, by the trained-in remnants of past insights and harsh experience...
I said, "Did you read A Technical Explanation of Technical Explanation?"
"Yes," said Marcello.
"Okay," I said, "saying 'complexity' doesn't concentrate your probability mass."
"Oh," Marcello said, "like 'emergence'. Huh. So... now I've got to think about how X might actually happen..."
That was when I thought to myself, "Maybe this one is teachable."
Complexity is not a useless concept. It has mathematical definitions attached to it, such as Kolmogorov complexity, and Vapnik-Chervonenkis complexity. Even on an intuitive level, complexity is often worth thinking about - you have to judge the complexity of a hypothesis and decide if it's "too complicated" given the supporting evidence, or look at a design and try to make it simpler.
But concepts are not useful or useless of themselves. Only usages are correct or incorrect. In the step Marcello was trying to take in the dance, he was trying to explain something for free, get something for nothing. It is an extremely common misstep, at least in my field. You can join a discussion on Artificial General Intelligence and watch people doing the same thing, left and right, over and over again - constantly skipping over things they don't understand, without realizing that's what they're doing.
In an eyeblink it happens: putting a non-controlling causal node behind something mysterious, a causal node that feels like an explanation but isn't. The mistake takes place below the level of words. It requires no special character flaw; it is how human beings think by default, since the ancient times.
What you must avoid is skipping over the mysterious part; you must linger at the mystery to confront it directly. There are many words that can skip over mysteries, and some of them would be legitimate in other contexts - "complexity", for example. But the essential mistake is that skip-over, regardless of what causal node goes behind it. The skip-over is not a thought, but a microthought. You have to pay close attention to catch yourself at it. And when you train yourself to avoid skipping, it will become a matter of instinct, not verbal reasoning. You have to feel which parts of your map are still blank, and more importantly, pay attention to that feeling.
I suspect that in academia there is a huge pressure to sweep problems under the rug so that you can present a paper with the appearance of completeness. You'll get more kudos for a seemingly complete model that includes some "emergent phenomena", versus an explicitly incomplete map where the label says "I got no clue how this part works" or "then a miracle occurs". A journal may not even accept the latter paper, since who knows but that the unknown steps are really where everything interesting happens? And yes, it sometimes happens that all the non-magical parts of your map turn out to also be non-important. That's the price you sometimes pay, for entering into terra incognita and trying to solve problems incrementally. But that makes it even more important to know when you aren't finished yet. Mostly, people don't dare to enter terra incognita at all, for the deadly fear of wasting their time.
And if you're working on a revolutionary AI startup, there is an even huger pressure to sweep problems under the rug; or you will have to admit to yourself that you don't know how to build an AI yet, and your current life-plans will come crashing down in ruins around your ears. But perhaps I am over-explaining, since skip-over happens by default in humans; if you're looking for examples, just watch people discussing religion or philosophy or spirituality or any science in which they were not professionally trained.
Marcello and I developed a convention in our AI work: when we ran into something we didn't understand, which was often, we would say "magic" - as in, "X magically does Y" - to remind ourselves that here was an unsolved problem, a gap in our understanding. It is far better to say "magic", than "complexity" or "emergence"; the latter words create an illusion of understanding. Wiser to say "magic", and leave yourself a placeholder, a reminder of work you will have to do later.
August 29, 2007
My Wild and Reckless Youth
It is said that parents do all the things they tell their children not to do, which is how they know not to do them.
Long ago, in the unthinkably distant past, I was a devoted Traditional Rationalist, conceiving myself skilled according to that kind, yet I knew not the Way of Bayes. When the young Eliezer was confronted with a mysterious-seeming question, the precepts of Traditional Rationality did not stop him from devising a Mysterious Answer. It is, by far, the most embarrassing mistake I made in my life, and I still wince to think of it.
What was my mysterious answer to a mysterious question? This I will not describe, for it would be a long tale and complicated. I was young, and a mere Traditional Rationalist who knew not the teachings of Tversky and Kahneman. I knew about Occam's Razor, but not the conjunction fallacy. I thought I could get away with thinking complicated thoughts myself, in the literary style of the complicated thoughts I read in science books, not realizing that correct complexity is only possible when every step is pinned down overwhelmingly. Today, one of the chief pieces of advice I give to aspiring young rationalists is "Do not attempt long chains of reasoning or complicated plans."
Nothing more than this need be said: Even after I invented my "answer", the phenomenon was still a mystery unto me, and possessed the same quality of wondrous impenetrability that it had at the start.
Make no mistake, that younger Eliezer was not stupid. All the errors of which the young Eliezer was guilty, are still being made today by respected scientists in respected journals. It would have taken a subtler skill to protect him, than ever he was taught as a Traditional Rationalist.
Indeed, the young Eliezer diligently and painstakingly followed the injunctions of Traditional Rationality in the course of going astray.
As a Traditional Rationalist, the young Eliezer was careful to ensure that his Mysterious Answer made a bold prediction of future experience. Namely, I expected future neurologists to discover that neurons were exploiting quantum gravity, a la Sir Roger Penrose. This required neurons to maintain a certain degree of quantum coherence, which was something you could look for, and find or not find. Either you observe that or you don't, right?
But my hypothesis made no retrospective predictions. According to Traditional Science, retrospective predictions don't count - so why bother making them? To a Bayesian, on the other hand, if a hypothesis does not today have a favorable likelihood ratio over "I don't know", it raises the question of why you today believe anything more complicated than "I don't know". But I knew not the Way of Bayes, so I was not thinking about likelihood ratios or focusing probability density. I had Made a Falsifiable Prediction; was this not the Law?
As a Traditional Rationalist, the young Eliezer was careful not to believe in magic, mysticism, carbon chauvinism, or anything of that sort. I proudly professed of my Mysterious Answer, "It is just physics like all the rest of physics!" As if you could save magic from being a cognitive isomorph of magic, by calling it quantum gravity. But I knew not the Way of Bayes, and did not see the level on which my idea was isomorphic to magic. I gave my allegiance to physics, but this did not save me; what does probability theory know of allegiances? I avoided everything that Traditional Rationality told me was forbidden, but what was left was still magic.
Beyond a doubt, my allegiance to Traditional Rationality helped me get out of the hole I dug myself into. If I hadn't been a Traditional Rationalist, I would have been completely screwed. But Traditional Rationality still wasn't enough to get it right. It just led me into different mistakes than the ones it had explicitly forbidden.
When I think about how my younger self very carefully followed the rules of Traditional Rationality in the course of getting the answer wrong, it sheds light on the question of why people who call themselves "rationalists" do not rule the world. You need one whole hell of a lot of rationality before it does anything but lead you into new and interesting mistakes.
Traditional Rationality is taught as an art, rather than a science; you read the biography of famous physicists describing the lessons life taught them, and you try to do what they tell you to do. But you haven't lived their lives, and half of what they're trying to describe is an instinct that has been trained into them.
The way Traditional Rationality is designed, it would have been acceptable for me to spend 30 years on my silly idea, so long as I succeeded in falsifying it eventually, and was honest with myself about what my theory predicted, and accepted the disproof when it arrived, et cetera. This is enough to let the Ratchet of Science click forward, but it's a little harsh on the people who waste 30 years of their lives. Traditional Rationality is a walk, not a dance. It's designed to get you to the truth eventually, and gives you all too much time to smell the flowers along the way.
Traditional Rationalists can agree to disagree. Traditional Rationality doesn't have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence. In Traditional Rationality, you're allowed to guess, and then test your guess. But experience has taught me that if you don't know, and you guess, you'll end up being wrong.
The Way of Bayes is also an imprecise art, at least the way I'm holding forth upon it. These blog posts are still fumbling attempts to put into words lessons that would be better taught by experience. But at least there's underlying math, plus experimental evidence from cognitive psychology on how humans actually think. Maybe that will be enough to cross the stratospherically high threshold required for a discipline that lets you actually get it right, instead of just constraining you into interesting new mistakes.
August 30, 2007
Failing to Learn from History
Continuation of: My Wild and Reckless Youth
Once upon a time, in my wild and reckless youth,
when I knew not the Way of Bayes, I gave a Mysterious Answer to a
mysterious-seeming question. Many failures occurred in
sequence, but one mistake stands out as most critical: My
younger self did not realize that solving a mystery should make
it feel less confusing. I was trying to explain a
Mysterious Phenomenon - which to me meant providing a cause for it,
fitting it into an integrated model of reality. Why should
this make the phenomenon less Mysterious, when that is its
nature? I was trying to explain the Mysterious
Phenomenon, not render it (by some impossible alchemy) into a
mundane phenomenon, a phenomenon that wouldn't even call out for an
unusual explanation in the first place.
As a Traditional Rationalist, I knew the historical tales of
astrologers and astronomy, of alchemists and chemistry, of
vitalists and biology. But the Mysterious Phenomenon was not
like this. It was something new, something stranger,
something more difficult, something that ordinary science had
failed to explain for centuries -
- as if stars and matter and life had not been mysteries for hundreds of years and thousands of years, from the dawn of human thought right up until science finally solved them -
We learn about astronomy and chemistry and biology in school, and it seems to us that these matters have always been the proper realm of science, that they have never been mysterious. When science dares to challenge a new Great Puzzle, the children of that generation are skeptical, for they have never seen science explain something that feels mysterious to them. Science is only good for explaining scientific subjects, like stars and matter and life.
I thought the lesson of history was that astrologers and alchemists and vitalists had an innate character flaw, a tendency toward mysterianism, which led them to come up with mysterious explanations for non-mysterious subjects. But surely, if a phenomenon really was very weird, a weird explanation might be in order?
It was only afterward, when I began to see the mundane structure inside the mystery, that I realized whose shoes I was standing in. Only then did I realize how reasonable vitalism had seemed at the time, how surprising and embarrassing had been the universe's reply of, "Life is mundane, and does not need a weird explanation."
We read history but we don't live it, we don't experience it. If only I had personally postulated astrological mysteries and then discovered Newtonian mechanics, postulated alchemical mysteries and then discovered chemistry, postulated vitalistic mysteries and then discovered biology. I would have thought of my Mysterious Answer and said to myself: No way am I falling for that again.
August 31, 2007
Making History Available
Followup to: Failing to Learn from History
There is a habit of thought which I call the logical fallacy of generalization from fictional evidence, which deserves a blog post in its own right, one of these days. Journalists who, for example, talk about the Terminator movies in a report on AI, do not usually treat Terminator as a prophecy or fixed truth. But the movie is recalled - is available - as if it were an illustrative historical case. As if the journalist had seen it happen on some other planet, so that it might well happen here. More on this in Section 6 of this paper.
There is an inverse error to generalizing from fictional evidence: failing to be sufficiently moved by historical evidence. The trouble with generalizing from fictional evidence is that it is fiction - it never actually happened. It's not drawn from the same distribution as this, our real universe; fiction differs from reality in systematic ways. But history has happened, and should be available.
In our ancestral environment, there were no movies; what you saw with your own eyes was true. Is it any wonder that fictions we see in lifelike moving pictures have too great an impact on us? Conversely, things that really happened, we encounter as ink on paper; they happened, but we never saw them happen. We don't remember them happening to us.
The inverse error is to treat history as mere story, process it with the same part of your mind that handles the novels you read. You may say with your lips that it is "truth", rather than "fiction", but that doesn't mean you are being moved as much as you should be. Many biases involve being insufficiently moved by dry, abstract information.
Once upon a time, I gave a Mysterious Answer to a mysterious question, not realizing that I was making exactly the same mistake as astrologers devising mystical explanations for the stars, or alchemists devising magical properties of matter, or vitalists postulating an opaque "elan vital" to explain all of biology.
When I finally realized whose shoes I was standing in, there was a sudden shock of unexpected connection with the past. I realized that the invention and destruction of vitalism - which I had only read about in books - had actually happened to real people, who experienced it much the same way I experienced the invention and destruction of my own mysterious answer. And I also realized that if I had actually experienced the past - if I had lived through past scientific revolutions myself, rather than reading about them in history books - I probably would not have made the same mistake again. I would not have come up with another mysterious answer; the first thousand lessons would have hammered home the moral.
So (I thought), to feel sufficiently the force of history, I should try to approximate the thoughts of an Eliezer who had lived through history - I should try to think as if everything I read about in history books, had actually happened to me. (With appropriate reweighting for the availability bias of history books - I should remember being a thousand peasants for every ruler.) I should immerse myself in history, imagine living through eras I only saw as ink on paper.
Why should I remember the Wright Brothers' first flight? I was not there. But as a rationalist, could I dare to not remember, when the event actually happened? Is there so much difference between seeing an event through your eyes - which is actually a causal chain involving reflected photons, not a direct connection - and seeing an event through a history book? Photons and history books both descend by causal chains from the event itself.
I had to overcome the false amnesia of being born at a particular time. I had to recall - make available - all the memories, not just the memories which, by mere coincidence, belonged to myself and my own era.
The Earth became older, of a sudden.
To my former memory, the United States had always existed - there was never a time when there was no United States. I had not remembered, until that time, how the Roman Empire rose, and brought peace and order, and lasted through so many centuries, until I forgot that things had ever been otherwise; and yet the Empire fell, and barbarians overran my city, and the learning that I had possessed was lost. The modern world became more fragile to my eyes; it was not the first modern world.
So many mistakes, made over and over and over again, because I did not remember making them, in every era I never lived...
And to think, people sometimes wonder if overcoming bias is important.
Don't you remember how many times your biases have killed you? You don't? I've noticed that sudden amnesia often follows a fatal mistake. But take it from me, it happened. I remember; I wasn't there.
So the next time you doubt the strangeness of the future, remember how you were born in a hunter-gatherer tribe ten thousand years ago, when no one knew of Science at all. Remember how you were shocked, to the depths of your being, when Science explained the great and terrible sacred mysteries that you once revered so highly. Remember how you once believed that you could fly by eating the right mushrooms, and then you accepted with disappointment that you would never fly, and then you flew. Remember how you had always thought that slavery was right and proper, and then you changed your mind. Don't imagine how you could have predicted the change, for that is amnesia. Remember that, in fact, you did not guess. Remember how, century after century, the world changed in ways you did not guess.
Maybe then you will be less shocked by what happens next.
September 01, 2007
Stranger Than History
Suppose I told you that I knew for a fact that the following statements were true:
- If you paint yourself a certain exact color between blue and green, it will reverse the force of gravity on you and cause you to fall upward.
- In the future, the sky will be filled by billions of floating black spheres. Each sphere will be larger than all the zeppelins that have ever existed put together. If you offer a sphere money, it will lower a male prostitute out of the sky on a bungee cord.
- Your grandchildren will think it is not just foolish, but evil, to put thieves in jail instead of spanking them.
You'd think I was crazy, right?
Now suppose it were the year 1901, and you had to choose between believing those statements I have just offered, and believing statements like the following:
- There is an absolute speed limit on how fast two objects can seem to be traveling relative to each other, which is exactly 670616629.2 miles per hour. If you hop on board a train going almost this fast and fire a gun out the window, the fundamental units of length change around, so it looks to you like the bullet is speeding ahead of you, but other people see something different. Oh, and time changes around too.
- In the future, there will be a superconnected global network of billions of adding machines, each one of which has more power than all pre-1901 adding machines put together. One of the primary uses of this network will be to transport moving pictures of lesbian sex by pretending they are made out of numbers.
- Your grandchildren will think it is not just foolish, but evil, to say that someone should not be President of the United States because she is black.
Based on a comment of Robin's: "I wonder if one could describe in enough detail a fictional story of an alternative reality, a reality that our ancestors could not distinguish from the truth, in order to make it very clear how surprising the truth turned out to be."
September 02, 2007
Explain/Worship/Ignore?
Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions
As our tribe wanders through the grasslands, searching for fruit trees and prey, it happens every now and then that water pours down from the sky.
"Why does water sometimes fall from the sky?" I ask the bearded wise man of our tribe.
He thinks for a moment, this question having never occurred to him before, and then says, "From time to time, the sky spirits battle, and when they do, their blood drips from the sky."
"Where do the sky spirits come from?" I ask.
His voice drops to a whisper. "From the before time. From the long long ago."
When it rains, and you don't know why, you have several options. First, you could simply not ask why - not follow up on the question, or never think of the question in the first place. This is the Ignore command, which the bearded wise man originally selected. Second, you could try to devise some sort of explanation, the Explain command, as the bearded man did in response to your first question. Third, you could enjoy the sensation of mysteriousness - the Worship command.
Now, as you are bound to notice from this story, each time you select Explain, the best-case scenario is that you get an explanation, such as "sky spirits". But then this explanation itself is subject to the same dilemma - Explain, Worship, or Ignore? Each time you hit Explain, science grinds for a while, returns an explanation, and then another dialog box pops up. As good rationalists, we feel duty-bound to keep hitting Explain, but it seems like a road that has no end.
You hit Explain for life, and get chemistry; you hit Explain for chemistry, and get atoms; you hit Explain for atoms, and get electrons and nuclei; you hit Explain for nuclei, and get quantum chromodynamics and quarks; you hit Explain for how the quarks got there, and get back the Big Bang...
We can hit Explain for the Big Bang, and wait while science grinds through its process, and maybe someday it will return a perfectly good explanation. But then that will just bring up another dialog box. So, if we continue long enough, we must come to a special dialog box, a new option, an Explanation That Needs No Explanation, a place where the chain ends - and this, maybe, is the only explanation worth knowing.
There - I just hit Worship.
Never forget that there are many more ways to worship something than lighting candles around an altar.
If I'd said, "Huh, that does seem paradoxical. I wonder how the apparent paradox is resolved?" then I would have hit Explain, which does sometimes take a while to produce an answer.
And if the whole issue seems to you unimportant, or irrelevant, or if you'd rather put off thinking about it until tomorrow, than you have hit Ignore.
Select your option wisely.
September 03, 2007
"Science" as Curiosity-Stopper
Followup to: Semantic Stopsigns, Mysterious Answers to Mysterious Questions, Say Not 'Complexity'
Imagine that I, in full view of live television cameras, raised my hands and chanted abracadabra and caused a brilliant light to be born, flaring in empty space beyond my outstretched hands. Imagine that I committed this act of blatant, unmistakeable sorcery under the full supervision of James Randi and all skeptical armies. Most people, I think, would be fairly curious as to what was going on.
But now suppose instead that I don't go on television. I do not wish to share the power, nor the truth behind it. I want to keep my sorcery secret. And yet I also want to cast my spells whenever and wherever I please. I want to cast my brilliant flare of light so that I can read a book on the train - without anyone becoming curious. Is there a spell that stops curiosity?
Yes indeed! Whenever anyone asks "How did you do that?", I just say "Science!"
It's not a real explanation, so much as a curiosity-stopper. It doesn't tell you whether the light will brighten or fade, change color in hue or saturation, and it certainly doesn't tell you how to make a similar light yourself. You don't actually know anything more than you knew before I said the magic word. But you turn away, satisfied that nothing unusual is going on.
Better yet, the same trick works with a standard light switch.
Flip a switch and a light bulb turns on. Why?
In school, one is taught that the password to the light bulb is "Electricity!" By now, I hope, you're wary of marking the light bulb "understood" on such a basis. Does saying "Electricity!" let you do calculations that will control your anticipation of experience? There is, at the least, a great deal more to learn. (Physicists should ignore this paragraph and substitute a problem in evolutionary theory, where the substance of the theory is again in calculations that few people know how to perform.)
If you thought the light bulb was scientifically inexplicable, it would seize the entirety of your attention. You would drop whatever else you were doing, and focus on that light bulb.
But what does the phrase "scientifically explicable" mean? It means that someone else knows how the light bulb works. When you are told the light bulb is "scientifically explicable", you don't know more than you knew earlier; you don't know whether the light bulb will brighten or fade. But because someone else knows, it devalues the knowledge in your eyes. You become less curious.
Since this is an econblog, someone out there is bound to say, "If the light bulb were unknown to science, you could gain fame and fortune by investigating it." But I'm not talking about greed. I'm not talking about career ambition. I'm talking about the raw emotion of curiosity - the feeling of being intrigued. Why should your curiosity be diminished because someone else, not you, knows how the light bulb works? Is this not spite? It's not enough for you to know; other people must also be ignorant, or you won't be happy?
There are goods that knowledge may serve besides curiosity, such as the social utility of technology. For these instrumental goods, it matters whether some other entity in local space already knows. But for my own curiosity, why should it matter?
Besides, consider the consequences if you permit "Someone else knows the answer" to function as a curiosity-stopper. One day you walk into your living room and see a giant green elephant, seemingly hovering in midair, surrounded by an aura of silver light.
"What the heck?" you say.
And a voice comes from above the elephant, saying, "SOMEONE ELSE ALREADY KNOWS WHY THIS ELEPHANT IS HERE."
"Oh," you say, "in that case, never mind," and walk on to the kitchen.
I don't know the grand unified theory for this universe's laws of physics. I also don't know much about human anatomy with the exception of the brain. I couldn't point out on my body where my kidneys are, and I can't recall offhand what my liver does. (I am not proud of this. Alas, with all the math I need to study, I'm not likely to learn anatomy anytime soon.)
Should I, so far as curiosity is concerned, be more intrigued by my ignorance of the ultimate laws of physics, than the fact that I don't know much about what goes on inside my own body?
If I raised my hands and cast a light spell, you would be intrigued. Should you be any less intrigued by the very fact that I raised my hands? When you raise your arm and wave a hand around, this act of will is coordinated by (among other brain areas) your cerebellum. I bet you don't know how the cerebellum works. I know a little - though only the gross details, not enough to perform calculations... but so what? What does that matter, if you don't know? Why should there be a double standard of curiosity for sorcery and hand motions?
Look at yourself in the mirror. Do you know what you're looking at? Do you know what looks out from behind your eyes? Do you know what you are? Some of that answer, Science knows, and some of it Science does not. But why should that distinction matter to your curiosity, if you don't know?
Do you know how your knees work? Do you know how your shoes were made? Do you know why your computer monitor glows? Do you know why water is wet?
The world around you is full of puzzles. Prioritize, if you must. But do not complain that cruel Science has emptied the world of mystery. With reasoning such as that, I could get you to overlook an elephant in your living room.
September 04, 2007
Absurdity Heuristic, Absurdity Bias
Followup to: Stranger Than History, Robin's post What Evidence Ease of Imagination?
I've been pondering lately the notion of "absurdity" - wondering what exactly goes on in people's minds when they utter the adjective "absurd" or the objection "Absurd!"
If there is an absurdity heuristic, it would seem, at first glance, to be the mirror image of the well-known representativeness heuristic. The less X resembles Y, or the more X violates typicality assumptions of Y, the less probable that X is the product, explanation, or outcome of Y. A sequence of events is less probable when it involves an egg unscrambling itself, water flowing upward, machines thinking or dead people coming back to life. Since human psychology is not a pure structure of quantitative probabilities, it is easy to imagine that the absurdity heuristic is separate from the representativeness heuristic - implemented by separate absurdity-detecting brainware.
I suspect people may also be more sensitive to "absurdity" that
invalidates a plan or indicates cheating. Consider the
difference between "I saw a little blue man yesterday, walking down
the street" versus "I'm going to jump off this cliff and a little
blue man will catch me on the way down" or "If you give me your
wallet, a little blue man will bring you a pot of gold." (I'm
thinking, in particular, about how projections of future technology
are often met by the objection, "That's absurd!", and how the
objection seems more violent than usual in this case.)
As Robin observed, a
heuristic is not necessarily a bias. The vast majority of
objects do not fall upward. And yet helium balloons are an
exception. When are exceptions predictable?
I can think of three major circumstances where the absurdity heuristic gives rise to an absurdity bias:
The first case is when we have information about underlying laws which should override surface reasoning. If you know why most objects fall, and you can calculate how fast they fall, then your calculation that a helium balloon should rise at such-and-such a rate, ought to strictly override the absurdity of an object falling upward. If you can do deep calculations, you have no need for qualitative surface reasoning. But we may find it hard to attend to mere calculations in the face of surface absurdity, until we see the balloon rise.
(In 1913, Lee de Forest was accused of fraud for selling stock in an impossible endeavor, the Radio Telephone Company: "De Forest has said in many newspapers and over his signature that it would be possible to transmit human voice across the Atlantic before many years. Based on these absurd and deliberately misleading statements, the misguided public...has been persuaded to purchase stock in his company...")
The second case is a generalization of the first - attending to surface absurdity in the face of abstract information that ought to override it. If people cannot accept that studies show that marginal spending on medicine has zero net effect, because it seems absurd - violating the surface rule that "medicine cures" - then I would call this "absurdity bias". There are many reasons that people may fail to attend to abstract information or integrate it incorrectly. I think it worth distinguishing cases where the failure arises from absurdity detectors going off.
The third case is when the absurdity heuristic simply doesn't work - the process is not stable in its surface properties over the range of extrapolation - and yet people use it anyway. The future is usually "absurd" - it is unstable in its surface rules over fifty-year intervals.
This doesn't mean that anything can happen. Of all the events in the 20th century that would have been "absurd" by the standards of the 19th century, not a single one - to the best of our knowledge - violated the law of conservation of energy, which was known in 1850. Reality is not up for grabs; it works by rules even more precise than the ones we believe in instinctively.
The point is not that you can say anything you like about the future and no one can contradict you; but, rather, that the particular practice of crying "Absurd!" has historically been an extremely poor heuristic for predicting the future. Over the last few centuries, the absurdity heuristic has done worse than maximum entropy - ruled out the actual outcomes as being far too absurd to be considered. You would have been better off saying "I don't know".
September 06, 2007
Availability
The availability heuristic is judging the frequency or probability of an event, by the ease with which examples of the event come to mind.
A famous 1978 study by Lichtenstein, Slovic, Fischhoff, Layman, and Combs, "Judged Frequency of Lethal Events", studied errors in quantifying the severity of risks, or judging which of two dangers occurred more frequently. Subjects thought that accidents caused about as many deaths as disease; thought that homicide was a more frequent cause of death than suicide. Actually, diseases cause about 16 times as many deaths as accidents, and suicide is twice as frequent as homicide.
An obvious hypothesis to account for these skewed beliefs is that murders are more likely to be talked about than suicides - thus, someone is more likely to recall hearing about a murder than hearing about a suicide. Accidents are more dramatic than diseases - perhaps this makes people more likely to remember, or more likely to recall, an accident. In 1979, a followup study by Combs and Slovic showed that the skewed probability judgments correlated strongly (.85 and .89) with skewed reporting frequencies in two newspapers. This doesn't disentangle whether murders are more available to memory because they are more reported-on, or whether newspapers report more on murders because murders are more vivid (hence also more remembered). But either way, an availability bias is at work.
Selective reporting is one major source of availability biases. In the ancestral environment, much of what you knew, you experienced yourself; or you heard it directly from a fellow tribe-member who had seen it. There was usually at most one layer of selective reporting between you, and the event itself. With today's Internet, you may see reports that have passed through the hands of six bloggers on the way to you - six successive filters. Compared to our ancestors, we live in a larger world, in which far more happens, and far less of it reaches us - a much stronger selection effect, which can create much larger availability biases.
In real life, you're unlikely to ever meet Bill Gates. But thanks to selective reporting by the media, you may be tempted to compare your life success to his - and suffer hedonic penalties accordingly. The objective frequency of Bill Gates is 0.00000000015, but you hear about him much more often. Conversely, 19% of the planet lives on less than $1/day, and I doubt that one fifth of the blog posts you read are written by them.
Using availability seems to give rise to an absurdity bias; events that have never happened, are not recalled, and hence deemed to have probability zero. When no flooding has recently occurred (and yet the probabilities are still fairly calculable), people refuse to buy flood insurance even when it is heavily subsidized and priced far below an actuarially fair value. Kunreuther et. al. (1993) suggests underreaction to threats of flooding may arise from "the inability of individuals to conceptualize floods that have never occurred... Men on flood plains appear to be very much prisoners of their experience... Recently experienced floods appear to set an upward bound to the size of loss with which managers believe they ought to be concerned."
Burton et. al. (1978) report that when dams and levees are
built, they reduce the frequency of floods, and thus apparently
create a false sense of security, leading to reduced precautions.
While building dams decreases the frequency of floods,
damage per flood is afterward so much greater that average
yearly damage increases.
The wise would extrapolate from a memory of small hazards
to the possibility of large hazards. Instead, past experience
of small hazards seems to set a perceived upper bound on
risk. A society well-protected against minor hazards takes no
action against major risks, building on flood plains once the
regular minor floods are eliminated. A society subject to
regular minor hazards treats those minor hazards as an upper bound
on the size of the risks, guarding against regular minor floods but
not occasional major floods.
Memory is not always a good guide to probabilities in the past, let alone the future.
Burton, I., Kates, R. and White, G. 1978. Environment as Hazard. New York: Oxford University Press.
Combs, B. and Slovic, P. 1979. Causes of death: Biased newspaper coverage and biased judgments. Journalism Quarterly, 56: 837-843.
Kunreuther, H., Hogarth, R. and Meszaros, J. 1993. Insurer ambiguity and market failure. Journal of Risk and Uncertainty, 7: 71-87.
Lichtenstein, S., Slovic, P., Fischhoff, B., Layman, M. and Combs, B. 1978. Judged Frequency of Lethal Events. Journal of Experimental Psychology: Human Learning and Memory, 4(6), November: 551-78.
September 07, 2007
Why is the Future So Absurd?
Followup to: Stranger than History, Absurdity Heuristic / Absurdity Bias
Why is the future more absurd than people seem to expect? (That is: Why, historically, has the future so often turned out to be more "absurd" than people seem to have expected?)
One obvious reason is hindsight bias. Hindsight does not just cause people to severely underestimate how much they would have been surprised. Hindsight also leads people to overestimate how much attention they would have paid to the key factors, the factors that turned out to be important. As R. H. Tawney put it:
"Historians give an appearance of inevitability to an existing order by dragging into prominence the forces which have triumphed and thrusting into the background those which they have swallowed up."
When people look at historical changes and think "I could have predicted X" or "You could have predicted X if you looked at factors 1, 2, and 3"; then they forget that people did not, in fact, predict X, perhaps because they were distracted by factors 4 through 117. People read history books, see coherent narratives, and think that's how Time works. Underestimating the surprise of the present, they overestimate the predictability of the future.
I suspect that a major factor contributing to absurdity bias is that, when we look over history, we see changes away from absurd conditions such as everyone being a peasant farmer and women not having the vote, toward normal conditions like a majority middle class and equal rights. When people look at history, they see a series of normalizations. They learn the rule, "The future grows ever less absurd over time."
Perhaps one way to comprehend the bizarreness of the future would be to try and imagine historical changes occurring in reverse - how absurd would it be if all your electrical appliances suddenly disappeared, or you were transformed into a peasant farmer? Even if the future is nicer than the past, it will feel at least that absurd.
The correspondence bias of social psychology may also play a role in how we fail to learn from history - or so my own experience suggests. When we read about the strange behaviors of people in other eras, we may see them as people with a disposition to that strange behavior, rather than properly comprehending the strangeness of the times. In the 16th century, one popular entertainment was setting a cat on fire. If you think to yourself "What horrible people they must be!" then you have, to the same extent, diminished your appreciation of what horrible times they lived in.
We see at least some social and technological changes during our own lifetime. We do have some experience of genuine future shock. Why wouldn't this be enough to extrapolate forward?
According to Ray Kurzweil's thesis of accelerating change, our intuitions about the future are linear - we expect around as much change as occurred in the past - but technological change feeds on itself, and therefore has a positive second derivative. We should expect more technological change in the future than we have seen in the past, and insofar as technology drives cultural change, we should expect more cultural change too.
Or that, in my opinion, is the strongest version of Kurzweil's theory that can be put forward. Kurzweil dwells on Moore's Law and smoothly predictable exponential curves, but this seems to me both iffy and unnecessary. A curve does not need to be smooth or exponential to have a positive second derivative. And our cultural sensitivity to, say, computing power, is probably logarithmic anyway, obeying Weber's Law - a 20% increase in computing power probably feels the same whether it's from 1MHz to 1.2MHz, or 2GHz to 2.4GHz. In which case, people extrapolating the future "linearly" should get it pretty much correct.
But if you pull back and view the last few millennia, not just the last few decades, the strength of the core idea becomes obvious - technology change does feed on itself and therefore does speed up.
I would actually question Kurzweil's assertion that people extrapolate the past linearly into the future. Kurzweil may be too optimistic here. As discussed earlier, dwellers on flood plains do not extrapolate from small floods to large floods; instead, small floods set a perceived upper bound on risk. I suspect that when people try to visualize the strangeness of the future, they focus on a single possible change, of no greater magnitude than the largest single change they remember in their own lifetime.
The real future is not composed of single developments, but many developments together. Even if one change can pass the futurism filter, to suppose three absurdities simultaneously - never mind twenty - would entirely overload the absurdity meter. This may also explain why future projections get wronger and wronger as they go further out. People seem to imagine futures that are minimally counterintuitive, with one or two interesting changes to make a good story, rather than a realistic number of changes that would overload their extrapolation abilities.
What other biases could lead us to underestimate the absurdity of the future?
September 07, 2007
Anchoring and Adjustment
Suppose I spin a Wheel of Fortune device as you watch, and it comes up pointing to 65. Then I ask: Do you think the percentage of African countries in the UN is above or below this number? What do you think is the percentage of African countries in the UN? Take a moment to consider these two questions yourself, if you like, and please don't Google.
Also, try to guess, within 5 seconds, the value of the following arithmetical expression. 5 seconds. Ready? Set... Go!
1 x 2 x 3 x 4 x 5 x 6 x 7 x 8
Tversky and Kahneman (1974) recorded the estimates of subjects who saw the Wheel of Fortune showing various numbers. The median estimate of subjects who saw the wheel show 65 was 45%; the median estimate of subjects who saw 10 was 25%.
The current theory for this and similar experiments is that subjects take the initial, uninformative number as their starting point or anchor; and then they adjust upward or downward from their starting estimate until they reached an answer that "sounded plausible"; and then they stopped adjusting. This typically results in under-adjustment from the anchor - more distant numbers could also be "plausible", but one stops at the first satisfying-sounding answer.
Similarly, students shown "1 x 2 x 3 x 4 x 5 x 6 x 7 x 8" made a median estimate of 512, while students shown "8 x 7 x 6 x 5 x 4 x 3 x 2 x 1" made a median estimate of 2,250. The motivating hypothesis was that students would try to multiply (or guess-combine) the first few factors of the product, then adjust upward. In both cases the adjustments were insufficient, relative to the true value of 40,320; but the first set of guesses were much more insufficient because they started from a lower anchor.
Tversky and Kahneman report that offering payoffs for accuracy did not reduce the anchoring effect.
Strack and Mussweiler (1997) asked for the year Einstein first visited the United States. Completely implausible anchors, such as 1215 or 1992, produced anchoring effects just as large as more plausible anchors such as 1905 or 1939.
There are obvious applications in, say, salary negotiations, or buying a car. I won't suggest that you exploit it, but watch out for exploiters.
And: Watch yourself thinking, and try to notice when you are adjusting a figure in search of an estimate.
Debiasing manipulations for anchoring have generally proved not very effective. I would suggest these two: First, if the initial guess sounds implausible, try to throw it away entirely and come up with a new estimate, rather than sliding from the anchor. But this in itself may not be sufficient - subjects instructed to avoid anchoring still seem to do so (Quattrone et. al. 1981). So second, even if you are trying the first method, try also to think of an anchor in the opposite direction - an anchor that is clearly too small or too large, instead of too large or too small - and dwell on it briefly.
Quattrone, G.A., Lawrence, C.P., Finkel, S.E., & Andrus, D.C. (1981). Explorations in anchoring: The effects of prior range, anchor extremity, and suggestive hints. Manuscript, Stanford University.
Strack, F. & Mussweiler, T. (1997). Explaining the enigmatic anchoring effect: Mechanisms of selective accessibility. Journal of Personality and Social Psychology, 73, 437-446.
Tversky, A. and Kahneman, D. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185:1124--1131.
September 08, 2007
The Crackpot Offer
When I was very young - I think thirteen or maybe fourteen - I thought I had found a disproof of Cantor's Diagonal Argument, a famous theorem which demonstrates that the real numbers outnumber the rational numbers. Ah, the dreams of fame and glory that danced in my head!
My idea was that since each whole number can be decomposed into a bag of powers of 2, it was possible to map the whole numbers onto the set of subsets of whole numbers simply by writing out the binary expansion. 13, for example, 1101, would map onto {0, 2, 3}. It took a whole week before it occurred to me that perhaps I should apply Cantor's Diagonal Argument to my clever construction, and of course it found a counterexample - the binary number ...1111, which does not correspond to any finite whole number.
So I found this counterexample, and saw that my attempted disproof was false, along with my dreams of fame and glory.
I was initially a bit disappointed.
The thought went through my mind: "I'll get that theorem eventually! Someday I'll disprove Cantor's Diagonal Argument, even though my first try failed!" I resented the theorem for being obstinately true, for depriving me of my fame and fortune, and I began to look for other disproofs.
And then I realized something. I realized that I had made a mistake, and that, now that I'd spotted my mistake, there was absolutely no reason to suspect the strength of Cantor's Diagonal Argument any more than other major theorems of mathematics.
I saw then very clearly that I was being offered the opportunity to become a math crank, and to spend the rest of my life writing angry letters in green ink to math professors. (I'd read a book once about math cranks.)
I did not wish this to be my future, so I gave a small laugh, and let it go. I waved Cantor's Diagonal Argument on with all good wishes, and I did not question it again.
And I don't remember, now, if I thought this at the time, or if I thought it afterward... but what a terribly unfair test to visit upon a child of thirteen. That I had to be that rational, already, at that age, or fail.
The smarter you are, the younger you may be, the first time you have what looks to you like a really revolutionary idea. I was lucky in that I saw the mistake myself; that it did not take another mathematician to point it out to me, and perhaps give me an outside source to blame. I was lucky in that the disproof was simple enough for me to understand. Maybe I would have recovered eventually, otherwise. I've recovered from much worse, as an adult. But if I had gone wrong that early, would I ever have developed that skill?
I wonder how many people writing angry letters in green ink were thirteen when they made that first fatal misstep. I wonder how many were promising minds before then.
I made a mistake. That was all. I was not really right, deep down; I did not win a moral victory; I was not displaying ambition or skepticism or any other wondrous virtue; it was not a reasonable error; I was not half right or even the tiniest fraction right. I thought a thought I would never have thought if I had been wiser, and that was all there ever was to it.
If I had been unable to admit this to myself, if I had reinterpreted my mistake as virtuous, if I had insisted on being at least a little right for the sake of pride, then I would not have let go. I would have gone on looking for a flaw in the Diagonal Argument. And, sooner or later, I might have found one.
Until you admit you were wrong, you cannot get on with your life; your self-image will still be bound to the old mistake.
Whenever you are tempted to hold on to a thought you would never have thought if you had been wiser, you are being offered the opportunity to become a crackpot - even if you never write any angry letters in green ink. If no one bothers to argue with you, or if you never tell anyone your idea, you may still be a crackpot. It's the clinging that defines it.
It's not true. It's not true deep down. It's not
half-true or even a little true. It's nothing but a thought
you should never have thought. Not every cloud has a silver
lining. Human beings make mistakes, and not all of them are
disguised successes. Human beings make mistakes; it happens,
that's all. Say "oops", and get on with your
life.
September 10, 2007
Radical Honesty
I recently ran across this interesting article about Radical Honesty, a movement founded by a psychotherapist named Brad Blanton who suggests that we should kick our addiction to lying and just tell the complete truth all the time. I also like this quote from the Wikipedia article on Radical Honesty: "The significant majority of participants in the Radical Honesty workshops report dramatic changes in their lives after taking the course, though they are not always comfortable and positive." The movement visibly suffers from having been founded by a psychotherapist - it's more about the amazing happiness that absolute truth-telling can bring to your relationships (!!) rather than such rationalist values as seeking truth by teaching yourself a habit of honesty, or not wishing to deceive others because it infringes on their autonomy.
I once suggested a notion called "Crocker's Rules", which was
the mirror image of Radical Honesty - rather than telling the whole
truth to other people, you would strive to always allow others to
tell you the complete truth without being offended.
Crocker's Rules didn't give you the right to say anything offensive, but other people could say potentially offensive things to you, and it was your responsibility not to be offended. This was surprisingly hard to explain to people; many people would read the careful explanation and hear, "Crocker's Rules mean you can say offensive things to other people."
I was initially a bit suspicious of Blanton's movement - it seemed like the mirror-image that so many people misinterpreted, the option of saying offensive things to other people. But Blanton makes it not only optional, but mandatory to speak your mind - a far greater inconvenience than Crocker's Rules would ever impose on anyone.
Crocker's Rules didn't catch on. Maybe it was too hard to tell the difference between someone delivering a slap in the face, and someone deliberately invoking Crocker's Rules - you don't want to miss a real clue to real hostility because of your acceptance; you wouldn't want to not believe a true fact, even if the true fact is that someone else hates you. And third parties may assume the truthteller is an offensive person no matter how much the receiver disclaims offense - they may assume the receiver is "just being polite", or that requesting honesty does not excuse its offensiveness.
Will Blanton's Rules ever catch on? I worry that Radical Honesty would selectively disadvantage rationalists in human relationships. Broadcasting your opinions is much easier when you can deceive yourself about anything you'd feel uncomfortable saying to others. I wonder whether practitioners of Radical Honesty tend to become more adept at self-deception, as they stop being able to tell white lies or admit private thoughts to themselves. I have taken a less restrictive kind of honesty upon myself - to avoid statements that are literally false - and I know that this becomes more and more difficult, more and more of a disadvantage, as I deceive myself less and less.
I suspect that the neural circuits that we use to lie to others, also censor our own thoughts. Honesty to others is important unto a rationalist, even one who is seeking a strictly selfish advantage in finding truth only for themselves. If there were a Bayesian Order, would its practitioners take a vow of Radical Honesty?
I think that if there is ever a vow of honesty among rationalists, it will be restricted in scope. Normally, perhaps, you would avoid making statements that were literally false, and be ready to accept brutal honesty from anyone who first said "Crocker's Rules". Maybe you would be Radically Honest, but only with others who had taken a vow of Radical Honesty, and who understood the trust required to tell someone the truth.
Maybe Radical Honesty would be reserved for matters sacred unto a rationalist? In some domains this is already the case. We believe that scientists should always tell the whole truth about science. It's one thing to lie in everyday life, lie to your boss, lie to the police, lie to your lover; but whoever lies in a journal article is guilty of utter heresy and will be excommunicated.
I wonder what it would be like to have anyone in the world, even a single person, who you could absolutely trust. Or what it would be like for there to be anyone in the world, even a single person, whom you had to tell all your thoughts, without possibility of concealment.
September 10, 2007
We Don't Really Want Your Participation
At the Singularity Summit yesterday, several speakers alleged that we should "reach out" to artists and poets to encourage their participation in the Singularity dialogue. So at the end of one such session, a woman went up to the audience microphone and said:
"I am an artist. I want to participate. What should I do?"
And there was a brief, frozen silence.
I wanted to leap up and say:
No, no, I'm afraid you've misunderstood. We're just calling for greater participation by artists. We can get plenty of credit for being enlightened just by issuing the call. If we really cared what artists thought, we would find some artists and ask them questions, not call for artists to participate. We don't actually want to hear from artists. We think your opinions are stupid.
And if she'd asked me afterward, my real answer would have been:
You are not an artist, you are a human being; art is only one facet in which you express your humanity. Your reactions to the Singularity should arise from your entire self. It's perfectly all right to have a boringly normal and nonunique reaction like "I'm afraid," or "I don't think we should do this," or "I want to help, where do I send the check?" The right answer is not always unusual. Your natural reaction does not need to be unique, and that's why you don't need to try to come up with an "artist's viewpoint" on the Singularity. I would call on you to participate as a human being, not an artist. If your artistry has something to say, it will express itself naturally in your responses, without you needing to make a conscious effort to say something artist-like.
But I didn't say any of this, of course. It would have been indecorous.
And while we're on the subject, I would feel rather patronized -
like a dog commanded to perform a trick - if someone presented me
with a painting and said, "Say something mathematical!"
September 11, 2007
Applause Lights
Followup to: Semantic Stopsigns, We Don't Really Want Your Participation
At the Singularity Summit 2007, one of the speakers called for democratic, multinational development of AI. So I stepped up to the microphone and asked:
Suppose that a group of democratic republics form a consortium to develop AI, and there's a lot of politicking during the process - some interest groups have unusually large influence, others get shafted - in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world - dropping cellphones to anyone who doesn't have them - and do whatever the majority says. Which of these do you think is more "democratic", and would you feel safe with either?
I wanted to find out whether he believed in the pragmatic adequacy of the democratic political process, or if he believed in the moral rightness of voting. But the speaker replied:
The first scenario sounds like an editorial in Reason magazine, and the second sounds like a Hollywood movie plot.
Confused, I asked:
Then what kind of democratic process did you have in mind?
The speaker replied:
Something like the Human Genome Project - that was an internationally sponsored research project.
I asked:
How would different interest groups resolve their conflicts in a structure like the Human Genome Project?
And the speaker said:
I don't know.
This exchange puts me in mind of a quote
(which I failed to Google found by Jeff Grey and Miguel)
from some dictator or other, who was asked if he had any intentions
to move his pet state toward democracy:
We believe we are already within a democratic system. Some factors are still missing, like the expression of the people's will.
The substance of a democracy is the specific mechanism that resolves policy conflicts. If all groups had the same preferred policies, there would be no need for democracy - we would automatically cooperate. The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an AI, but it has to be something. What does it mean to call for a "democratic" solution if you don't have a conflict-resolution mechanism in mind?
I think it means that you have said the word "democracy", so the audience is supposed to cheer. It's not so much a propositional statement, as the equivalent of the "Applause" light that tells a studio audience when to clap.
This case is remarkable only in that I mistook the applause light for a policy suggestion, with subsequent embarrassment for all. Most applause lights are much more blatant, and can be detected by a simple reversal test. For example, suppose someone says:
We need to balance the risks and opportunities of AI.
If you reverse this statement, you get:
We shouldn't balance the risks and opportunities of AI.
Since the reversal sounds abnormal, the unreversed statement is probably normal, implying it does not convey new information. There are plenty of legitimate reasons for uttering a sentence that would be uninformative in isolation. "We need to balance the risks and opportunities of AI" can introduce a discussion topic; it can emphasize the importance of a specific proposal for balancing; it can criticize an unbalanced proposal. Linking to a normal assertion can convey new information to a bounded rationalist - the link itself may not be obvious. But if no specifics follow, the sentence is probably an applause light.
I am tempted to give a talk sometime that consists of nothing but applause lights, and see how long it takes for the audience to start laughing:
I am here to propose to you today that we need to balance the risks and opportunities of advanced Artificial Intelligence. We should avoid the risks and, insofar as it is possible, realize the opportunities. We should not needlessly confront entirely unnecessary dangers. To achieve these goals, we must plan wisely and rationally. We should not act in fear and panic, or give in to technophobia; but neither should we act in blind enthusiasm. We should respect the interests of all parties with a stake in the Singularity. We must try to ensure that the benefits of advanced technologies accrue to as many individuals as possible, rather than being restricted to a few. We must try to avoid, as much as possible, violent conflicts using these technologies; and we must prevent massive destructive capability from falling into the hands of individuals. We should think through these issues before, not after, it is too late to do anything about them...
September 12, 2007
Rationality and the English Language
Yesterday, someone said that my writing reminded them of George Orwell's Politics and the English Language. I was honored. Especially since I'd already thought of today's topic.
If you really want an artist's perspective on rationality, then read Orwell; he is mandatory reading for rationalists as well as authors. Orwell was not a scientist, but a writer; his tools were not numbers, but words; his adversary was not Nature, but human evil. If you wish to imprison people for years without trial, you must think of some other way to say it than "I'm going to imprison Mr. Jennings for years without trial." You must muddy the listener's thinking, prevent clear images from outraging conscience. You say, "Unreliable elements were subjected to an alternative justice process."
Orwell was the outraged opponent of totalitarianism and the muddy thinking in which evil cloaks itself - which is how Orwell's writings on language ended up as classic rationalist documents on a level with Feynman, Sagan, or Dawkins.
"Writers are told to avoid usage of the passive voice." A rationalist whose background comes exclusively from science, may fail to see the flaw in the previous sentence; but anyone who's done a little writing should see it right away. I wrote the sentence in the passive voice, without telling you who tells authors to avoid passive voice. Passive voice removes the actor, leaving only the acted-upon. "Unreliable elements were subjected to an alternative justice process" - subjected by who? What does an "alternative justice process" do? With enough static noun phrases, you can keep anything unpleasant from actually happening.
Journal articles are often written in passive voice. (Pardon me, some scientists write their journal articles in passive voice. It's not as if the articles are being written by no one, with no one to blame.) It sounds more authoritative to say "The subjects were administered Progenitorivox" than "I gave each college student a bottle of 20 Progenitorivox, and told them to take one every night until they were gone." If you remove the scientist from the description, that leaves only the all-important data. But in reality the scientist is there, and the subjects are college students, and the Progenitorivox wasn't "administered" but handed over with instructions. Passive voice obscures reality.
Judging from the comments I get on Overcoming Bias, someone will protest that using the passive voice in a journal article is hardly a sin - after all, if you think about it, you can realize the scientist is there. It doesn't seem like a logical flaw. And this is why rationalists need to read Orwell, not just Feynman or even Jaynes.
Nonfiction conveys knowledge, fiction conveys experience. Medical science can extrapolate what would happen to a human unprotected in a vacuum. Fiction can make you live through it.
Some rationalists will try to analyze a misleading phrase, try to see if there might possibly be anything meaningful to it, try to construct a logical interpretation. They will be charitable, give the author the benefit of the doubt. Authors, on the other hand, are trained not to give themselves the benefit of the doubt. Whatever the audience thinks you said is what you said, whether you meant to say it or not; you can't argue with the audience no matter how clever your justifications.
A writer knows that readers will not stop for a minute to think. A fictional experience is a continuous stream of first impressions. A writer-rationalist pays attention to the experience words create. If you are evaluating the public rationality of a statement, and you analyze the words deliberatively, rephrasing propositions, trying out different meanings, searching for nuggets of truthiness, then you're losing track of the first impression - what the audience sees, or rather feels.
A novelist would notice the screaming wrongness of "The subjects were administered Progenitorivox." What life is here for a reader to live? This sentence creates a distant feeling of authoritativeness, and that's all - the only experience is the feeling of being told something reliable. A novelist would see nouns too abstract to show what actually happened - the postdoc with the bottle in his hand, trying to look stern; the student listening with a nervous grin.
My point is not to say that journal articles should be written like novels, but that a rationalist should become consciously aware of the experiences which words create. A rationalist must understand the mind and how to operate it. That includes the stream of consciousness, the part of yourself that unfolds in language. A rationalist must become consciously aware of the actual, experiential impact of phrases, beyond their mere propositional semantics.
Or to say it more bluntly: Meaning does not excuse impact!
I don't care what rational interpretation you can construct for an applause light like "AI should be developed through democratic processes". That cannot excuse its irrational impact of signaling the audience to applaud, not to mention its cloudy question-begging vagueness.
Here is Orwell, railing against the impact of cliches, their effect on the experience of thinking:
When one watches some tired hack on the platform mechanically repeating the familiar phrases -- bestial, atrocities, iron heel, bloodstained tyranny, free peoples of the world, stand shoulder to shoulder -- one often has a curious feeling that one is not watching a live human being but some kind of dummy... A speaker who uses that kind of phraseology has gone some distance toward turning himself into a machine. The appropriate noises are coming out of his larynx, but his brain is not involved, as it would be if he were choosing his words for himself...
What is above all needed is to let the meaning choose the word, and not the other way around. In prose, the worst thing one can do with words is surrender to them. When you think of a concrete object, you think wordlessly, and then, if you want to describe the thing you have been visualising you probably hunt about until you find the exact words that seem to fit it. When you think of something abstract you are more inclined to use words from the start, and unless you make a conscious effort to prevent it, the existing dialect will come rushing in and do the job for you, at the expense of blurring or even changing your meaning. Probably it is better to put off using words as long as possible and get one's meaning as clear as one can through pictures and sensations.
Peirce might have written that last paragraph. More than one path can lead to the Way.
September 13, 2007
Human Evil and Muddled Thinking
Followup to: Rationality and the English Language
George Orwell saw the descent of the civilized world into totalitarianism, the conversion or corruption of one country after another; the boot stamping on a human face, forever, and remember that it is forever. You were born too late to remember a time when the rise of totalitarianism seemed unstoppable, when one country after another fell to secret police and the thunderous knock at midnight, while the professors of free universities hailed the Soviet Union's purges as progress. It feels as alien to you as fiction; it is hard for you to take seriously. Because, in your branch of time, the Berlin Wall fell. And if Orwell's name is not carved into one of those stones, it should be.
Orwell saw the destiny of the human species, and he put forth a convulsive effort to wrench it off its path. Orwell's weapon was clear writing. Orwell knew that muddled language is muddled thinking; he knew that human evil and muddled thinking intertwine like conjugate strands of DNA:
In our time, political speech and writing are largely the defence of the indefensible. Things like the continuance of British rule in India, the Russian purges and deportations, the dropping of the atom bombs on Japan, can indeed be defended, but only by arguments which are too brutal for most people to face, and which do not square with the professed aims of the political parties. Thus political language has to consist largely of euphemism, question-begging and sheer cloudy vagueness. Defenceless villages are bombarded from the air, the inhabitants driven out into the countryside, the cattle machine-gunned, the huts set on fire with incendiary bullets: this is called pacification...
Orwell was clear on the goal of his clarity:
If you simplify your English, you are freed from the worst follies of orthodoxy. You cannot speak any of the necessary dialects, and when you make a stupid remark its stupidity will be obvious, even to yourself.
To make our stupidity obvious, even to ourselves - this is the heart of Overcoming Bias.
Evil sneaks, hidden, through the unlit shadows of the mind. We look back with the clarity of history, and weep to remember the planned famines of Stalin and Mao, which killed tens of millions. We call this evil, because it was done by deliberate human intent to inflict pain and death upon innocent human beings. We call this evil, because of the revulsion that we feel against it, looking back with the clarity of history. For perpetrators of evil to avoid its natural opposition, the revulsion must remain latent. Clarity must be avoided at any cost. Even as humans of clear sight tend to oppose the evil that they see; so too does human evil, wherever it exists, set out to muddle thinking.
1984 sets this forth starkly: Orwell's ultimate villains are cutters and airbrushers of photographs (based on historical cutting and airbrushing in the Soviet Union). At the peak of all darkness in the Ministry of Love, O'Brien tortures Winston to admit that two plus two equals five:
'Do you remember,' he went on, 'writing in your diary, "Freedom is the freedom to say that two plus two make four"?'
'Yes,' said Winston.
O'Brien held up his left hand, its back towards Winston, with the thumb hidden and the four fingers extended.
'How many fingers am I holding up, Winston?'
'Four.'
'And if the party says that it is not four but five -- then how many?'
'Four.'
The word ended in a gasp of pain. The needle of the dial had shot up to fifty-five. The sweat had sprung out all over Winston's body. The air tore into his lungs and issued again in deep groans which even by clenching his teeth he could not stop. O'Brien watched him, the four fingers still extended. He drew back the lever. This time the pain was only slightly eased.
I am continually aghast at apparently intelligent folks - such as Robin's colleague Tyler Cowen - who don't think that overcoming bias is important. This is your mind we're talking about. Your human intelligence. It separates you from an ape. It built this world. You don't think how the mind works is important? You don't think the mind's systematic malfunctions are important? Do you think the Inquisition would have tortured witches, if all were ideal Bayesians?
Tyler Cowen apparently feels that overcoming bias is just as biased as bias: "I view Robin's blog as exemplifying bias, and indeed showing that bias can be very useful." I hope this is only the result of thinking too abstractly while trying to sound clever. Does Tyler seriously think that scope insensitivity to the value of human life is on the same level with trying to create plans that will really save as many lives as possible?
Orwell was forced to fight a similar attitude - that to admit to any distinction is youthful naïveté:
Stuart Chase and others have come near to claiming that all abstract words are meaningless, and have used this as a pretext for advocating a kind of political quietism. Since you don't know what Fascism is, how can you struggle against Fascism?
Maybe overcoming bias doesn't look quite exciting enough, if it's framed as a struggle against mere accidental mistakes. Maybe it's harder to get excited if there isn't some clear evil to oppose. So let us be absolutely clear that where there is human evil in the world, where there is cruelty and torture and deliberate murder, there are biases enshrouding it. Where people of clear sight oppose these biases, the concealed evil fights back. The truth does have enemies. If Overcoming Bias were a newsletter in the old Soviet Union, every poster and commenter of this blog would have been shipped off to labor camps.
In all human history, every great leap forward has been driven by a new clarity of thought. Except for a few natural catastrophes, every great woe has been driven by a stupidity. Our last enemy is ourselves; and this is a war, and we are soldiers.
September 14, 2007
Doublethink (Choosing to be Biased)
An oblong slip of newspaper had appeared between O'Brien's fingers. For perhaps five seconds it was within the angle of Winston's vision. It was a photograph, and there was no question of its identity. It was the photograph. It was another copy of the photograph of Jones, Aaronson, and Rutherford at the party function in New York, which he had chanced upon eleven years ago and promptly destroyed. For only an instant it was before his eyes, then it was out of sight again. But he had seen it, unquestionably he had seen it! He made a desperate, agonizing effort to wrench the top half of his body free. It was impossible to move so much as a centimetre in any direction. For the moment he had even forgotten the dial. All he wanted was to hold the photograph in his fingers again, or at least to see it.
'It exists!' he cried.
'No,' said O'Brien.
He stepped across the room.
There was a memory hole in the opposite wall. O'Brien lifted the grating. Unseen, the frail slip of paper was whirling away on the current of warm air; it was vanishing in a flash of flame. O'Brien turned away from the wall.
'Ashes,' he said. 'Not even identifiable ashes. Dust. It does not exist. It never existed.'
'But it did exist! It does exist! It exists in memory. I remember it. You remember it.'
'I do not remember it,' said O'Brien.
Winston's heart sank. That was doublethink. He had a feeling of deadly helplessness. If he could have been certain that O'Brien was lying, it would not have seemed to matter. But it was perfectly possible that O'Brien had really forgotten the photograph. And if so, then already he would have forgotten his denial of remembering it, and forgotten the act of forgetting. How could one be sure that it was simple trickery? Perhaps that lunatic dislocation in the mind could really happen: that was the thought that defeated him.
-- George Orwell, 1984
What if self-deception helps us be happy? What if just running out and overcoming bias will make us - gasp! - unhappy? Surely, true wisdom would be second-order rationality, choosing when to be rational. That way you can decide which cognitive biases should govern you, to maximize your happiness.
Leaving the morality aside, I doubt such a lunatic dislocation in the mind could really happen.
Second-order rationality implies that at some point, you will think to yourself, "And now, I will irrationally believe that I will win the lottery, in order to make myself happy." But we do not have such direct control over our beliefs. You cannot make yourself believe the sky is green by an act of will. You might be able to believe you believed it - though I have just made that more difficult for you by pointing out the difference. (You're welcome!) You might even believe you were happy and self-deceived; but you would not in fact be happy and self-deceived.
For second-order rationality to be genuinely rational, you would first need a good model of reality, to extrapolate the consequences of rationality and irrationality. If you then chose to be first-order irrational, you would need to forget this accurate view. And then forget the act of forgetting. I don't mean to commit the logical fallacy of generalizing from fictional evidence, but I think Orwell did a good job of extrapolating where this path leads.
You can't know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception.
The other alternative is to choose blindly to remain biased, without any clear idea of the consequences. This is not second-order rationality. It is willful stupidity.
Be irrationally optimistic about your driving skills, and you
will be happily unconcerned where others sweat and fear. You
won't have to put up with the inconvenience of a seatbelt.
You will be happily unconcerned for a day, a week, a year.
Then CRASH, and spend the rest of your life wishing you
could scratch the itch in your phantom limb. Or paralyzed
from the neck down. Or dead. It's not inevitable, but
it's possible; how probable is it? You can't make that
tradeoff rationally unless you know your real driving
skills, so you can figure out how much danger you're placing
yourself in. You can't make that tradeoff rationally unless
you know about biases like neglect of
probability.
No matter how many days go by in blissful ignorance, it only takes a single mistake to undo a human life, to outweigh every penny you picked up from the railroad tracks of stupidity.
One of chief pieces of advice I give to aspiring rationalists is "Don't try to be clever." And, "Listen to those quiet, nagging doubts." If you don't know, you don't know what you don't know, you don't know how much you don't know, and you don't know how much you needed to know.
There is no second-order rationality. There is only a
blind leap into what may or may not be a flaming lava pit.
Once you know, it will be too late for blindness.
But people neglect this, because they do not know what they do not know. Unknown unknowns are not available. They do not focus on the blank area on the map, but treat it as if it corresponded to a blank territory. When they consider leaping blindly, they check their memory for dangers, and find no flaming lava pits in the blank map. Why not leap?
Been there. Tried that. Got burned. Don't try
to be clever.
I once said to a friend that I suspected the happiness of stupidity was greatly overrated. And she shook her head seriously, and said, "No, it's not; it's really not."
Maybe there are stupid happy people out there. Maybe they are happier than you are. And life isn't fair, and you won't become happier by being jealous of what you can't have. I suspect the vast majority of Overcoming Bias readers could not achieve the "happiness of stupidity" if they tried. That way is closed to you. You can never achieve that degree of ignorance, you cannot forget what you know, you cannot unsee what you see.
The happiness of stupidity is closed to you. You will never have it short of actual brain damage, and maybe not even then. You should wonder, I think, whether the happiness of stupidity is optimal - if it is the most happiness that a human can aspire to - but it matters not. That way is closed to you, if it was ever open.
All that is left to you now, is to aspire to such happiness as a rationalist can achieve. I think it may prove greater, in the end. There are bounded paths and open-ended paths; plateaus on which to laze, and mountains to climb; and if climbing takes more effort, still the mountain rises higher in the end.
Also there is more to life than happiness; and other happinesses
than your own may be at stake in your decisions.
But that is moot. By the time you realize you have a choice, there is no choice. You cannot unsee what you see. The other way is closed.
September 15, 2007
Why I'm Blooking
Yesterday being my 100th Overcoming Bias post, it seems an opportune time to answer a commenter's question: Why am I posting?
For a long time I've suffered from writer's molasses. Like writer's block, only instead of not writing, I write very slooowly. At least when it comes to writing Documents - papers, book chapters, website material. If I haven't published a hundred papers, it's not for lack of a hundred ideas, but because writing one paper - at my current pace - takes four months full time. I sometimes wonder if I could become a respectable academic if I wrote at a respectable pace.
Oddly enough, I can write most emails around as fast as I type. Such disorders are hard to self-diagnose, but I suspect that part of the problem is that on Documents I repeatedly reread and tweak material I've already written, instead of writing new material. James Hogan (an SF author) once told me that he was more productive on a typewriter than a word processor, because the typewriter prevented him from tweaking until the second draft.
A blook is a collection of blog posts that have been edited into a book. Logically, then, publishing a book as a series of blog posts ought to be known as "blooking".
It would be more precise to say that I'm generating raw material to be edited into a book, and collecting some feedback along the way. I make no promises for this project. (I hate promising anything unless I have already done it.) The first part of the plan, generating the raw material as blog posts, has proceeded at a respectable pace so far. Will I be able to edit the posts into chapters, so long as all the raw material is there? Will I be able to generate all the raw material, or will the project, ahem, "blog down"?
In August I decided that I was going to write one blog post per day for Overcoming Bias. This challenge began to hone my writing speed somewhat - for example, I would look at the clock and try not to take longer than an hour... or three hours... but nonetheless I began to feel the need to shove the post out the door instead of perfecting it further. This is necessary and proper.
Near the end of August, I faced a new challenge - I also had to prepare two talks for the Singularity Summit 2007 (Sep 8-9). Those were actual Documents. I knew, from previous experience, that I couldn't possibly prepare the two talks and also keep up the pace of blogging on Overcoming Bias. Blogging was using up all my writing energy already - I have only a limited supply of words per day. If I overreach one day's budget I can't write at all the next day. So (I knew) I would have to temporarily stop blogging and resume after the Summit.
And then I said to myself, Hey, if I never try to do anything "impossible", I'll never grow.
I decided I would keep up the pace on Overcoming Bias while simultaneously writing my two Summit talks. Tsuyoku naritai!
I lost sleep, and skipped exercise. But I did it. I'll remember that the next time I'm thinking of trying something impossible.
September 17, 2007
Planning Fallacy
The Denver International Airport opened 16 months late, at a cost overrun of $2 billion (I've also seen $3.1 billion asserted). The Eurofighter Typhoon, a joint defense project of several European countries, was delivered 54 months late at a cost of £19 billion instead of £7 billion. The Sydney Opera House may be the most legendary construction overrun of all time, originally estimated to be completed in 1963 for $7 million, and finally completed in 1973 for $102 million.
Are these isolated disasters brought to our attention by selective availability? Are they symptoms of bureaucracy or government incentive failures? Yes, very probably. But there's also a corresponding cognitive bias, replicated in experiments with individual planners.
Buehler et. al. (1995) asked their students for estimates of when they (the students) thought they would complete their personal academic projects. Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done. Would you care to guess how many students finished on or before their estimated 50%, 75%, and 99% probability levels?
- 13% of subjects finished their project by the time they had assigned a 50% probability level;
- 19% finished by the time assigned a 75% probability level;
- and only 45% (less than half!) finished by the time of their 99% probability level.
As Buehler et. al. (2002) wrote, "The results for the 99% probability level are especially striking: Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students' confidence in their time estimates far exceeded their accomplishments."
More generally, this phenomenon is known as the "planning fallacy". The planning fallacy is that people think they can plan, ha ha.
A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et. al. (2000), who found that:
- Asking subjects for their predictions based on realistic "best guess" scenarios; or
- Asking subjects for their hoped-for "best case" scenarios...
...produced indistinguishable results.
When people are asked for a "realistic" scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes - the same vision as their "best case".
Reality, it turns out, usually delivers results somewhat worse than the "worst case".
Unlike most cognitive biases, we know a good debiasing heuristic for the planning fallacy. It won't work for messes on the scale of the Denver International Airport, but it'll work for a lot of personal planning, and even some small-scale organizational stuff. Just use an "outside view" instead of an "inside view".
People tend to generate their predictions by thinking about the particular, unique features of the task at hand, and constructing a scenario for how they intend to complete the task - which is just what we usually think of as planning. When you want to get something done, you have to plan out where, when, how; figure out how much time and how much resource is required; visualize the steps from beginning to successful conclusion. All this is the "inside view", and it doesn't take into account unexpected delays and unforeseen catastrophes. As we saw before, asking people to visualize the "worst case" still isn't enough to counteract their optimism - they don't visualize enough Murphyness.
The outside view is when you deliberately avoid thinking about the special, unique features of this project, and just ask how long it took to finish broadly similar projects in the past. This is counterintuitive, since the inside view has so much more detail - there's a temptation to think that a carefully tailored prediction, taking into account all available data, will give better results.
But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become. Buehler et. al. (2002) asked an experimental group of subjects to describe highly specific plans for their Christmas shopping - where, when, and how. On average, this group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas.
Likewise, Buehler et. al. (2002), reporting on a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline. They actually finished 1 day before deadline. Asked when they had previously completed similar tasks, they responded, "1 day before deadline." This is the power of the outside view over the inside view.
A similar finding is that experienced outsiders, who know less of the details, but who have relevant memory to draw upon, are often much less optimistic and much more accurate than the actual planners and implementers.
So there is a fairly reliable way to fix the planning fallacy, if you're doing something broadly similar to a reference class of previous projects. Just ask how long similar projects have taken in the past, without considering any of the special properties of this project. Better yet, ask an experienced outsider how long similar projects have taken.
You'll get back an answer that sounds hideously long, and clearly reflects no understanding of the special reasons why this particular task will take less time. This answer is true. Deal with it.
Buehler, R., Griffin, D. and Ross, M. 1994. Exploring the "planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67: 366-381.
Buehler, R., Griffin, D. and Ross, M. 1995. It's about time: Optimistic predictions in work and love. Pp. 1-32 in European Review of Social Psychology, Volume 6, eds. W. Stroebe and M. Hewstone. Chichester: John Wiley & Sons.
Buehler, R., Griffin, D. and Ross, M. 2002. Inside the planning fallacy: The causes and consequences of optimistic time predictions. Pp. 250-270 in Gilovich, T., Griffin, D. and Kahneman, D. (eds.) Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge, U.K.: Cambridge University Press.
Newby-Clark, I. R., Ross, M., Buehler, R., Koehler, D. J. and Griffin, D. 2000. People focus on optimistic and disregard pessimistic scenarios while predicting their task completion times. Journal of Experimental Psychology: Applied, 6: 171-182.
September 17, 2007
Kahneman's Planning Anecdote
Followup to: Planning Fallacy
From "Timid Choices and Bold Forecasts: Cognitive Perspective on Risk Taking" by Nobel Laureate Daniel Kahneman and Dan Lovallo, in a discussion on "Inside and Outside Views":
In 1976 one of us (Daniel Kahneman) was involved in a project designed to develop a curriculum for the study of judgment and decision making under uncertainty for high schools in Israel. When the team had been in operation for about a year, with some significant achievements already to its credit, the discussion at one of the team meetings turned to the question of how long the project would take. To make the debate more useful, I asked everyone to indicate on a slip of paper their best estimate of the number of months that would be needed to bring the project to a well-defined stage of completion: a complete draft ready for submission to the Ministry of education. The estimates, including my own, ranged from 18 to 30 months.
At this point I had the idea of turning to one of our members, a distinguished expert in curriculum development, asking him a question phrased about as follows:
"We are surely not the only team to have tried to develop a curriculum where none existed before. Please try to recall as many such cases as you can. Think of them as they were in a stage comparable to ours at present. How long did it take them, from that point, to complete their projects?"
After a long silence, something much like the following answer was given, with obvious signs of discomfort: "First, I should say that not all teams that I can think of in a comparable stage ever did complete their task. About 40% of them eventually gave up. Of the remaining, I cannot think of any that was completed in less than seven years, nor of any that took more than ten."
In response to a further question, he answered: "No, I cannot think of any relevant factor that distinguishes us favorably from the teams I have been thinking about. Indeed, my impression is that we are slightly below average in terms of our resources and potential."
Facing the facts can be intolerably demoralizing. The participants in the meeting had professional expertise in the logic of forecasting, and none even ventured to question the relevance of the forecast implied by our expert's statistics: an even chance of failure, and a completion time of seven to ten years in case of success. Neither of these outcomes was an acceptable basis for continuing the project, but no one was willing to draw the embarrassing conclusion that it should be scrapped.
So, the forecast was quietly dropped from active debate, along with any pretense of long-term planning, and the project went on along its predictably unforeseeable path to eventual completion some eight years later.
September 18, 2007
Conjunction Fallacy
The following experiment has been slightly modified for ease of blogging. You are given the following written description, which is assumed true:
Bill is 34 years old. He is intelligent, but unimaginative, compulsive, and generally lifeless. In school, he was strong in mathematics but weak in social studies and humanities.
No complaints about the description, please, this experiment was done in 1974. Anyway, we are interested in the probability of the following propositions, which may or may not be true, and are not mutually exclusive or exhaustive:
A: Bill is an accountant.
B: Bill is a physician who plays poker for a hobby.
C: Bill plays jazz for a hobby.
D: Bill is an architect.
E: Bill is an accountant who plays jazz for a hobby.
F: Bill climbs mountains for a hobby.
Take a moment before continuing to rank these six propositions by probability, starting with the most probable propositions and ending with the least probable propositions. Again, the starting description of Bill is assumed true, but the six propositions may be true or untrue (they are not additional evidence) and they are not assumed mutually exclusive or exhaustive.
In a very similar experiment conducted by Tversky and Kahneman (1982), 92% of 94 undergraduates at the University of British Columbia gave an ordering with A > E > C. That is, the vast majority of subjects indicated that Bill was more likely to be an accountant than an accountant who played jazz, and more likely to be an accountant who played jazz than a jazz player. The ranking E > C was also displayed by 83% of 32 grad students in the decision science program of Stanford Business School, all of whom had taken advanced courses in probability and statistics.
There is a certain logical problem with saying that Bill is more likely to be an account who plays jazz, than he is to play jazz. The conjunction rule of probability theory states that, for all X and Y, P(X&Y) <= P(Y). That is, the probability that X and Y are simultaneously true, is always less than or equal to the probability that Y is true. Violating this rule is called a conjunction fallacy.
Imagine a group of 100,000 people, all of whom fit Bill's description (except for the name, perhaps). If you take the subset of all these persons who play jazz, and the subset of all these persons who play jazz and are accountants, the second subset will always be smaller because it is strictly contained within the first subset.
Could the conjunction fallacy rest on students interpreting the experimental instructions in an unexpected way - misunderstanding, perhaps, what is meant by "probable"? Here's another experiment, Tversky and Kahneman (1983), played by 125 undergraduates at UBC and Stanford for real money:
Consider a regular six-sided die with four green faces and two red faces. The die will be rolled 20 times and the sequences of greens (G) and reds (R) will be recorded. You are asked to select one sequence, from a set of three, and you will win $25 if the sequence you chose appears on successive rolls of the die. Please check the sequence of greens and reds on which you prefer to bet.
1. RGRRR
2. GRGRRR
3. GRRRRR
65% of the subjects chose sequence 2, which is most representative of the die, since the die is mostly green and sequence 2 contains the greatest proportion of green rolls. However, sequence 1 dominates sequence 2, because sequence 1 is strictly included in 2. 2 is 1 preceded by a G; that is, 2 is the conjunction of an initial G with 1. This clears up possible misunderstandings of "probability", since the goal was simply to get the $25.
Another experiment from Tversky and Kahneman (1983) was conducted at the Second International Congress on Forecasting in July of 1982. The experimental subjects were 115 professional analysts, employed by industry, universities, or research institutes. Two different experimental groups were respectively asked to rate the probability of two different statements, each group seeing only one statement:
- "A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983."
- "A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983."
Estimates of probability were low for both statements, but significantly lower for the first group than the second (p < .01 by Mann-Whitney). Since each experimental group only saw one statement, there is no possibility that the first group interpreted (1) to mean "suspension but no invasion".
The moral? Adding more detail or extra assumptions can make an event seem more plausible, even though the event necessarily becomes less probable.
Do you have a favorite futurist? How many details do they tack onto their amazing, futuristic predictions?
Tversky, A. and Kahneman, D. 1982. Judgments of and by representativeness. Pp 84-98 in Kahneman, D., Slovic, P., and Tversky, A., eds. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.
Tversky, A. and Kahneman, D. 1983. Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90: 293-315.
September 19, 2007
Conjunction Controversy (Or, How They Nail It Down)
Followup to: Conjunction Fallacy
When a single experiment seems to show that subjects are guilty of some horrifying sinful bias - such as thinking that the proposition "Bill is an accountant who plays jazz" has a higher probability than "Bill is an accountant" - people may try to dismiss (not defy) the experimental data. Most commonly, by questioning whether the subjects interpreted the experimental instructions in some unexpected fashion - perhaps they misunderstood what you meant by "more probable".
Experiments are not beyond questioning; on the other hand, there should always exist some mountain of evidence which suffices to convince you. It's not impossible for researchers to make mistakes. It's also not impossible for experimental subjects to be really genuinely and truly biased. It happens. On both sides, it happens. We're all only human here.
If you think to extend a hand of charity toward experimental subjects, casting them in a better light, you should also consider thinking charitably of scientists. They're not stupid, you know. If you can see an alternative interpretation, they can see it too. This is especially important to keep in mind when you read about a bias and one or two illustrative experiments in a blog post. Yes, if the few experiments you saw were all the evidence, then indeed you might wonder. But you might also wonder if you're seeing all the evidence that supports the standard interpretation. Especially if the experiments have dates on them like "1982" and are prefaced with adjectives like "famous" or "classic".
So! This is a long post. It is a long post because nailing down a theory requires more experiments than the one or two vivid illustrations needed to merely explain. I am going to cite maybe one in twenty of the experiments that I've read about, which is maybe a hundredth of what's out there. For more information, see Tversky and Kahneman (1983) or Kahneman and Frederick (2002), both available online, from which this post is primarily drawn.
Here is (probably) the single most questioned experiment in the literature of heuristics and biases, which I reproduce here exactly as it appears in Tversky and Kahneman (1982):
Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Please rank the following statements by their probability, using 1 for the most probable and 8 for the least probable:
(5.2) Linda is a teacher in elementary school.
(3.3) Linda works in a bookstore and takes Yoga classes.
(2.1) Linda is active in the feminist movement. (F)
(3.1) Linda is a psychiatric social worker.
(5.4) Linda is a member of the League of Women Voters.
(6.2) Linda is a bank teller. (T)
(6.4) Linda is an insurance salesperson.
(4.1) Linda is a bank teller and is active in the feminist movement. (T & F)
(The numbers at the start of each line are the mean ranks of each proposition, lower being more probable.)
How do you know that subjects did not interpret "Linda is a bank teller" to mean "Linda is a bank teller and is not active in the feminist movement"? For one thing, dear readers, I offer the observation that most bank tellers, even the ones who participated in anti-nuclear demonstrations in college, are probably not active in the feminist movement. So, even so, Teller should rank above Teller & Feminist. You should be skeptical of your own objections, too; else it is disconfirmation bias. But the researchers did not stop with this observation; instead, in Tversky and Kahneman (1983), they created a between-subjects experiment in which either the conjunction or the two conjuncts were deleted. Thus, in the between-subjects version of the experiment, each subject saw either (T&F), or (T), but not both. With a total of five propositions ranked, the mean rank of (T&F) was 3.3 and the mean rank of (T) was 4.4, N=86. Thus, the fallacy is not due solely to interpreting "Linda is a bank teller" to mean "Linda is a bank teller and not active in the feminist movement."
Similarly, the experiment discussed yesterday used a between-subjects design (where each subject only saw one statement) to elicit higher probabilities for "A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983" versus "A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983".
Another way of knowing whether subjects have misinterpreted an experiment is to ask the subjects directly. Also in Tversky and Kahneman (1983), a total of 103 medical internists (including 37 internists taking a postgraduate course at Harvard, and 66 internists with admitting privileges at New England Medical Center) were given problems like the following:
A 55-year-old woman had pulmonary embolism documented angiographically 10 days after a cholecstectomy. Please rank order the following in terms of the probability that they will be among the conditions experienced by the patient (use 1 for the most likely and 6 for the least likely). Naturally, the patient could experience more than one of these conditions.
- Dyspnea and hemiparesis
- Calf pain
- Pleuritic chest pain
- Syncope and tachycardia
- Hemiparesis
- Hemoptysis
As Tversky and Kahneman note, "The symptoms listed for each problem included one, denoted B, that was judged by our consulting physicians to be nonrepresentative of the patient's condition, and the conjunction of B with another highly representative symptom denoted A. In the above example of pulmonary embolism (blood clots in the lung), dyspnea (shortness of breath) is a typical symptom, whereas hemiparesis (partial paralysis) is very atypical."
In indirect tests, the mean ranks of A&B and B respectively were 2.8 and 4.3; in direct tests, they were 2.7 and 4.6. In direct tests, subjects ranked A&B above B between 73% to 100% of the time, with an average of 91%.
The experiment was designed to eliminate, in four ways, the possibility that subjects were interpreting B to mean "only B (and not A)". First, carefully wording the instructions: "...the probability that they will be among the conditions experienced by the patient", plus an explicit reminder, "the patient could experience more than one of these conditions". Second, by including indirect tests as a comparison. Third, the researchers afterward administered a questionnaire:
In assessing the probability that the patient described has a particular symptom X, did you assume that (check one):
X is the only symptom experienced by the patient?
X is among the symptoms experienced by the patient?
60 of 62 physicians, asked this question, checked the second answer.
Fourth and finally, as Tversky and Kahneman write, "An additional group of 24 physicians, mostly residents at Stanford Hospital, participated in a group discussion in which they were confronted with their conjunction fallacies in the same questionnaire. The respondents did not defend their answers, although some references were made to 'the nature of clinical experience.' Most participants appeared surprised and dismayed to have made an elementary error of reasoning."
A further experiment is also discussed in Tversky and Kahneman (1983) in which 93 subjects rated the probability that Bjorn Borg, a strong tennis player, would in the Wimbledon finals "win the match", "lose the first set", "lose the first set but win the match", and "win the first set but lose the match". The conjunction fallacy was expressed: "lose the first set but win the match" was ranked more probable than"lose the first set". Subjects were also asked to verify whether various strings of wins and losses would count as an extensional example of each case, and indeed, subjects were interpreting the cases as conjuncts which were satisfied iff both constituents were satisfied, and not interpreting them as material implications, conditional statements, or disjunctions; also, constituent B was not interpreted to exclude constituent A. The genius of this experiment was that researchers could directly test what subjects thought was the meaning of each proposition, ruling out a very large class of misunderstandings.
Does the conjunction fallacy arise because subjects misinterpret what is meant by "probability"? This can be excluded by offering students bets with payoffs. In addition to the colored dice discussed yesterday, subjects have been asked which possibility they would prefer to bet $10 on in the classic Linda experiment. This did reduce the incidence of the conjunction fallacy, but only to 56% (N=60), which is still more than half the students.
But the ultimate proof of the conjunction fallacy is also the most elegant. In the conventional interpretation of the Linda experiment, subjects substitute judgment of representativeness for judgment of probability: Their feelings of similarity between each of the propositions and Linda's description, determines how plausible it feels that each of the propositions is true of Linda. If this central theory is true, then the way in which the conjunction fallacy follows is obvious - Linda more closely resembles a feminist than a feminist bank teller, and more closely resembles a feminist bank teller than a bank teller. Well, that is our theory about what goes on in the experimental subjects minds, but how could we possibly know? We can't look inside their neural circuits - not yet! So how would you construct an experiment to directly test the standard model of the Linda experiment?
Very easily. You just take another group of experimental subjects, and ask them how much each of the propositions "resembles" Linda. This was done - see Kahneman and Frederick (2002) - and the correlation between representativeness and probability was nearly perfect. 0.99, in fact. Here's the (rather redundant) graph:
This has been replicated for numerous other experiments. For example, in the medical experiment described above, an independent group of 32 physicians from Stanford University was asked to rank each list of symptoms "by the degree to which they are representative of the clinical condition of the patient". The correlation between probability rank and representativeness rank exceeded 95% on each of the five tested medical problems.
Now, a correlation near 1 does not prove that subjects are substituting judgment of representativeness for judgment of probability. But if you want to claim that subjects are doing something else, I would like to hear the explanation for why the correlation comes out so close to 1. It will really take quite a complicated story to explain, not just why the subjects have an elaborate misunderstanding that produces an innocent and blameless conjunction fallacy, but also how it comes out to a completely coincidental correlation of nearly 1 with subjects' feeling of similarity. Across multiple experimental designs.
And we all know what happens to the probability of complicated stories: They go down when you add details to them.
Really, you know, sometimes people just make mistakes. And I'm not talking about the researchers here.
The conjunction fallacy is probably the single most questioned bias ever introduced, which means that it now ranks among the best replicated. The conventional interpretation has been nearly absolutely nailed down. Questioning, in science, calls forth answers.
I emphasize this, because it seems that when I talk about biases (especially to audiences not previously familiar with the field), a lot of people want to be charitable to experimental subjects. But it is not only experimental subjects who deserve charity. Scientists can also be unstupid. Someone else has already thought of your alternative interpretation. Someone else has already devised an experiment to test it. Maybe more than one. Maybe more than twenty.
A blank map is not a blank territory; if you don't know whether someone has tested it, that doesn't mean no one has tested it. This is not a hunter-gatherer tribe of two hundred people, where if you do not know a thing, then probably no one in your tribe knows. There are six billion people in the world, and no one can say with certitude that science does not know a thing; there is too much science. Absence of such evidence is only extremely weak evidence of absence. So do not mistake your ignorance of whether an alternative interpretation has been tested, for the positive knowledge that no one has tested it. Be charitable to scientists too. Do not say, "I bet what really happened was X", but ask, "Which experiments discriminated between the standard interpretation versus X?"
If it seems that I am driving this point home with a sledgehammer, well, yes, I guess I am. It does become a little frustrating, sometimes - to know about this overwhelming mountain of evidence from thousands of experiments, but other people have no clue that it exists. After all, if there are other experiments supporting the result, why haven't they heard of them? It's a small tribe, after all; surely they would have heard. By the same token, I have to make a conscious effort to remember that other people don't know about the evidence, and they aren't deliberately ignoring it in order to annoy me. Which is why it gets a little frustrating sometimes! We just aren't built for worlds of 6 billion people.
I'm not saying, of course, that people should stop asking questions. If you stop asking questions, you'll never find out about the mountains of experimental evidence. Faith is not understanding, only belief in a password. It is futile to believe in something, however fervently, when you don't really know what you're supposed to believe in. So I'm not saying that you should take it all on faith. I'm not saying to shut up. I'm not trying to make you feel guilty for asking questions.
I'm just saying, you should suspect the existence of other evidence, when a brief account of accepted science raises further questions in your mind. Not believe in that unseen evidence, just suspect its existence. The more so if it is a classic experiment with a standard interpretation. Ask a little more gently. Put less confidence in your brilliant new alternative hypothesis. Extend some charity to the researchers, too.
And above all, talk like a pirate. Arr!
Kahneman, D. and Frederick, S. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. Pp 49-81 in Gilovich, T., Griffin, D. and Kahneman, D., eds. Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, Cambridge.
Tversky, A. and Kahneman, D. 1982. Judgments of and by representativeness. Pp 84-98 in Kahneman, D., Slovic, P., and Tversky, A., eds. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.
Tversky, A. and Kahneman, D. 1983. Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90: 293-315.
