I published Part One of my notes to “How Not to Be Wrong” yesterday.

1. Noise is just as likely to push you in the opposite direction from the real effect as it is to tell the truth. So we’re left in the dark by a result that offers plenty of statistical significance but very little confidence. Scientists call this problem “the winner’s curse,” and it’s one reason that impressive and loudly touted experimental results often melt into disappointing sludge when the experiments are repeated. (See more like this in “The Signal and the Noise”)
2. The replicability crisis is simply a reflection of the fact that science is hard and that most ideas we have are wrong—even most of those ideas that survive a first round of prodding.
3. The investor, like the scientist, gets to see the one rendition of the experiment that went well by chance, but is blind to the much larger group of experiments that failed. There’s one big difference, though. In science, there’s no shady con man and no innocent victim. When the scientific community file-drawers its failed experiments, it plays both parts at once. They’re running the con on themselves. This is all assuming that the scientists in question are playing fair. But that doesn’t always happen. Remember the wiggle-room problem that ensnared the Bible coders? Scientists, subject to the intense pressure to publish lest they perish, are not immune to the same wiggly temptations. If you run your analysis and get a p-value of 0.06, you’re supposed to conclude that your results are statistically insignificant. But it takes a lot of mental strength to snuff years of work in the file drawer. After all, don’t’ the numbers for that one experimental subject look a little screwy? Probably an outlier, maybe try deleting that line of the spreadsheet. Did we control for age? Did we control for weather outside? Did we control for age and weather outside? Give yourself license to tweak and shade the statistical tests you carry out on your results, and you can often get that 0.06 down to a 0.04.
4. The p-hackers truly believe in their hypotheses, just as the Bible coders do, and when you’re a believer, it’s easy to come up with reasons that the analysis that gives a publishable p-value is the one you should have done in the first place.
5. But a conventional boundary, obeyed long enough, can be easily mistaken for an actual thing in the world. Imagine if we talked abo the state of the economy this way! Economists have a formal definition of a “recession,” which depends on arbitrary thresholds just as “statistical significance” does. One doesn’t say, “I don’t care about the unemployment rate, or housing starts, or the aggregate burden of student loans, or the federal deficit; if it’s not a recession, we’re not going to talk about it.” One would be nuts to say so. The critics– and there are more of them, and they are louder, each year—say that a great deal of scientific practice is nuts in just this way.
6. Data is messy, and inference is hard.
7. A statistically significant finding gives you a clue, suggesting a promising place to focus your research energy. The significance test is the detective, not the judge. You know how when you read an article about a breakthrough finding that this thing causes that thing, or that thing prevents the other thing, and at the end there’s always a banal sort of quote from a senior scientist not involved in the study intoning some very minor variant of “The finding is quite interesting, and suggests that more research in this direction is needed”?
8. Replication: if an effect can’t be replicated, despite repeated trials, science backs apologetically away. The replication process is supposed to be science’s immune system, swarming over newly introduced objects and killing the ones that don’t belong.
9. The age of big data is frightening to a lot of people, and it’s frightening in part because of the implicit promise that algorithms, sufficiently supplied with data, are better at inference than we are.
10. It’s possible we ought to spend less time worrying about eerily superpowered algorithms and more time worrying about the crappy ones.
11. “It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth.” ~Sherlock Holmes.
12. If gambling is exciting, you’re doing it wrong.
13. One percent of infinite bliss is still infinite bliss, and outweighs whatever finite costs attach to a life of piety. The same goes for 0.1% or 0.000001%. All that matters is that the probability of God exists in not zero. Don’t you have to concede that point? That the existence of the Deity is at least possible? If so, then the expected value computation seems unequivocal: it is worth it to believe. The expected value of that choice is not only positive, but infinitely positive.
14. A ducat in the hand of a rich man is not worth the same as a ducat in the hand of a peasant, as is plainly visible from the different levels of care with which the two men treat their cash. In particular, having two thousand ducats isn’t twice as good as having one thousand; it is less than twice as good, because a thousand ducats is worth less to a person who already has a thousand ducats than it is to the person who has none. Twice as many ducats doesn’t translate into twice as many utils: not all curves are lines, and the relation between money and utility is governed by one of those nonlinear curves.
15. In a bigger departure from classical theory, Daniel Kahnemann and Amos Tversky suggested that people in general tend to follow a different path from the one the utility curve demands, not just when Daniel Ellsberg sticks an urn in front of them, but in the general course of life. Their “prospect theory,” for which Kahnemann later won the Nobel prize, is not seen as the founding document of behavioral economics, which aims to model with the greatest possible fidelity the way people do act, not the way that, according to an abstract notion of rationality, they should. In the Kahnemann-Tversky theory, people tend to place more weight on low-probability events than a person obedient to the von Neumann-Morgenstern axioms would; so the allure of the jackpot exceeds what a strict utility calculation would license.
16. It’s hard to separate our moral feelings about an activity from the judgments we make about its rationality.
17. There is real danger that, by strengthening our abilities to analyze some questions mathematically, we acquire a general confidence in our beliefs, which extends unjustifiably to those things we’re still wrong about. We become like those pious people who, over time, accumulate a sense of their own virtuousness so powerful as to make them believe the bad things they do are virtuous too.
18. If you never give advice until you’re sure it’s right, your not giving enough advice.
19. Are we trying to figure out what’s true, or are we trying to figure out what conclusions are licensed by our rules and procedures? Hopefully the two notions frequently agree; but all the difficulty, and thus all the conceptually interesting stuff, happens at the points where they diverge.
20. I thought “hardworking” was a kind of veiled insult—something to say about a student when you can’t honestly say they’re smart. But the ability to work hard—to keep one’s whole attention and energy focused on a problem, systematically turning it over and over and pushing at everything that looks like a crack, despite the lack of outward signs of progress—is not a skill everybody has.
21. That moment of inspiration is the product of weeks of work, both conscious and unconscious, which somehow prepare the mind to make the necessary connection of ideas. Sitting around waiting for inspiration leads to failure, no matter how much of a whiz kid you are.
22. “It takes a thousand men to invent a telegraph, or a steam engine, or a phonograph, or a telephone or any other important thing—and the last man gets the credit and we forget the others.” ~Mark Twain. (See more like this in “The Innovators”)
23. See this post about Theodore Roosevelt and John Ashbery. It’s a more complex and accurate portrait of life’s enterprise than Roosevelt’s hard-charging man’s man, sore and broken but never doubting his direction.
24. Math gives us a way of being unsure in a principled way: not just throwing up our hands and saying “huh,” but rather making a firm assertion: “I’m not sure, this is why I’m not sure, and this is roughly how not-sure I am.” Or even more: “I’m unsure, and you should be too.”
25. The lessons of mathematics are simple ones and there are not numbers in them: that there is structure in the world; that we can hope to understand some of it and not just gape at what our senses present s; that our intuition is stronger with a formal exoskeleton that without one. And that mathematical certainty is one thing, the softer convictions we find attached to us in everyday life another, and we should keep track of the difference if we can.

How Not to Be Wrong: The Power of Mathematical Thinking