The Signal and the Noise: Why So Many Predictions Fail, but Some Don't

We must not expect more precision than the subject matter admits.
Aristotle

Not long after I embarked on the study of psychology, a vague feeling of dissatisfaction with the field began to take hold of me. It took me a while before I realized why I felt this way, but eventually I came to understand that it was due to two factors:

1. The limited ability of the discipline to capture the emotional truths of ordinary experience.

2. The high degree of uncertainty and lack of agreement about research findings.

Perhaps I was asking too much of the discipline, too much at this time in its development. However, in my lifetime, I really didn't see anything to indicate it was making much progress. Quite to the contrary, all I could see was growing conflict between theoretical accounts, increasing neuro-physiologizing, and continuing contradictions between empirical investigations.

I have come to believe that psychology will always have to be content with limitations on the generality of its findings. They may hold for some people, some of the time, but one never can be sure on any given occasion if they apply to a particular individual in the situation at hand.

Nate Silver in The Signal and the Noise: Why So Many Predictions Fail But Some Don’t suggests I really need to change my way of thinking. He very persuasively argues that we can never make the kind of objective predictions I am seeking, that we must learn to think probabilistically about them, and understand that there will always be a certain degree of uncertainty to any prediction.

…we must think differently about our ideas—and how to test them. We must become more comfortable with probability and uncertainty. We must think more carefully about the assumptions and beliefs that we bring to a problem.

Silver has written an important and extensively documented analysis of predictions made in wide range of subjects including the recent financial meltdown, elections (Silver correctly predicted the Obama’s electoral and popular vote in the last election on his 538 Blog at the Times.), baseball player performance, weather, earthquakes, disease epidemics, the stock market, global warming, terrorist attacks, etc. The notes alone consist of 55 pages of densely annotated citations.

He also treats at some length how we can improve on the accuracy of the predictions we make. The more eagerly we commit to scrutinizing and testing our theories, the more readily we accept that our knowledge of the world is uncertain, the more willingly we acknowledge that perfect prediction is impossible.

Silver confronts head on the biases that influence many predictions. Overconfidence is especially common, as is our fundamental ignorance of probability theory. This prevents us from thinking in terms of the conditional probability of a future event.

The virtue in thinking probabilistically is that you will force yourself to stop and smell the data—slow down, and consider the imperfections in your thinking. Over time, you should find that this makes your decision making better.

But nowhere does Silver treat at any length the kinds of questions that psychologists pose, say, for example, how an individual will feel about a future stressful period in their life, the likelihood that they will change a long-held habit, or the nature of their social relationships, and cognitive processes.

Whether or not his methodology can be usefully applied to such issues is an open question. Regardless, whatever forms his analysis would take, it would inevitably be formulated in probabilistic terms and its value would depend on a certain willingness to accept a considerable degree of uncertainty. Thus, we circle back to the basic concerns I expressed about psychology at the outset.

No doubt Silver will remind us once again that we will inevitably be faced with epistemological uncertainty, especially when it comes to the very considerable difficulties of predicting human behavior.