A Skeptical Eye

The central concern that Grayling expresses in his essay, Pressing Questions for Our Century, (discussed yesterday) is the responsibility that each of us has to keep abreast of developments in science and technology. To me, this not only means being well informed about scientific research, but also knowing what is required to evaluate the findings reported in the press and in the books that we read. This isn’t easy for a layperson to do. It requires considerable education and careful research.

So for example when you read one of Malcolm Gladwell’s extraordinarily popular book about decision making (Blink: The Power of Thinking Without Thinking) or his latest (Outliers: The Story of Success) on the sources of distinguished accomplishment, you know how to approach his claims critically, judge their accuracy, and, determine whether or not the evidence that he cites or that you find on your own, is adequate grounds for his assertions.

This means knowing something about the methodology of scientific research and the techniques one employs to identify alternative accounts of research findings. The first step in achieving this is in asking the “right” questions, in knowing what questions to ask and what not to ask. This is always the hardest part. It is like composing the first sentence in a book or story you wish to write. Often a writer will struggle with that first sentence for months but once it clear what it will be, the tale often takes off on its own.

In Blink for example, Gladwell claims the rapidly made decisions (“in the blink of an eye”) are every bit as reliable and sometimes more so than decisions made more deliberately. However, that is not exactly the finding reported in the experiment he discusses in support of this claim. What is reported are judgments of teaching effectiveness made by female subjects only, from repeated video clips that lasted as long as several seconds each, and that the female students never actually rated the teacher’s overall effectiveness but instead a set of verbal behaviors that, aggregated together, were presumed to measure it.

Nor does he fairly describe studies suggesting that predictions of marital happiness can be accurately made after observing an hour of a couples’ interaction or that a decision-making algorithm for predicting cardiac arrest can be made on “very thin slices of evidence.” In fact, research in both areas developed after long and complicated testing procedures that involved sifting through a good deal of data and many hours of personal experience. These were not snap judgments or carried out in the blink of an eye.

So it isn’t only the reader has the responsibility to approach science critically. It is also the science writer who has an obligation to report accurately and know how to carefully evaluate the generality of the reported primary studies. A science writer also doesn’t create a very deep understanding of a phenomenon on the basis of a series of (unrepresentative) anecdotes, as Gladwell often relates, no matter how colorful they are.

In the press we often read about a relationship between what we eat and behavior, or between some type of experience and a disease. But we rarely read about the strength of the relationship or whether it might be due to any number of other factors. Nor do we read for whom the findings hold and for whom they don’t. And most such press reports are framed in the language of causality when it fact one can never presume there is a causal relationship from correlational data

No doubt I can say all this because of my research background and training. Nevertheless, I believe any reader can approach press reports and books about science with a critical eye. It takes a certain style of thinking and of questioning to do this. But with practice it can become “second nature” to anyone. Just keeping asking questions, read with a skeptical eye, and be mindful of how the findings were derived and for whom they do and do not hold.