The Limits of Research

Jonah Lehrer’s dazzling essay, The Decline Effect and the Scientific Method, has been the most frequently e-mailed article from the New Yorker since it was published last month. This pleases me a lot. For Lehrer describes one of the most serious problems associated with research in the social sciences, especially research in psychology and, as he points out, the natural sciences as well.

In a nutshell the problem is the difficulty of replicating experimental findings. A derivative problem is the frequently observed decline in the overall effect of any particular variable(s) with repeated testing. Replication is the heart of scientific research. That is the purpose of making the findings public—so that investigators in other labs can see if they can observe the same outcome. He writes,

“Different scientists in different labs need to repeat the protocols and publish their rests. The test of replicability, as it is know, is the foundation of modern research. Replicability is how the community reinforces itself.”

Lehrer cites many examples of the failure to replicate. He notes how often the therapeutic power of a drug wanes on repeated testing and then eventually disappears or turns out on analysis to be largely a function of the placebo effect. He discusses at some length the difficulty of replicating the phenomenon of verbal overshadowing—how the act of describing memories seems or seemed at one time, to interfere with recall.

And he cites the well-known failure to replicate the early studies by Joseph Rhine on extrasensory perception (ESP) that turned out to be a statistical fluke. It was Rhine who first coined the term “the decline effect.”

It has even been observed in a series of studies where the same genetic strains of mice were shipped to three different labs on the same day from the same supplier and raised under identical conditions, including the way they were handled. The investigator who reported the widely different results from each lab concluded that a lot of scientific data are nothing but noise. “The end result is a scientific accident that can take years to unravel.”

What accounts for the fact that so many research studies cannot be replicated? Several factors are at work. Proper control conditions may have been omitted from the original experiments, the samples may not have been randomly selected or consist of a highly uniform, unrepresentative group of individuals, usually college sophomores. Or the results may have occurred because of experimenter biases that inevitably led to evidence supporting the hypothesis. Few experimenters really design studies to disprove, rather than confirm their hypothesis. This is a point Karl Popper emphasized many years ago.

Then there is the publication biases characteristic of most scientific journals. Researchers who do not report positive outcomes cannot get their findings published. According to one study, ninety-seven percent of psychology studies proved their hypothesis. We know this can’t be the case. As one investigator (Richard Palmer, a biologist) noted, “Once I realized that selective reporting is everywhere in science, I got quite depressed.” And he continued:

“We cannot escape the troubling conclusion that some—perhaps many cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori believes often repeated.”

Lehrer has written a powerful, persuasive essay on the limits of scientific research. It is the kind of essay I wish I had written, as I have been aware of this for years based on a lifetime of reading research in psychology, as well as my own studies. I have been guilty of looking for data that confirms my hypothesis, interpreting outcomes in a way that will insure their publication, and no doubt biasing (conservatively) the design of my studies without really trying to test (falsify) them.

But I didn’t write the essay and no doubt could never have done so with the skill and breadth that Lehrer has. I commend him for his piece, as well as the New Yorker for publishing it and thereby making possible its wide dissemination.

What's Next?

Last Tuesday, November 9th, The New York Times published a special anniversary issue of its regular Tuesday edition of Science Times. They asked “top researchers” in ten sciences to predict the future direction of their field and what the most important discoveries are likely to be during the next ten years.

The researcher’s disciplines ranged from genomics to mathematics to earth science. Here is a very brief list of ten disciplines and the predictions of the researchers:

Space Science: Learning more about asteroids

Game Science: Developing games for the study of women’s rights, climate change and medical innovation

Climate Change: Testing the accuracy of current climate change models

Engineering: Growing integration of biological processes in engineering real systems

Biotechnology: Refining methods for producing stem cells that will be “cheap, fast, and relatively easy.”

Conservation Biology: Increasing knowledge of marine biodiversity

Ocean Science: Exploration of the oceans (70% of the planet) with particular emphasis of the “mysterious” yet major worldwide effects of the Indian Ocean

Genomics: Reading new types of genomes and completing DNA sequencing (ordering) of an individual’s (or any organism’s) genome at a single time

Neuroscience: Determination of the physical organization of a memory within the brain

Mathematics: Discovery of scientific results that are correct and predictive but are without explanation. “We may be able to do science without insight, and we may have to learn to live without it.” [Personal note: My favorite prediction and one with particular relevance to psychology.]

There are several features of this list that concern me. I am struck by the overlap of disciplines. There are two pointing to improved understanding of climate change, two focusing on developments in the biological sciences, and two related to the marine sciences. Indeed, the majority of disciplines are drawn from the natural sciences.

Secondly, the social sciences are completely ignored. Not a single researcher in economics, psychology, sociology, anthropology, etc. is interviewed. Are these not sciences? Do they not seek general laws about the subject matter of their inquires? Are there no more discoveries to be made in these fields? Or is it simply that anyone with a head on their shoulders already knows the results of their studies?

Perhaps the editors at the Times do not consider these fields as sciences. Or perhaps they believe we now have a complete understanding of human behavior and are easily able to predict and control it. No, I am sure they do not hold these views. What then can account for their failure to interview a few representative investigators from these areas?

I am aware that they are not unified sciences and that there are frequent disagreements among investigators in these disciplines. For example, Rom Harre, researcher, teacher, and writer in the philosophy of science, and later in social psychology has written about his field, one that was mine for many years:

“It has been about 30 years since the first rumblings of discontent with the state of academic psychology began to be heard…Methods that have long been shown to be ineffective or worse are still used on a routine basis by hundreds, perhaps thousands of people. Conceptual muddles long exposed to view are evidence in almost every issue of standard psychology journals.”

Geoffrey Loftus, a leading cognitive psychologist has also spoken of his concerns about the field. “But I have developed a certain angst over the intervening 30 something years, a constant nagging sensation that our field spends a lot of time spinning its wheels without really making much progress.”

I suspect the same can also be said of the other social sciences. Still that doesn’t make them any less scientific in objectives and methods. Rather it reflects their developing state and the extraordinarily complex subject matter they have carved out to investigate. As others have said, the social sciences are the most difficult of all the sciences.

Finally, I might note that there are also wide areas of uncertainty and theoretical disagreements within the natural sciences. So in this respect they are no different than the social sciences, indeed probably any field of inquiry especially one that in comparison with the natural sciences is in an early stage of development.