Science Is Not an Exact Science

Psychology has come a long way. Theories of human personality propounded by the originators of this science were based on scant evidence, and some were even just dreamed up. And because experience has proven so many of those ideas to be wrong, few psychologists today accept them or practice them.

Other schools of thought have replaced the earlier ones, and the field has adopted the scientific method, replete with clinical controls and statistical analysis — the fact-based rigor that Freud himself — consciously or otherwise — avoided.

An essential part of scientific method is reproducibility. In order for one scientist to verify the experimental findings of another, he must be able to reproduce, or replicate, the results, using the same materials, following the same steps, as his predecessor. If results claimed by the first scientist cannot be duplicated by his colleagues — if the quarks spin the wrong way — then whatever the first scientist claimed comes into question.

That is why a study published a few days ago in the journal Science has caused such a to-do. Some 300 researchers tried to reproduce 100 experiments that were written up in the most prestigious journals of psychology in 2008 — and mostly failed. Only about 40 percent of the experiments could be repeated!

(Bear in mind that they were reviewing some of the best work done that year; had they tried to replicate results from lesser quality publications, the outcome would likely have been even more traumatic.)

For some, this would be the ultimate hammer blow, the makeh b’patish, that demonstrates once and for all that not only Freud but the various post-Freudians, too, are wrong in their whole approach to what ails us — that, as somebody once observed, the similarity between Freud and fraud are more than orthographical.

But to invoke a pre-Freudian psychological axiom — misery loves company — the psychologists are not the only ones suffering from what has been called a “crisis of irreproducibility.”
“The results are more or less consistent with what we’ve seen in other fields,” said Ivan Oransky, one of the founders of Retraction Watch, which tracks scientific retractions.

In other words, there is no reason to get hysterical (a psychological term that also predated Freud). Science is all like this. It’s not perfect. The logic isn’t always sound, the proofs not always waterproof, the science isn’t always so scientific.

But this doesn’t mean we should dismiss all science as bunk and all scientists as frauds. Generally, scientific research does provide good evidence. The thousands of effective and life-saving medicines on the market, as well as the diabolical effectiveness of marketing strategies, are, after all, the products of just such research. Somebody is doing something right.

What, then, is the explanation for the lack of reproducibility?

Actually, replicating a study is much harder than you might think. It’s not always so simple as heating water to 212˚F and getting it to boil every time. The studies in question were often extremely complicated, involving hundreds or thousands of respondents, precisely formulated questions, specific laboratory environments, and so on. In the case of psychological studies, cultural factors (attitudes toward risk or ethical conduct, in different societies or ethnic groups, for example) must be taken into account.

As well, the exact physical set-up of an experiment, and the order in which things are done, step-by-step, must be repeated accurately to offer a fair chance of replication. In the best efforts, as in this case, the reviewing scientists work closely with the original researchers to obtain their data in full and discuss the details of their methods.

Brian Nosek, the psychology professor at the University of Virginia who led the experiments in reproducibility, summed it up this way: “Any one study is not going to be the last word. Each individual study has some evidence. It contributes some information toward a conclusion. But the real conclusion, when you can say confidently that something is true or false, is based on an accumulation of evidence over many studies.”

Now we know why psychologists aren’t running in panic to their therapists after getting the news about the study. They knew not to expect all the studies to come out the same way twice.

As for the rest of us, it’s a worthwhile lesson to learn: Not to get too worked up over the latest revelation about the secrets of the human psyche.

Wait until the follow-up studies come out. See if they get the same results twice in a row.

This article appeared in print on page 10 of the September 3rd, 2015 edition of Hamodia.