Treat Studies with Skepticism

“Study: Shoe Size Correlates With Math Fluency” isn’t a news headline you’ve likely seen, but it could well be offered. After all, small children have small feet; and, as they grow, they become more proficient in making mathematical calculations.

That example points to the fact that scientific study results can be misleading, even when their methodologies may seem impeccable. The general public is often simply not equipped to discern when reported scientific “conclusions” are in fact inconclusive.

What’s more, an alarming percentage of even mainstream scientific papers’ results have been shown to be irreproducible. A year ago, researchers at biotech firm Amgen found they could reproduce a mere six of 53 “landmark” studies in cancer research. Earlier, a group at the drug company Bayer managed to repeat only a quarter of 67 similarly important papers.

Professional pressure, competition and ambition all combine to push scientists to publish more quickly than would be wise.

And then there’s the influence of funding from parties seeking particular results.

A recent Associated Press story highlighted the latter concern, specifically with regard to the food industry, and the reportage of nutrition study results by the media.

As competition for government funding intensifies, industry funding is increasingly critical for advancing science. Food companies have taken advantage of the situation by funding research pertinent to their products. The companies insist that scientists they fund to conduct studies are not pushed to achieve any particular results, and the scientists proclaim their unfettered objectivity.

There is little reason to imagine that companies place overt pressure to reach any particular conclusions. But there are subtle influences at work and researchers are well aware of the sources of their funding.

As Chazal teach us, negius, or ulterior factors, can be at play even when people are convinced that they are acting objectively. Rabbi Yishmael’s sharecropper would bring him a basket of fruit from Rabi Yishmael’s land every Friday. The Gemara (Kesuvos, 105b) recounts how, one week, the worker brought the fruit to him on a Thursday.

When Rabi Yishmael asked why, the worker explained that he was party to a court case before the Tanna that day and thought that he may as well bring the fruit then too. Rabi Yishmael immediately recused himself from the sharecropper’s case.

If a Tanna can be subject to subconscious bias, all the more so a mere scientist.

And what a funder does with data provides even more reason for suspicion. “Does candy,” a CBS News headline asks, “keep kids from getting fat?” The teaser was born of a scientific paper that utilized a government database of surveys that asked people to recall what they ate in the past 24 hours. The paper was funded by a trade organization representing chocolate bar producers.

Although the researchers cautioned in their report that “cause and effect associations cannot be drawn” from their study, the group that funded the undertaking issued a press release declaring: “New study shows children and adolescents who eat candy are less overweight or obese.”

If it’s not quite a math proficiency/shoe size correlation, it’s close. AP’s Food Industry Writer, Candice Choi, rifled through thousands of pages of e-mails and discovered one from an author of that study, a professor of nutrition at Louisiana State University, commenting on an attached abstract for the paper with the telling words: “We’re hoping they can do something with it — it’s thin and clearly padded.”

Then there’s PepsiCo, which owns Quaker and has funded studies showing the benefits of oats. In 2011, the company hired scientists to test the hypothesis that its Quaker oatmeal and cold cereal squares would each be more filling than a popular breakfast cereal made by one of its rivals.

The trial, which involved less than 50 participants, yielded commercially useful data for the hot oatmeal but only mixed results for the cold cereal.

“I am sorry that the oat squares did not perform as well as hoped,” reported the scientist overseeing the study, “but your hypotheses were validated with the oatmeal.”

When PepsiCo published the study, it chose to include only the happy results, deeming the inconclusive ones insufficiently significant to share with the public.

There’s nothing wrong with industry funding research it believes may yield results that will be useful in marketing. But shoe size doesn’t really correlate with math ability, and eating candy is no diet aid. And so it’s important that consumers be clearly apprised of whose money is behind a study, so they can treat industry-funded research results with a healthy dose of skepticism.

This article appeared in print on page 10 of the June 6th, 2016 edition of Hamodia.