Ideas

Horses Doing Math: On the Quality of Scientific Studies

February 12, 2015

Follow Jon
Clever Hans the horse.
Clever Hans the horse.

How difficult is it to overcome biases in scientific studies?

Probably more difficult than you think.

Even an honest, well-meaning scientist can come up with a study design that could strongly influence the results. My favorite example of how expectations can bias results is the story of Clever Hans the horse.

Clever Hans is not a science story, but it could well serve as a warning as we try to be unbiased on controversial issues from climate change to GMO’s.

Clever Hans’ owner gave demonstrations where he would ask the horse mathematical questions, ranging from simple arithmetic to rather complicated calculations: “If the eighth day of the month comes on a Tuesday, what is the date of the following Friday?”

Hans answered them by tapping his hoof the correct number of times.

Given that the latter question would require understanding words, grammar and even abstract concepts (like a “month”), in addition to math, people were understandably skeptical.

Psychologist Oskar Pfungst found that Hans’ real talent was in reading the unconscious cues from the person asking the question. If the questioner didn’t know the answer (a double-blind experiment), Hans didn’t either.

But otherwise (in a “single-blind” setup where Hans didn’t know the answer but the questioner did) Hans would pick up on cues like changes in posture and facial expression as his tapping approached the right number, and stop tapping at the right place.

The key point here is that this worked even when the questioner was not trying to give the answer away!

The Clever Hans in All of Us

There may well be a bit of “Clever Hans” in all of us. We are generally very sensitive to both cues given off by other humans and our preconceived notions even if we’re not fully aware of it.

You may have seen a variant of this when kids guess an answer to a question, and keep guessing until the person asking the question smiles or congratulates them to declare their final answer.

Imagine you’re taking the “Pepsi Challenge” to determine whether you like Pepsi or Coke better. If you know which is which, your expectations about which one you think you will like will influence your experience (see this example with beer).

If the samples are unlabeled but the person serving you knows which is which, the server may provide subconscious cues (e.g. holding their breath or making a face when you taste something they don’t think you will like). So in a double-blind study, no one knows which is which until the experiment concludes.

Still not convinced this kind of rigor is important? If you want to see it for yourself, you can actually do your own experiments with taste tests (e.g. here are some I ran with cola and other foods and drinks). Vary your design, and see the impact it has.

Overcoming Bias in Conservation Science

So what does this have to do with conservation science?

There are at least three important lessons.

First, not all studies are created equal; it is important to consider the experimental design and overall quality of a study. Sometimes scientists conduct meta-analyses (a study attempting to find patterns by looking at the aggregate of multiple studies) showing that the studies with the most rigorous experimental design have a more clear pattern and arguing that we should therefore consider experimental rigor when assessing evidence (e.g. this study on homeopathy).

Second, while it’s always exciting to read new research, we need multiple studies (conducted by different people who hopefully have different biases) to increase our confidence in the findings of any given study.

Third, when we process information on important but controversial issues (from climate change, to the use of genetically modified organisms in agriculture, to the roles large predators play in ecosystems) our preconceptions affect how we treat that new data.

So it’s important to take a moment to review things like experimental design and results from similar studies on the topic (and meta-analyses if available), and not fall into the trap of assuming everyone who disagrees you is dead wrong because you found a study that says so.

So how do we critically review new research, and what are some key things to look for?

Essentially, a good study design means the study does a good job of testing the hypothesis being investigated by ruling out other explanations (confounding variables), following procedures that minimize the effect of scientists’ bias, and quantifying the strength of evidence obtained. There have been a couple of discussions on this topic on Cool Green Science by Timm Kroeger and Paul Ferraro, among others.

In my own work I have been thinking a lot about how to provide better evidence (via well-designed studies) about the relationship between agricultural or livestock practices and conservation outcomes.

While most studies are fairly clear on their experimental design (whether single-blind or double-blind, what variables were controlled for, randomization protocols, qualitative vs quantitative metrics, whether a control was used and how it was selected if so, sample size, statistical methods, etc.) it takes a lot of time to get this information from papers, and such details are rarely included in science news.

One reasonable proxy for study quality is which journal it was published in. While even the best journals sometimes have to issue retractions or corrections, chances are that an article in a top journal like Science or Nature has been vetted a lot more thoroughly than the predatory journals cropping up where just about anything can be published for a fee.

We are often inherently biased, even if we think we’re being rational. We pick up cues and we naturally focus on studies that confirm our worldviews.

We may always have these faults. Fortunately, the scientific method can in many ways help us separate cool but misleading stories like Clever Hans from solid science – helping us to see the science of conservation more clearly and more rigorously.

If a story sounds too good to be true, ask yourself what other explanations exist and how the research could be flawed. If you see research that doesn’t fit with your understanding of the world, but it seems to be taken seriously by reputable scientists, ask yourself what you might be missing.

And don’t despair: while horses may not be able to do math, there is a growing body of research on animal cognition that explores some pretty amazing feats that animals actually can accomplish, and some similar tasks that they find impossible.

Jon Fisher

Jon Fisher is a senior conservation scientist for the new Center for Sustainability Science at The Nature Conservancy. He is leading efforts to put rigorous science front and center in our sustainable agriculture work, and finding ways to improve sustainability through corporate practices and public policy. More from Jon

Follow Jon

Join the Discussion

Please note that all comments are moderated and may take some time to appear.

1 comment