Sciency Words: File-Drawer Effect

Hello, friends!  Welcome to Sciency Words, a special series here on Planet Pailly where we take a closer look at new and interesting scientific terms.  This week’s Sciency Word is:

THE FILE-DRAWER EFFECT

This came up as part of my ongoing research about research, and it’s another example of how scientific research can go wrong.

Okay, so let’s say I have this hypothesis: people who watch Star Trek are better at math than people who do not watch Star Trek.  Ten different research teams set up experiments to test my hypothesis.  Only one of those ten teams manages to find a statistically significant relationship between watching Star Trek and being good at math.

The other eight teams are unable to find a statistically significant relationship, conclude that this was a huge waste of time, and move on to researching other things.  They decide not to bother publishing any of their findings.  Instead, all that research gets stuffed into a file-drawer, never to see the light of day again.

Meanwhile, that one team that did find a statistically significant relationship… they do publish their findings.  It’s such an astonishing result!  How could they not?  Soon, their results are being reported all over the news, and every Star Trek forum on the Internet goes wild, and parents start forcing their kids to watch extra episodes of Star Trek so they’ll do better on their math homework.

But that one research paper is totally contradicted by all the other research—or it would be, if any of that other research had been published.  As a result, the scientific community—and the general public as well—now have a terribly skewed understanding of the relationship between watching Star Trek and being good at math.  This is the file-drawer effect, also known as publication bias, at work.

P.S.: I mean, I’ve watched a ton of Star Trek, and everyone knows I’m good at math.  That sort of anecdotal evidence, plus a single peer-reviewed research paper, should be enough to convince everybody!