Are Scientific Papers Worth Reading?

Hello, friends!

So over the course of the last few months, I’ve been learning about metascience.  I’ve been reading lots of metascientific articles and papers, and I’ve been watching a few metascientific lectures on YouTube.  For those of you who are unfamiliar with the concept, metascience is the scientific study of science itself, for the specific purpose of identifying fraud, correcting errors in the scientific process, and making science overall a more accurate and trustworthy thing.

Before I go any further with this topic, I think it’s extra important for you to understand who I am and what my perspective on science (and metascience) is.  I am not a scientist.  I have no professional or educational background in science.  What I am is a science fiction writer who wants to do his research so that science (as I portray it in my fiction) is accurate.  Well, somewhat accurate, or at least somewhat plausible.  At the very least, I want to make sure the science in my stories is not laughably implausible.

In order to do my research (as a science fiction writer), I have challenged myself to read peer-reviewed scientific papers.  I try to read at least one peer-reviewed paper each week.  As you can imagine, this is not easy.  These papers are packed full of jargon (some papers define their own jargon; most do not) and a whole lot of math (the kind of math where you see more of the Greek alphabet than Arabic numbers).

And now I learn, thanks to metascience, that the peer-review process is deeply flawed, and that science has way more problems than I ever realized.  There’s a lot of fraud going on, and also a lot of laziness and complacency, and scientists are not double checking each other’s work the way that they should.  That last problem—scientists not double checking each other’s work—is commonly known as the replication crisis.  It’s a problem which this article from Vox.com calls “an ongoing rot in the scientific process.”

No branch of science is immune to these problems, but I can take some solace in the fact that some branches of science seem to be more afflicted with problems than others.  Fields like medical science, computer science, and engineering (i.e.: the big money-maker sciences) are far more prone to fraud than fields like cosmology, astrophysics, or planetary science (i.e.: fields that I, as a science fiction writer, take the most interest in).  But still, as I said, no branch of science is immune.  Lazy and/or biased and/or unscrupulous researchers are everywhere.

And yet, despite some very valid concerns, I intend to keep reading these peer reviewed papers.  Why?  Because my alternative would be to get most of my science news and information from the popular press.  When it comes to science, the popular press has an annoying tendency to dumb things down, to gloss over boring (but important) details, and to hype up hypotheses that are the most likely to attract clicks and views but are the least likely to actually be true.  If I wrote my Sci-Fi based solely on what I read in the popular press, the science in my fiction would be laughably implausible.

I’d rather struggle through reading a peer-reviewed paper once a week.  Those papers may not be perfect, but reading them will get me much closer to the truth than relying on any other source of information currently available to me.

WANT TO LEARN MORE?

If you’d like to learn more about metascience and the replication crisis, I suggest checking out some of the links below.  These links are organized from “easiest and most accessible” at the top to “most technical” at the bottom.

Sciency Words: The Replication Crisis

Hello, friends!  Welcome back to Sciency Words, a special series here on Planet Pailly where we talk about new and interesting scientific terms so we can expand our scientific vocabularies together!  In this week’s episode of Sciency Words, we’re talking about:

THE REPLICATION CRISIS

There’s a quote that I hate which is frequently misattributed to Albert Einstein: “The definition of insanity is doing the same thing over and over again and expecting different results.”  Why do I hate this quote?  First off, as a matter of historical record, Einstein never said this.  But more importantly, doing the same thing over and over again to see if anything different happens is a surprisingly good definition of science.

Or it least it should be, which brings us to this week’s Sciency Word: the replication crisis.  As this brief introductory article retells it, the replication crisis began with “a series of unhappy events” in 2011.  Certain “questionable research practices” were exposed, along with several cases of outright fraud.  I’m going to focus on just one very noteworthy example: the American Psychological Association published a paper titled “Feeling the Future,” which claimed to show statistically significant evidence that human beings have precognitive powers.

When other researchers tried to replicate the “Feeling the Future” experiments, they failed to find this statistically significant evidence.  However, according to this episode of Veritasium, the American Psychological Association had a policy at the time that they would not publish replication studies, and so they would not publish any of the research debunking the original “Feeling the Future” paper (I do not know if they still have that policy—I would hope that they do not).

The act of repeating experiments to see if anything different happens is a crucial part of how science works.  Or rather how it should work.  But this is not being done often enough, it seems.  And on those rare occasions when replication studies are performed (and published), a shocking number of high profile research turns out to be non-replicable.  This article from Vox.com sums up just how bad the replication crisis is:

One 2015 attempt to reproduce 100 psychology studies was able to replicate only 39 of them.  A big international effort in 2018 to reproduce prominent studies found that 14 of the 28 replicated, and an attempt to replicate studies from top journals Nature and Science found that 13 of the 21 results looked at could be reproduced.

That same Vox.com article calls the replication crisis “an ongoing rot in the scientific process.”

But as I’ve been trying to say in several of my recent posts, science is self-correcting.  With the introduction of metascience—the scientific study of science itself—there is some hope that the root causes of the replication crisis can be identified, and perhaps changes can be made to the way the scientific community operates.

Sciency Words: Metascience

Hello, friends!  Welcome to Sciency Words, a special series here on Planet Pailly where we talk about those weird and wonderful words scientists use.  Today on Sciency Words, we’re talking about:

METASCIENCE

Metascience is when science “gets meta” and studies itself, with the specific aim of making published scientific research more accurate and trustworthy.  That goal, that stated purpose, is an important part of the definition.  Or at least it should be, according to this YouTube video by Professor Fiona Fidler.

You see, metascience overlaps with certain other fields of research, like the philosophy of science or the sociology of science.  But a key part of a metascientist’s job is to identify problems with the current culture and methodology of scientific research and try to figure out ways to make science better.

The word metascience can be traced back to the 1930’s, with the earliest known usage attributed to American philosopher and semiotician Charles William Morris.  But as an actual field of research, metascience is not nearly that old.  This 2005 paper entitled “Why Most Published Research Findings Are False” is apparently a foundational document for modern metascience (or at least that’s what Wikipedia told me).

For a few months now, I’ve been doing lots of research about research, trying to improve the way I do my own research as a science fiction writer, and also trying to better understand what can go right (and wrong) with science.  With that in mind, I’m surprised I didn’t come across this term sooner.  Now that I do know about metascience, a whole new world of metascientific research has been revealed to me.

Reading about metascience has been kind of unsettling for me, actually.  Modern science has a lot more problems than I realized; however, there are people out there working to identify and fix those problems, so that science can live up to its promises.  And that, I think, is a very encouraging thing to know.