Category: Behind the Science

Are Scientific Papers Worth Reading?

~ J.S. Pailly ~ 21 Comments

Hello, friends!

So over the course of the last few months, I’ve been learning about metascience.  I’ve been reading lots of metascientific articles and papers, and I’ve been watching a few metascientific lectures on YouTube.  For those of you who are unfamiliar with the concept, metascience is the scientific study of science itself, for the specific purpose of identifying fraud, correcting errors in the scientific process, and making science overall a more accurate and trustworthy thing.

Before I go any further with this topic, I think it’s extra important for you to understand who I am and what my perspective on science (and metascience) is.  I am not a scientist.  I have no professional or educational background in science.  What I am is a science fiction writer who wants to do his research so that science (as I portray it in my fiction) is accurate.  Well, somewhat accurate, or at least somewhat plausible.  At the very least, I want to make sure the science in my stories is not laughably implausible.

In order to do my research (as a science fiction writer), I have challenged myself to read peer-reviewed scientific papers.  I try to read at least one peer-reviewed paper each week.  As you can imagine, this is not easy.  These papers are packed full of jargon (some papers define their own jargon; most do not) and a whole lot of math (the kind of math where you see more of the Greek alphabet than Arabic numbers).

And now I learn, thanks to metascience, that the peer-review process is deeply flawed, and that science has way more problems than I ever realized.  There’s a lot of fraud going on, and also a lot of laziness and complacency, and scientists are not double checking each other’s work the way that they should.  That last problem—scientists not double checking each other’s work—is commonly known as the replication crisis.  It’s a problem which this article from Vox.com calls “an ongoing rot in the scientific process.”

No branch of science is immune to these problems, but I can take some solace in the fact that some branches of science seem to be more afflicted with problems than others.  Fields like medical science, computer science, and engineering (i.e.: the big money-maker sciences) are far more prone to fraud than fields like cosmology, astrophysics, or planetary science (i.e.: fields that I, as a science fiction writer, take the most interest in).  But still, as I said, no branch of science is immune.  Lazy and/or biased and/or unscrupulous researchers are everywhere.

And yet, despite some very valid concerns, I intend to keep reading these peer reviewed papers.  Why?  Because my alternative would be to get most of my science news and information from the popular press.  When it comes to science, the popular press has an annoying tendency to dumb things down, to gloss over boring (but important) details, and to hype up hypotheses that are the most likely to attract clicks and views but are the least likely to actually be true.  If I wrote my Sci-Fi based solely on what I read in the popular press, the science in my fiction would be laughably implausible.

I’d rather struggle through reading a peer-reviewed paper once a week.  Those papers may not be perfect, but reading them will get me much closer to the truth than relying on any other source of information currently available to me.

WANT TO LEARN MORE?

If you’d like to learn more about metascience and the replication crisis, I suggest checking out some of the links below.  These links are organized from “easiest and most accessible” at the top to “most technical” at the bottom.

Sciency Words: The Replication Crisis

~ J.S. Pailly ~ 8 Comments

Hello, friends!  Welcome back to Sciency Words, a special series here on Planet Pailly where we talk about new and interesting scientific terms so we can expand our scientific vocabularies together!  In this week’s episode of Sciency Words, we’re talking about:

THE REPLICATION CRISIS

There’s a quote that I hate which is frequently misattributed to Albert Einstein: “The definition of insanity is doing the same thing over and over again and expecting different results.”  Why do I hate this quote?  First off, as a matter of historical record, Einstein never said this.  But more importantly, doing the same thing over and over again to see if anything different happens is a surprisingly good definition of science.

Or it least it should be, which brings us to this week’s Sciency Word: the replication crisis.  As this brief introductory article retells it, the replication crisis began with “a series of unhappy events” in 2011.  Certain “questionable research practices” were exposed, along with several cases of outright fraud.  I’m going to focus on just one very noteworthy example: the American Psychological Association published a paper titled “Feeling the Future,” which claimed to show statistically significant evidence that human beings have precognitive powers.

When other researchers tried to replicate the “Feeling the Future” experiments, they failed to find this statistically significant evidence.  However, according to this episode of Veritasium, the American Psychological Association had a policy at the time that they would not publish replication studies, and so they would not publish any of the research debunking the original “Feeling the Future” paper (I do not know if they still have that policy—I would hope that they do not).

The act of repeating experiments to see if anything different happens is a crucial part of how science works.  Or rather how it should work.  But this is not being done often enough, it seems.  And on those rare occasions when replication studies are performed (and published), a shocking number of high profile research turns out to be non-replicable.  This article from Vox.com sums up just how bad the replication crisis is:

One 2015 attempt to reproduce 100 psychology studies was able to replicate only 39 of them.  A big international effort in 2018 to reproduce prominent studies found that 14 of the 28 replicated, and an attempt to replicate studies from top journals Nature and Science found that 13 of the 21 results looked at could be reproduced.

That same Vox.com article calls the replication crisis “an ongoing rot in the scientific process.”

But as I’ve been trying to say in several of my recent posts, science is self-correcting.  With the introduction of metascience—the scientific study of science itself—there is some hope that the root causes of the replication crisis can be identified, and perhaps changes can be made to the way the scientific community operates.

Sciency Words: Metascience

~ J.S. Pailly ~ 8 Comments

Hello, friends!  Welcome to Sciency Words, a special series here on Planet Pailly where we talk about those weird and wonderful words scientists use.  Today on Sciency Words, we’re talking about:

METASCIENCE

Metascience is when science “gets meta” and studies itself, with the specific aim of making published scientific research more accurate and trustworthy.  That goal, that stated purpose, is an important part of the definition.  Or at least it should be, according to this YouTube video by Professor Fiona Fidler.

You see, metascience overlaps with certain other fields of research, like the philosophy of science or the sociology of science.  But a key part of a metascientist’s job is to identify problems with the current culture and methodology of scientific research and try to figure out ways to make science better.

The word metascience can be traced back to the 1930’s, with the earliest known usage attributed to American philosopher and semiotician Charles William Morris.  But as an actual field of research, metascience is not nearly that old.  This 2005 paper entitled “Why Most Published Research Findings Are False” is apparently a foundational document for modern metascience (or at least that’s what Wikipedia told me).

For a few months now, I’ve been doing lots of research about research, trying to improve the way I do my own research as a science fiction writer, and also trying to better understand what can go right (and wrong) with science.  With that in mind, I’m surprised I didn’t come across this term sooner.  Now that I do know about metascience, a whole new world of metascientific research has been revealed to me.

Reading about metascience has been kind of unsettling for me, actually.  Modern science has a lot more problems than I realized; however, there are people out there working to identify and fix those problems, so that science can live up to its promises.  And that, I think, is a very encouraging thing to know.

Sciency Words: File-Drawer Effect

~ J.S. Pailly ~ 4 Comments

Hello, friends!  Welcome to Sciency Words, a special series here on Planet Pailly where we take a closer look at new and interesting scientific terms.  This week’s Sciency Word is:

THE FILE-DRAWER EFFECT

This came up as part of my ongoing research about research, and it’s another example of how scientific research can go wrong.

Okay, so let’s say I have this hypothesis: people who watch Star Trek are better at math than people who do not watch Star Trek.  Ten different research teams set up experiments to test my hypothesis.  Only one of those ten teams manages to find a statistically significant relationship between watching Star Trek and being good at math.

The other eight teams are unable to find a statistically significant relationship, conclude that this was a huge waste of time, and move on to researching other things.  They decide not to bother publishing any of their findings.  Instead, all that research gets stuffed into a file-drawer, never to see the light of day again.

Meanwhile, that one team that did find a statistically significant relationship… they do publish their findings.  It’s such an astonishing result!  How could they not?  Soon, their results are being reported all over the news, and every Star Trek forum on the Internet goes wild, and parents start forcing their kids to watch extra episodes of Star Trek so they’ll do better on their math homework.

But that one research paper is totally contradicted by all the other research—or it would be, if any of that other research had been published.  As a result, the scientific community—and the general public as well—now have a terribly skewed understanding of the relationship between watching Star Trek and being good at math.  This is the file-drawer effect, also known as publication bias, at work.

P.S.: I mean, I’ve watched a ton of Star Trek, and everyone knows I’m good at math.  That sort of anecdotal evidence, plus a single peer-reviewed research paper, should be enough to convince everybody!

Sciency Words: Academic Paper Mills

~ J.S. Pailly ~ 6 Comments

Hello, friends!  Welcome back to Sciency Words, a special series about those weird and wacky words scientists use.  In this week’s episode of Sciency Words, we’re talking about:

ACADEMIC PAPER MILLS

A paper mill is a factory that produces paper.  It’s a perfectly legitimate business.  An academic paper mill is a business that, in an almost factory-like manner, cranks out fraudulent academic papers.

This term came up in my ongoing research about research.  Academic paper mills are a growing concern in the scientific community.  An extraordinary number of these paper mill papers have gone through the peer review process and been published in highly respected journals.

Distressingly, even when the origins of a paper mill paper are exposed, publishers do not always make that clear.  As this article from Nature explains:

Publishers almost never explicitly declare on retraction notices that a particular study is fraudulent or was created by a company to order, because it is difficult to prove.

Even so, that same article from Nature says that at least 370 published papers have been retracted since January of 2020 due to their suspected paper mill origins.  Another 45 have been flagged with “expressions of concern” by the journals that published them.  And since academic journals started cracking down on paper mill papers, it seems that some researchers have decided to voluntarily retract their own research “without stating the reason for retraction.”

Based on what I read in that Nature article, as well as in other articles like this one from Chemistry World, I get the sense that this is a bigger problem in some scientific fields than it is in others.  Fields like biomedical science, computer science, and engineering seem to be getting paper milled the hardest—in other words, fields where there’s the most money to be made and where researchers are under the most pressure to rack up publication credits.

For my own purposes as a science fiction writer who wants to do his research, I read a fair number of academic papers: mostly papers on astrobiology and planetary science.  I doubt I have to worry much about paper mill papers in those fields.  There are, however, other red flags I know to look out for.

How to Be Smart

~ J.S. Pailly ~ 14 Comments

Hello, friends!

So there’s this anecdote I heard once, way back when I was a kid, about a math teacher who didn’t know the value of pi.  This teacher had to stop in the middle of class and look the number up in a book.  Naturally, this drew some snarky comments from the students.  The teacher replied, sagely: “Why should I waste valuable brain space on information I can easily look up?”

Why indeed?

I haven’t been doing much research lately.  Right now, I’m trying to pick the habit up again, and I thought I’d start by doing a little research on how to do research.  Specifically, I thought I could use a refresher course on how to tell the difference between facts and fabrications on the Internet.  I wound up reading several papers (this one, this one, and this one), and I still have at least one more paper (this one) that I want to read.  So what have I learned so far?

Well, the main take away from my research on research is that a lot of people implicitly share the philosophy of that math teacher who didn’t know the value of pi.  I may not know the answer, but I know where to find the answer, and in the end that’s good enough.  And maybe it is good enough, so long as you recognize that you’re getting your information from an external source.

Unfortunately, according to this paper from the Journal of Experimental Psychology, the act of using a search engine can trick our brains into thinking we know more than we actually we do.  In a series of memory-related tests, people tended to overestimate their “unplugged knowledge” and underestimate their dependency on Internet search engines.  You don’t even have to have successful search engine results to get this inflated knowledge ego.  As the paper explains:

The illusion of knowledge from Internet use appears to be driven by the act of searching.  The effect does not depend on previous success on a specific search engine, but rather generalizes to less popular search engines as well (Experiment 4a).  It persists when the queries posed to the search engine are not answered (Experiment 4b) and remains even in cases where the search query fails to provide relevant answers or even any results at all (Experiment 4c).

I don’t think the lesson here is that we should stop using the Internet for research.  Rather, I think the lesson is that we need to stay humble.  It’s a little too easy to forget where our information comes from when information comes so easily through the Internet.  Unlike that math teacher who had to spend time flipping through a book to find the value of pi, I can just google it—or, faster yet, I can ask Siri.  But that does not mean I actually know the answer any better than that math teacher did.

P.S.: Yes, I did all my research for today’s post using Google.

Blue People Don’t Know That They’re Blue

~ J.S. Pailly ~ 6 Comments

Hello, friends!

So before anybody asks: no, the Eiffel 65 song “Blue (Da Ba Dee)” was not inspired by the work of Benjamin Lee Whorf.  I checked.  The two things are totally unrelated.

I told you about Benjamin Whorf in last week’s episode of Sciency Words.  Whorf, along with Edward Sapir, was one of the key researchers in the development of the linguistic relativity hypothesis (a.k.a. the Sapir-Whorf hypothesis).  While reading Sapir and Whorf’s original papers on the topic (from 1929 and 1940, respectively), I noticed something.  The two of them seem to spend more time defending linguistics in general as a legitimate science than they do explaining or defending linguistic relativity in particular.

I can imagine the kinds of skepticism early linguists must have faced.  I mean, language is just language, right?  You learn your grammar.  You learn your vocabulary.  And that’s that.  What more is there for this “science” of linguistics to study?

In his 1940 paper, titled “Science and Linguistics,” Whorf uses an analogy.  Imagine a race of people who, for whatever reason, are only capable of seeing the color blue.  They can see light blues and dark blues and medium blues.  The variety of shades of blue they can see must seem very impressive to them.  But in the end, all they see is blue.  If such a race of people existed, Whorf tells us, you can expect that their language would not have a word for blue.  These blue people would have absolutely no concept of blueness.  How could they?  How could anyone have a concept of blueness unless they could compare and contrast blue with other colors?

In a similar way, linguists compare and contrast languages, and by doing this they can learn far more about how language works, beyond the obvious grammar and vocabulary stuff.

Benjamin Whorf seems to have been pretty optimistic about the future of this relatively new science called linguistics.  As someone writing about 80 year later, I can say Whorf’s optimism was well founded.  Linguistics is, in my opinion, one of the coolest sciences we have!

P.S.: And seriously, the Eiffel 65 song was not inspired by Whorf, or intended as a tribute to Whorf, or anything like that.  It seems like it must have been, but apparently it was not.

Do Planets Have Genders?

~ J.S. Pailly ~ 17 Comments

Hello, friends!

So a while back, I got some unsolicited feedback from a person I know in real life.  This person had seen one of the illustrations I’d drawn for this blog, and she was incensed—absolutely incensed—that I would depict the planet Saturn as female.  You see, Saturn is a very masculine planet.  That’s a fact, apparently.

A lot of my thinking about planets—including my thinking on the gender identities of planets—was shaped by a book called Venus Revealed, by David Grinspoon.  That book was my first serious introduction to planetary science.  In a section titled “Men are from Venus, Women are from Mars,” Grinspoon has this to say:

At first I tried being completely gender-neutral in my writing, but this was unsatisfying because, to me, Venus is not just a “thing.”  Venus is not, in my mind, inanimate, and so “Cousin It” will never do to describe him… or her.

In that same section, Grinspoon does a little cross-cultural analysis and finds that Venus has been “a real gender bender” across human cultures and human history.  Sometimes she’s male; other times he’s female, depending on which mythological tradition you’re looking at.  And some cultures have apparently assigned different genders to the Morning Star and Evening Star, thus effectively making Venus genderfluid.

So do planet’s have genders?  No, of course not.  But much like David Grinspoon, I can’t see the planets as purely inanimate objects.  Planets have too much personality for that.  And since I think of the planets as having personalities, then, for better or worse, I also think of them as having genders.

For purely arbitrary reasons, I tend to think of Saturn as female.  But if you’d prefer to think of Saturn as male, or as something else entirely, that’s okay.  I’m not going to fight you over it.  I can love Saturn (and all the other planets, too) just the same, no matter what gender identities we pretend they have.

P.S.: While doing research for this post, I ended up reading a lot about how astrology assigns genders to planets (and also to numbers, elements, constellations, etc). I don’t want to dive too far down that particular rabbit hole, but I thought I should at least share this article on the subject. I used to think astrology was just silly. Now I think it’s problematic for reasons that go beyond mere pseudoscience.

Is There Life on Earth?

~ J.S. Pailly ~ 5 Comments

Hello, friends!

Let’s imagine some space aliens are cruising by our Solar System.  They turn their scanners on our planet and see… what?

Among other things, they’d notice that Earth’s landmasses are partially covered with a strange, green-colored substance.  Of course, you and I know what that green substance is.  It’s chlorophyll.  But would those extraterrestrial observers, who have no prior knowledge of our planet, be able to figure that out?  Even if they did, would they realize what chlorophyll is used for?  Maybe.  Probably not, though.

Which brings me to my all-time favorite scientific paper: “A search for life on Earth from the Galileo spacecraft,” by Carl Sagan et al.  I love this paper in part because it’s so clearly and concisely written, with jargon kept to a minimum.  Sagan was, after all, a talented science communicator.  But I also love this paper because its conclusions are so shocking, so eye-opening.

In 1990, NASA’s Galileo spacecraft turned all its high-tech instruments toward Earth and detected… not much, actually.  Galileo did pick up radio broadcasts emanating from the planet’s surface.  Aside from that, though, Galileo’s data offered highly suggestive (but also highly circumstantial) evidence on Earthly life.  The lesson: finding life on other planets is hard.  Even using our very best equipment, it was hard for NASA to detect signs of life right here on Earth!

At least that’s what I got out of reading Sagan’s Galileo experiment paper.  And based on various commentaries I’ve read or heard about this paper, that seems to be the lesson other people got out of it too.  So I was surprised to hear Sagan himself, approximately seven-and-a-half minutes into this interview, saying the exact opposite.

We’ve flown by some sixty worlds.  We claim that we haven’t found life anywhere, and that that is a significant result.  That is, that we would have found life had it been there.  But this has never been calibrated.  We’ve never flown by the Earth with a modern interplanetary spacecraft, all instruments on, and detected life here.  And so Galileo, because of this peculiar gravity assist VEEGA trajectory, permits us to do that.  And as I’ll describe tomorrow, we find life five or six different ways, including intelligent life.  And this then means that the negative results that we find elsewhere are, in fact, significant.

I’ve been puzzled by this for a while now, but I think I’ve finally figured out why Sagan would say this.  It’s politics.

On the one hand, scientists need to understand the challenges they’ll face (including the limitations of their own equipment) in searching for life on other worlds.  That really is, I think, the purpose of the Galileo experiment paper.  On the other hand, it would not do to say on public television, to cantankerous taxpayers and the listening ears of Congress, that NASA spends millions of dollars on space probes that are not even capable of detecting life right here on Earth.

Space exploration is expensive.  And like all expensive types of research, sooner or later the researchers involved have to learn how to play politics.

No Gospel Truth in Science

~ J.S. Pailly ~ 5 Comments

Hello, friends!

So there’s this notion in the popular press that when a new scientific paper comes out, that paper should be taken as the final definitive word on an issue.  Science has spoken.  This is a scientific fact now.  But that is not how science works.

When new research is published, you should expect there will be followup research, and then that followup research will be followed up by even more research.  A new scientific paper really shouldn’t be seen as a proclamation of fact but rather as the beginning of a dialogue among scientists, or perhaps as the continuation of a dialogue that’s already in progress.

The recent detection of phosphine in the atmosphere of Venus has turned out to be a fantastic example of this ongoing dialogue in action.  The initial research was published in two separate papers (click here or here).  Basically, astronomers found the spectral signature of phosphine (PH3) in the Venusian atmosphere, and they were at a loss to explain where all that phosphine could be coming from.

Based on everything we currently know about Venus, those two papers tried to rule out several possible explanations.  Such a large quantity of phosphine could not be created by Venus’s atmospheric chemistry.  It could not be spewing out of volcanoes on Venus’s surface.  It could not be delivered to Venus by asteroids or comets.  One very intriguing possibility that could not be ruled out: maybe there’s life on Venus.  On Earth, phosphine is produced almost exclusively by living things.

But those two papers were not the definitive final word on the matter.  A dialogue had begun.  Soon, followup research came out suggesting that phosphine could be spewing out of volcanoes after all.  It would still be pretty shocking to discover that Venus has enough active volcanoes to produce that much atmospheric phosphine—but it be nowhere near as shocking as discovering Venus has life.

And then even more followup research came out with this paper, which points out possible errors in the original research and suggests that we may be dealing with a false positive detection.  Venus might not have phosphine after all, or maybe it doesn’t have as much as originally believed.

And the dialogue continues.  More research will come.  More responses will be published, and then there will be responses to those responses, and so forth until the scientific community reaches some sort of consensus about this Venusian phosphine business.  And even then, that scientific consensus still might not be the 100% final word on the matter.

Based on the way the popular press reports science news, you could easily get the impression that scientific papers should be treated as gospel truth.  You would be understandably confused, then, when one scientific paper comes out refuting the findings of another.  Subsequently, you may come to the conclusion (as a great many people apparently have) that science must not know anything at all.  Science just keeps contradicting itself, it seems.

But scientific papers are not meant to be taken as gospel truth.  They’re part of an ongoing back-and-forth dialogue.  So the next time you hear about some new scientific discovery on the news, remember that scientific papers are not intended to be bold proclamations of fact.  And when you hear about some new paper refuting older research, you’ll understand what’s going on.