Good vs. bad science journalism
(8 minute read)
Most of us are not experts in any scientific field. So we would have a very tough time reading and understanding the new research published in scientific journals. Even a scientist can’t be an expert in everything. A geologist doesn’t typically read medical journals. Each field has its own jargon and base knowledge that is shared by the people who study it, and primary sources such as journal articles are written for people who are already in the know.
All of us, scientists and non-scientists alike, have to rely on secondary sources – books, news media, documentaries, and websites – to interpret the new results as they come out. This is how we can stay up to date on important advances in a wide range of fields. The distillation of technical, hard-to-understand information into general-public-friendly information is what I am calling science journalism.
But not all science journalism is created equal. Just because something is on the Internet (or, unfortunately, on the news) does not mean that it is true or accurate. We need to be careful about the sources that we believe. We all have a responsibility to read or watch critically, so we can recognize the fluff or the pseudoscience when we see it. Here are a few questions to keep in mind to help you sift good science journalism from bad.
Is this person trying to sell me something?
This is an easy one. If the purpose of the article (or TV spot, or whatever) is to get you to buy something, treat it with caution. For example, if you do an Internet search for which pesticides are safe to use in your backyard, you might find an article that seems fairly convincing. But then if you realize the article is on a website that sells those pesticides, maybe look elsewhere for a second opinion.
Where did they get their information?
Does the writer (or speaker) cite their sources? Do they mention specific studies, or interview experts in the field? Do they name names so that people like you can easily fact-check them?[1]
Do their sources seem credible?
If the source of information is a person, is their job title and place of work given? What are their credentials? Do they seem qualified to speak on this topic?
If the source is a study, what were the methods used? Do those methods seem like a reasonable way to collect the desired information?
And perhaps most importantly, does the writer (or speaker) reference more than one source to draw their conclusions? We as the general public should only be interested in the findings that are supported multiple times in many ways and by many different researchers.
For example, if you see a click-bait article that says something like: “Drinking coffee helps you live longer!” it’s probably not true, or at least very exaggerated. If a source is given, it’s most likely a single study and the results probably only hinted at a correlation between coffee and lifespan. But a correlation does not prove that one thing caused the other. We would need many studies done in a variety of ways to really prove that there is a link. But websites want to sell ads, so they take one study from one paper, and write it up in a way that makes it seem more exciting or important than it is. So you’re better off ignoring the sites that are always trying to publish the newest, flashiest research (because often it hasn’t been verified yet).
Did they build a clear, easy-to-follow bridge from the facts to their conclusion?
A good science journalist will give enough detail to show you that they’ve done their homework. They will have consulted many sources and will have learned everything they can. They will explain, in words you understand, what the evidence has to say. Then they will build a case to show you how the facts lead to the main idea of the piece.
Which is more convincing?
Carbs are bad. Don’t eat carbs. Arrrgggh.
When you eat carbohydrates, your blood glucose level goes up. In response, your body releases insulin, which tells your cells to use up the glucose for fuel. Eating too many carbs can lead to insulin resistance, which can lead to metabolic disease, weight gain, and diabetes.
Hopefully, the second one is more convincing. Which is not to say that wordier and more technical is better, just that the line of reasoning needs to be clear.
Is their main point easy to understand?
If you are in the intended audience for a piece of science journalism, yet you find it hard to follow or there is too much jargon, then it’s not good journalism. The writer (or speaker) has failed at their mission of passing new, interesting, or useful knowledge to you.
Furthermore, there should be a clear takeaway message from the piece. You’re not reading (or watching, or listening) to memorize the details of how a study was done or how scientists figured something out. You’re reading to learn the important thing they discovered that applies to you as a regular person.
. . .
Now that we’ve talked about what to look for, here’s an example of bad science journalism. I Googled “benefits of Himalayan salt lamps,” and the first article I saw was called “12 Reasons to Keep a Himalayan Salt Lamp in Every Room of Your House.”[2] It tells you all these wonderful things about salt lamps, like how they apparently purify your air, boost your mood, help you sleep better, and… balance electromagnetic radiation? Speaking as someone with an advanced degree in chemistry, that last one seems particularly ridiculous.
None of these claims were adequately supported by evidence. Under some bullet points, they used the words “studies have shown,” but they never cite a specific study or give any quotes from experts. So that is a warning sign. Plus the explanations that they give are fairly flimsy. Things like, “releases negative ions that purify your air,” but no indication of how negative ions might affect air quality. Again, speaking as a chemist, that claim makes no sense.
Then at the bottom of the article, they give a recommendation for “our favorite Himalayan salt lamp,” and why you should buy it. Another red flag.
When I went back to the search page and looked at the second, third, and fourth hits, they were all articles by health websites debunking all of the claims I just mentioned. So when you look for more evidence for the benefits of salt lamps, it seems there isn’t any.
. . .
And here’s an example of good science journalism. I Googled “benefits of exercise,” and I found an article on a website called Medical News Today.[3] The author lists many of the good things that can happen if you exercise: improved energy levels, better sleep, healthier weight, etc. The biggest difference from the salt lamp article was that under each bullet point, it specifically says “according to ___,” and mentions a national health organization that backs up each claim.
For example, under the bullet point about exercise helping to manage diabetes, the article includes a link to a document by the American Diabetes Association laying out the evidence that supports the claim. And clearly, the American Diabetes Association is an authority on the subject. And it wasn’t just this one instance – every claim made in the article included references and links for further information. Plus it was written clearly and easy to understand.
I will mention that when you scroll to the bottom of the article I am talking about, there is a paid advertisement for a weight loss app (or at least there was when I looked at it). But selling ad space on a website is not the same thing as the website itself selling the stuff in the article.
. . .
If you want to see examples of really good science journalism on more cutting-edge topics, look up Ed Yong of The Atlantic.[4] He has been writing on the coronavirus pandemic since early 2020, and won a Pulitzer Prize for this work in 2021. Each of his articles is so packed with information. It’s clear that he speaks to many, many experts when putting a piece together. He quotes them and includes their credentials, and then explains how the things each of them say tie together. By the end of each article, you know exactly how he arrived at his conclusions and what his main point is. And that’s the real purpose of science journalism: to convey relevant, reliable information to the people who need to know it.
In summary:
I’m not trying to tell you to doubt everything that you read (or watch, or hear). I’m only trying to plant some questions in your mind for when you come across new information. Where did the author get their info? Do they have multiple sources, and are the sources credible? Did they make their point clear and convincing? Am I confident they’re not just trying to sell me something? If the answer to any of these is no, take a little extra time to look around for more evidence before you decide what to believe.
[1] I realize it’s ironic that in an article where I tell you to question whether an author cites their sources, I have no sources listed. This particular piece is based solely on my experience as a scientist, an educator, and a consumer of science-related media.
[2] I won’t put a link here in case that can be misconstrued as an endorsement of the website. But I’m sure you can find it pretty easily if you wanted to take a look.
[3] https://www.medicalnewstoday.com/articles/benefits-of-exercise#diabetes
