Dissecting complex, scientific literature is not for the faint of heart. Research, when done correctly, is nuanced and complicated. It's systematic and follows a formula to ensure accuracy, transparency, and replicability — also known as the peer-review process.
But Reading Science Stories Doesn't Have To Be Just for Scientists.
The onset of the COVID-19 pandemic brought with it a steady stream of questionable scientific research and studies, and our newsroom an onslaught of reader questions about the legitimacy of so-called scientific study. When determining whether a study was conducted using verifiable methodologies, we often turn to the question of whether the research was reviewed by experts and institutions in its respective scientific field.
To help you navigate the digital barrage of "science," let's start with a definition of peer review and how to navigate a legitimate scientific study.
What is a Peer-Reviewed Study?
Peer-reviewed articles are those that have gone through a regimented evaluation process by journal editors and other experts in the related field. These reviewers check a given study for accuracy and ensure that it follows the scientific method.
"Peer-reviewed journal articles have gone through an evaluation process in which journal editors and other expert scholars critically assess the quality and scientific merit of the article and its research. Articles that pass this process are published in the peer-reviewed literature," writes the National Library of Medicine (NIH).
This research may include experiments, surveys, or an analysis of existing research on a given topic, as examples. Peer review ensures the quality of the research and makes it accessible via academic, established databases, some of which may require a subscription.
Even in open-access journals that are available to the public without a paywall, studies can be overly complex or highly detailed to a person without a scientific background. That's where this guide comes into play.
What Constitutes Sound Science?
Sound science is defined as a systematic collection of data that tests a specific hypothesis.
"Scientific research is the neutral, systematic, planned, and multiple-step process that uses previously discovered facts to advance knowledge that does not exist in the literature," writes NIH.
"It can be classified as observational or experimental with respect to data collection techniques, descriptive or analytical with respect to causality, and prospective, retrospective, or cross-sectional with respect to time."
Sound science follows a standard of ethics and requires a certain level of transparency from researchers. It also largely follows the scientific method, which establishes a testable hypothesis or research question that is transparently answered in a study. In all cases, it is open to scientific criticism and the way in which the research was conducted is explained in a way that can be replicated by other scientists.
How to Read a Study
Understanding what constitutes a solid study helps our team to determine whether a claim is legitimate. A key indicator of peer-reviewed research is confirming the following elements in a study:
- Abstract: A concise summary of the entire research paper or thesis.
- Introduction or Background: This provides background information about the topic at hand, including previously published research related to the hypothesis.
- Materials and Methods: Explains how the research, analysis, or experiment was designed and conducted in a manner that is thorough and replicable.
- Results: The findings of the study.
- Discussion: This establishes the greater meaning of the study in the context of this scientific field and sets forth areas of future study.
- Limitations: This category defines characteristics of the study that influenced or impacted the study findings. These are often flaws or shortcomings that might be addressed in future studies.
- Funding: Transparently lists partners or institutions involved in the study.
- Conflicts of Interest: Transparently lists financial or personal considerations that may compromise the conduct of its study or its how it may be publicly perceived.
As a general rule, transparency is key and studies are filled with nuance and context. Reading the entirety of the study is akin to reading beyond the headline of a news article.
What Is Pre-Publication?
In times where science is urgent and a steady stream of information and data is essential to supporting public health, as was the case at the onset of the COVID-19 pandemic, experts will often forego the sometimes-lengthy peer-review process to simply get their findings online for other researchers to reference.
A study in preprint means that it is in its early stage of production and has not yet been peer-critiqued by other scientists. As we has previously explained, research that has not been peer-reviewed has not been rigorously vetted for discrepancies, limitations, and other errors in observations or recording:
Research that has not been peer-reviewed is akin to a blog — anyone can publish one online with little expertise. A reviewed study, on the other hand, is on par with a well-vetted, expertly researched textbook. To facilitate the quick spread of new scientific information in the midst of the pandemic, prepublication research has become more common in order to facilitate the quick dissemination of important information, particularly as peer-reviewed research can take a long time to publish.
Two preprint journals our reporters keep their eyes on are Zenodo and arXiv, both of which allow scientists to submit their research directly without peer review.
What Is a Case Study?
Case studies are different than regimented scientific research in that they describe a specific event and do not establish a causation or correlation between one event and another. BMJ Case Reports is one such example that will publish case studies related to the medical world. As with pre-publications, case reports are a great way to push the science out into the broader community to establish areas for future experimentation and warning signs.
Illegitimate Study Red Flags
As a study will have gone through a thorough editorial vetting and standard publishing process before publication, typos and English grammar errors can sometimes point to the illegitimacy of a study. However, this is not always the case, particularly in non-Western publication houses. That's why it's important to look for several red flags, including a poorly designed website and lack of transparency in who is responsible for reviewing submissions and standards for submitting, as well as an inability to contact the journal or institution.
Also be on the lookout for processing charges required of submitters. In some cases, predatory publishing websites will require a researcher to pay a fee for their research to be published.
A Step-by-Step Guide to Determining Whether a Published Study is Peer-Reviewed
The University of Pittsburg Library System has published a step-by-step guide to help readers identify illegitimate and predatory publishers. We've summarized their advice below:
- First search for the journal in the database list in the "Journal Databases for At-Home Research" section below.
- If the journal did not appear in the database, look for the following red flags on its website:
- What is the timeline from submission to publication? Peer editing is a long process and quick turn-and-burn times can be indicatory of a predatory publisher.
- Look through the website content. What is the quality of both archived and published articles? Is the material and formatting standardized?
- Determine usage fees. As the University of Pittsburgh points out, publication fees can be standard, but submission fees are not. Legitimate sites may also require a user pay for access to a website.
- Research the journal editors. In some cases, predatory publishers may list academics as editors without their knowledge. Look through the list of editors presented on the website and first determine if they are real. If they are, check their websites or resumes to see if the journal in question is listed.
Journal Databases for At-Home Research
- Directory of Open Access Journals
- Open access simply means that the published research is publicly available and free to reuse and redistribute (as opposed to paywall or subscription sites).
- Committee on Publication Ethics (COPE)
- SHERPA/RoMEO
- Open Access Scholarly Publishers Association
- International Association of Scientific, Technical, and Medical Publishers (STM)
- Journals Online through (INASP)
- African Journals Online (AJOL)
- BioMed Central
- PubMed Central
- Think, Check, Submit
Aggregated Science News, Media Advisories, and Published Peer-Reviewed Studies:
This page is part of an ongoing effort by the Snopes newsroom to teach the public the ins and outs of online fact-checking and, as a result, strengthen people's media literacy skills. Misinformation is everyone’s problem. The more we can all get involved, the better job we can do combating it. Have a question about how we do what we do? Let us know.
Sources
"BMJ Case Reports." BMJ Case Reports, https://casereports.bmj.com/pages/. Accessed 30 Mar. 2022.
Collister, Lauren. LibGuides: Illegitimate & Predatory Publishing: Legitimate vs Illegitimate Publishers. https://pitt.libguides.com/c.php?g=718064&p=5112543. Accessed 30 Mar. 2022.
"Did Chinese Virologist Dr. Li-Meng Yan Say COVID-19 Was Made in a Wuhan Lab?" Snopes.Com, https://www.snopes.com/fact-check/li-meng-yan-covid-19-lab/. Accessed 30 Mar. 2022.
"Did Scientists Find Mushrooms on Mars?" Snopes.Com, https://www.snopes.com/fact-check/mushrooms-on-mars/. Accessed 30 Mar. 2022.
EROL, Almıla. "How to Conduct Scientific Research?" Archives of Neuropsychiatry, vol. 54, no. 2, June 2017, pp. 97–98. PubMed Central, https://doi.org/10.5152/npa.2017.0120102.
Peer-Reviewed Literature. https://www.nlm.nih.gov/nichsr/stats_tutorial/section3/mod6_peer.html. Accessed 30 Mar. 2022.
"This Headline Doesn't Tell You Everything You Need To Know." Snopes.Com, https://www.snopes.com/articles/401035/headlines-articles-context/. Accessed 30 Mar. 2022.