Vaccine Science: Evaluating Scientific Information and Studies

Evaluating media reports

Most of us get our information about topics related to science or health through media outlets. Sometimes it can seem like new studies are being reported every day and at times they directly conflict with other reports. When evaluating a media report, whether on the Internet, in print or on TV or radio, check for the following:

  • Is the organization providing the information reliable?
  • Is the report based on a scientific study or a personal anecdote (an individual story)?
  • If the report is about a scientific study, evaluate the information provided about the study using the criteria outlined in “Evaluating scientific studies” below. A good media report will not only include information about where the study was published, but also information about the way the study was done and the size of the study.
  • Who is interviewed or quoted? Is the person a scientific or medical expert? How much information is provided about the person being interviewed?
  • Are the people being interviewed sharing anecdotes or are they talking about the data or research? While it is sometimes good to hear from people who are personally affected by the topic, it is important to distinguish between an opinion based on someone’s experiences or biases, and a scientific evaluation of the strengths and limitations of the study. If you have ever read a scientific paper, you might recall that the paper not only included the findings of the study, but also its limitations. As a result, a study author or scientist being quoted in a media report will typically be quite specific in what they are willing to state and will typically reference the data, whereas someone voicing an opinion, especially someone with a personal bias, tends to speak broadly, sometimes delivering more assumptions, general statements, or emotion than facts.

For example, speaking about the same study, a scientist might say, “This study shows that drinking 100 cups of coffee every day for 10 years leads to a two-fold increase in the risk of developing stomach cancer.” Whereas, a person from the anti-coffee drinker’s club might say “This study confirms that drinking coffee causes cancer.”

  • Journalists often talk about presenting a balanced story. However, a few caveats are important to remember here:
    • An expert and a parent might be included so as to present a balanced story, but if one is motivated by data and the other a personal experience, this is not balance. It is scientifically based versus emotionally based information.
    • Consider the size and expertise of the group supporting each side of a story.
    • Which position is supported by scientific bodies or other researchers in the field?
    • The goal of a journalist is to appease a large audience. One of the tools that allows for ratings or skyrocketing readership in the media is controversy. Painting an accurate picture may be secondary to the goal of “getting a reaction.”

Evaluating scientific studies

Sound scientific studies have the following characteristics:

  • Random
    A study is randomized when the participants are separated into control and test groups in a random manner, such as by a pre-determined formula or software. By randomly assigning study participants, scientists decrease the possibility for biased results.
  • Double-blind
    In double-blind studies, both the study participants and the scientists are unaware of whether the participant is in the control or test group. For example, in some clinical trials, neither the researchers giving the treatment nor the study participants receiving it know if they are receiving a placebo (the control group) or the drug or vaccine (the test group). Double-blind studies are the most reliable because they eliminate potential for bias on the part of both the researchers and the participants.

    Sometimes, however, it is impossible to perform a double-blinded study. An example would be a study evaluating the best way to provide a patient with verbal instructions for taking a medication. In this case, the researcher will know which version of text was used, but the patients will not know whether they are in the test or control group. When only the study participants are unaware of the group they’ve been assigned to, it is called a single-blinded study. Sometimes, it’s unethical to do a double-blind, placebo-controlled study, such as the evaluation of Ebola vaccines during the 2014 outbreak of Ebola in West Africa.

    In rare instances both the researcher and the participant know the group to which the patient has been assigned, such as when testing a new cancer treatment in someone who has no other options for treatment.

    While the double-blind study design is considered the gold standard, this format may not always be an ethical or technically possible option. In these instances a single-blinded or un-blinded study format may be employed. As you read media reports about studies, it is important to determine the study method used.
  • Large sample size
    Large sample sizes allow researchers to account for individual differences such as genetics, income, race and environmental or lifestyle choices.
  • Multiple studies
    Study results must be repeatable in order to be widely accepted. If a researcher tries to replicate a study’s findings and fails, it is possible that an intentional or unintentional difference was introduced that caused the different findings. Many researchers will look at similar questions in different ways; only when a finding has been reproduced many times in a variety of populations is it widely accepted.

The best way to determine the strength of a study is to read the original paper. However, because most of us do not have the time or expertise to evaluate all scientific studies that are published each week, we rely on others, such as news outlets, to share accurate assessments with us. Therefore, these organizations should be held to high standards, and as consumers, we should assess each statement made in reports of scientific topics based on the criteria mentioned above.

Additional resources

Journal articles

Internet

Books

  • Nibbling on Einstein’s Brain: The Good, The Bad & The Bogus in Science by Diane Swanson, 2001
  • Unscientific America: How Scientific Illiteracy Threatens Our Future by Chris Mooney and Sheril Kirshenbaum, 2010.

Reviewed by Paul A. Offit, MD on May 20, 2020

Materials in this section are updated as new information and vaccines become available. The Vaccine Education Center staff regularly reviews materials for accuracy.

You should not consider the information in this site to be specific, professional medical advice for your personal health or for your family's personal health. You should not use it to replace any relationship with a physician or other qualified healthcare professional. For medical concerns, including decisions about vaccinations, medications and other treatments, you should always consult your physician or, in serious cases, seek immediate assistance from emergency personnel.