A new scientific paper is published about every 20 seconds; this means thousands of papers are published every day. Although most of these papers are reviewed by other scientists prior to publication, some describe studies that are more robust than others. In addition, most people don’t read the actual studies, but rather hear about them through media reports. Since some of these studies report science that may be important for keeping us safe and healthy, it’s important to understand the essential components of a sound study.
Studies typically include two groups, the control group and an experimental — or treatment — group. These groups are similar with one exception. Participants in the control group are not subjected to the experimental procedure. Controls are important because they provide information about what would occur without any outside influence. This allows scientists to determine whether the results would have occurred anyway.
Ask yourself: Did the study contain a control group that did not receive treatment and an experimental group?
Randomization, or the act of randomly assigning study participants to control and experimental groups, is an important tool used to avoid bias in the study. Scientists can use computer software or a pre-determined formula to assign participants to the control or experimental group.
Ask yourself: Were the study participants put into control and experimental groups in a random manner?
Double-blind studies are those in which both the scientists and study participants are unaware of who is assigned to the control and experimental groups. Double-blind studies are important for avoiding bias among both the scientists and study participants. While double-blind studies are the best option, sometimes, it’s impossible to design a study in this way. For example, if a new medication for end-stage cancer patients is being tested and a particular patient has exhausted all other treatment options, it would be unethical to withhold a medication that might help. In this case, it is likely that both the researcher and the patient would know that he or she was in the experimental group. Studies in which the scientist, but not the participants, knows who is in the treatment group can still be informative.
Ask yourself: Were both the scientists and study participants unaware of who was assigned to the control and experimental groups?
Large sample size
Many factors, such as genetics, environmental factors, lifestyle choices, income and race, can influence the results of a study. For this reason, studies often use large sample sizes to account for differences among individuals in a population.
Ask yourself: Was the study completed using a large number of study participants?
Sometimes a study may suggest a result that is different from previous studies. Therefore, it’s important for studies to be repeated multiple times by different scientists in a variety of populations. When study results can be reproduced, it ensures the results were real and not just an anomaly of one particular experiment. A study’s results should only be widely accepted after they have been reproduced.
Ask yourself: Was the study the first of its kind or was it repeated numerous times with similar results?
Because people typically read about studies in media reports, it’s important to evaluate the story for the details mentioned above. If these details are not reported, consider seeking out additional information to gain a better understanding of the study’s scientific merits.
Learn more about evaluating scientific information and media reports »
Read "Twenty tips for interpreting scientific claims," published in Nature magazine »