Feature Article — No. This Study Does Not Prove What You Think It Does: Part III

Editorial note: This series was originally published in the Parents PACK newsletter, a free monthly e-newsletter for the public that addresses vaccines and related topics. To learn more about the program, visit vaccine.chop.edu/parents.

In part 1 of of this series, we discussed some common misconceptions about science and scientists. In part II, we discussed some surprising findings that were initially not believed but were ultimately proven to change our previously established understanding of science.

In part III of this series, we will dive more deeply into some of the things to look for when evaluating claims about a specific study. We’ll do this by highlighting a few papers that caused fear, generated confusion, or spread misinformation by looking at the interpretation, what it was based on, and why it was not correct.

No. This study does not prove that COVID-19 vaccines cause antibody-dependent enhancement.

Study

Ricke DO. Two different antibody-dependent enhancement (ADE) risks for SARS-CoV-2 antibodies. Front Immunol. 2021 Feb 24;12:640093.

Brief summary

This paper suggested that COVID-19 vaccines that cause antibody production may cause a condition known as antibody-dependent enhancement (ADE). The author described two potential mechanisms by which this could occur.

Misconception

ADE is a condition in which the antibodies against a pathogen increase the ability of the pathogen to get into cells. The antibody binds to the pathogen, but instead of blocking it from entering cells, it facilitates entry of the pathogen into cells. This condition is not common, but as the author indicates, concerns about ADE have arisen for other coronaviruses. Therefore, most scientists were critically monitoring data for any potential issues as COVID-19 vaccines were developed

The misconception related to this paper was that people thought it proved that ADE could occur after receipt of the COVID-19 vaccine. The paper does not provide such evidence. Here’s why:

1. This paper is not a research study; it is, at best, a modeling study. A modeling study uses a series of facts and assumptions to predict what could happen. As such, by definition, a modeling study cannot serve as proof. Indeed, if you look at the article in the journal in which it was published, Frontiers in Immunology, you will see that it is classified as a “Hypothesis and Theory” article. According to the journal’s information for authors, “Hypothesis and Theory articles present a novel argument, interpretation or model intended to introduce a new hypothesis or theory.”
2. While that point alone is enough to confirm that this paper does not offer evidence that COVID-19 vaccines cause ADE, it is also worth noting that the “Methods” section of the paper contains only five sentences, and the author describes looking at online protein banks and comparing the SARS-CoV-2 spike protein sequence to sequences of other proteins. Likewise, the “Results” section has five sentences. Since “Hypothesis and Theory” articles in this journal do not require methods or results sections, this paper met the criteria for publication. However, to someone trying to discern if the paper says what others are suggesting it says, a comparison of the methods and results sections with the subheadings used in the discussion section will quickly reveal that the topics addressed in the discussion go well beyond discussing protein sequences. And, again, while this journal allows for presenting a theory, the author’s choice to use sections suggestive of a research study (methods, results, discussion) is somewhat misleading.

Problem with interpretation:

Misclassification of article

No. This study does not prove that COVID-19 or the vaccines cause severe disease or ADE via autoimmunity.

Study

Lyons-Weiler J. Pathogenic priming likely contributes to serious and critical illness and mortality in COVID-19 via autoimmunity. J Transl Autoimmun. 2020 Apr 9;3:100051.

Brief summary

This paper compared protein sequences from the SARS-CoV-2 virus to protein sequences in humans and described where the latter are found in the body. With this information, the author suggested that the quantity of overlap in some sequences could cause the immune system to attack one’s own proteins, called autoimmunity, leading to more severe disease.

Misconception

As with the Ricke paper, this report presents a theory but not evidence of something happening. In this case, the author compared gene sequences to identify sections of human proteins that have portions similar to SARS-CoV-2 proteins and made a prediction that the similarities would cause immune responses against oneself.

To understand why this is unlikely, one needs to know a bit about protein chemistry:

• First, all proteins are made of building blocks known as amino acids. Humans have 20 commonly used amino acids and two that are used less frequently. Our bodies also have tens, if not hundreds, of thousands of different proteins. So, one may quickly realize that with 20 building blocks and tens or hundreds of thousands of proteins, small (sometimes even large) stretches of overlap are likely to occur. Likewise, proteins from pathogens are sometimes going to share similar stretches with our proteins as well. As such, finding similar stretches of genes is not novel, unlikely, or particularly surprising. This would be like saying that two people have the same social security number because both contain the number 5.
• Another aspect of protein chemistry is also important. It is the three-dimensional (3D) structure. Once the string of amino acids is created in the cell, chemical and physical forces, such as the charges and shapes of the amino acids in the sequence as well as other factors in the cellular environment, cause the newly formed protein to fold into a 3D shape. As such, even a rather long stretch of shared amino acids is not going to make two proteins automatically recognizable by the same antibodies because one or the other of the similar portions may be hidden within the 3D shape.
• Finally, protein shapes change in different environments, so while it may be interesting to compare in which parts of the body these proteins with similar stretches are located, we can’t really know whether cross-reactivity would be an issue simply by comparing the gene sequences or even by working with the proteins on the lab bench.

An analysis like the one in this paper may allow one to create an interesting hypothesis, but it does not provide evidence of a concern.

Three other points about this paper are worth noting:

1. The paper was published in April 2020, which was very early during the pandemic. Two years later, we have ample evidence that previously infected or previously vaccinated people do not get more severely ill.
2. The paper was published in the Journal of Translational Autoimmunity. While the articles in this journal are peer-reviewed and any journal can publish a strong or a weak paper, this particular journal has a low ranking compared to many journals, meaning that it tends to publish less influential papers. If this paper was indeed presenting a strong evidence-based case, it would have been published in a more notable journal, so that more scientists would see it.
3. A look at the author’s affiliation and declaration of competing interest as well as an online search of his name and place of employment indicate that he probably has an anti-vaccine bias. As we discussed in part I of this series, scientists who start a study with the end in mind are less likely to make significant scientific contributions.

Problem with interpretation

Misclassification of study and logical fallacies (hasty generalization and appeal to ignorance)

No. This study does not prove that RNA from the COVID-19 vaccine changes DNA.

Study

Alden M, Falla FO, Yang D, et al. Intracellular reverse transcription of Pfizer BioNTech COVID-19 mRNA vaccine BNT162b2 in vitro in human liver. Current Issues in Molecular Biology. 2022;44(3):1115-26.

Brief summary

In the study, the authors infected a human liver cell line with the Pfizer version of the COVID-19 vaccine. They used different concentrations of the vaccine and multiple timepoints to measure:

• Conversion of RNA into DNA in the cells
• The quantity of a genetic tool called LINE-1
• The presence of DNA sequences similar to the vaccine RNA in the nuclei of these cells

Misconception

Some pointed to this paper as proof that COVID-19 vaccines alter DNA. However, this is not an appropriate conclusion from this paper for several reasons:

1. Perhaps most importantly, this experiment was done on cells being grown in a lab. Said another way, it was an “in vitro” experiment. In vitro experiments are done all the time and they are important for providing information and clues as to what might happen in a person (“in vivo”). However, to make a conclusion about what is happening in people, one must have some evidence that it is actually happening in people, not just that it might be possible. The authors acknowledged this when they wrote, “At this stage, we do not know if DNA reverse transcribed from BNT162b2 is integrated into the cell genome. Further studies are needed …” (p. 1122). They go on to suggest two alternative experimental methods for getting more information.
2. The authors used a cancerous liver cell line. This is important for two reasons; both of which were acknowledged by the authors. First, cancerous cell lines replicate, whereas our liver cells typically are not replicating. As such, even if DNA representing the viral RNA was integrated into the cell, no other cells with the altered DNA would be produced. The authors also pointed out that this cell line has been shown to have genetic and protein expression differences specific to RNA metabolism (p. 1123). This means that what is seen in these cells may not be representative of what would happen in non-cancerous liver cells (or even a different line of cancerous liver cells). Second, they measured LINE-1 activity. Importantly, LINE-1 has been associated with various disease-related conditions, including cancer. It has also been shown to affect immune responses. For these reasons, while the changes related to LINE-1 are interesting, we can’t be sure the effects would be the same in a non-cancerous cell line. The authors also made this point by stating, “The exact regulation of LINE-1 activity in response to BNT162b2 merits further study.” (p. 1123).
3. Finally, as the authors pointed out, expression of LINE-1 has been shown to increase during viral infections, including with SARS-CoV-2 virus. In fact, some scientists have suggested that integration of SARS-CoV-2 genetic material into human cells could be why some people still test positive by PCR well after they have recovered from their infection. However, it is important to note that more information would be needed to prove this hypothesis as well. The more likely explanation is that the virus is undergoing an incomplete cycle of replication, where the genetic material (RNA) is produced but whole virus particles are not.

Problem with interpretation

Logical fallacy called hasty generalization (It is important to note in this case, that the misconception was not because of the quality of the science or the messaging of the authors, but rather because others took the findings out of context.)

Yes, this study showed viral shedding. No, that doesn’t mean vaccinated people are as contagious as unvaccinated people.

Study

Chia PY, Ong SWX, Chiew CJ, et al. Virological and serological kinetics of SARS-CoV-2 Delta variant vaccine breakthrough infections: a multicentre cohort study. Clin Microbiol Infect. 2022 Apr;28(4):612.e1-612.e7.

Brief summary

This study of 218 people infected with COVID-19 compared their vaccination status, severity of illness and blood samples. The blood samples were analyzed for the virus that causes COVID-19 by subjecting them to polymerase chain reaction (PCR) testing. Antibody levels were also measured in separate tests.

Misconception

At the heart of the misconceptions about this paper is figure 1, which shows viral shedding among vaccinated (green) and unvaccinated (red) individuals. Each dot on the table represents a study subject. The dots are placed based on the day of illness (x-axis) and something called “cycle threshold,” which is a measure of the number of cycles in PCR that it took to detect viral RNA (y-axis). The more virus in a sample, the fewer cycles it would take to detect RNA. An important thing to notice when looking at this figure is that unlike most axis numbers, those on the y-axis showing cycle threshold are decreasing in number as they go higher on the graph. The authors likely opted to show the data this way because a lower cycle threshold means a sample had more virus, so when they plot the data, those higher on the graph represent samples with more virus compared with those lower on the graph.

Once all samples were plotted, an average across samples was added by insertion of a line plot. On the figure, the two lines (comparing vaccinated and unvaccinated patient data) are almost in the same place for the first four or five days of illness, but then they separate, and the vaccinated group more quickly requires higher numbers of PCR cycles to detect viral RNA, meaning there was less virus in those samples.

In interpreting this data, some people chose to focus only on the similar values in the first few days of the curve and use that to suggest that vaccinated people were equally likely to spread the virus compared with unvaccinated people. And, while this may be the case early during infection, there are a few additional considerations when thinking about these data:

1. The data represent RNA levels not viral particles, so while the patient samples required equal cycles of PCR to detect RNA, we can’t speak to a person’s contagiousness because this study does not tell us how much whole virus was present in their nasal passages. This, of course, does not mean these data are not important. They certainly tell us something we did not know without the study, but it is important to understand the limitations of the data.
2. Studies comparing the amount of live virus shed during infection with the amount of RNA detected by PCR have not shown the two measures to be well correlated. 
3. Likewise, even if these data were found to be representative of one’s ability to spread the virus, it is not appropriate to dismiss the rest of the curve, which shows differences between vaccinated and unvaccinated people after day four or five. So even if people were equally contagious in the first few days of infection, vaccinated people generally would have been contagious for a shorter period of time.
4. Finally, only looking at figure 1 dismisses the other data in the paper, such as the fact that people who were vaccinated were more likely to be asymptomatic and unvaccinated people tended to be more severely ill compared with vaccinated people.

Problem with interpretation

Cherry-picking data

Conclusion

Hopefully these examples demonstrated some of the ways that a paper should be examined. Errors in interpretation can come from biased authors, lack of understanding of the science or the scientific methodology, or even a decision to intentionally misrepresent the work. As such, consumers of information should always consider messages with a questioning attitude. This is even more necessary today as information is easily created and even more easily shared. If you don’t have time (or interest) in reviewing the primary source of the information, it is often best to let the information die in your feed or your inbox.

This three-part series was aimed at describing several important aspects of science:

• In part I, we focused on the big picture.
• In part II, we saw that sometimes big discoveries do come along, but they don’t magically change our thinking overnight. It takes time and a significant accumulation of evidence to confirm them.
• In part III, we saw a few of the many ways that science can be misinterpreted and misused.

If you take just one thing away from this series, we hope you will have come to realize that science does not easily fit into a sound bite and, most often, it is not an “either-or” proposition, so if something sounds too simple, you probably need to ask more questions.

Resources for evaluating information

Find resources that can help when evaluating information.

Download a PDF version of this article.