Bad studies are those in which the standards of scientific protocol were not met. For example, a study may not use appropriate controls — or any controls for that matter. The study may have been done using a method that doesn’t make sense for the kind of question being asked, or inappropriate statistical tests could have been employed. Bad and misleading studies are published all the time. Bad studies can even be published in good journals. So what to do?
Dr. Offit talks about how to evaluate scientific studies using the two pillars of science — peer review and reproducibility.
Transcript
Paul Offit, MD: Hi, my name is Paul Offit. I am talking to you today from the Vaccine Education Center, here at Children’s Hospital of Philadelphia. I think one thing parents and doctors and the media, frankly, struggles with is what to do with studies that aren’t very good. So for example, there are in this world about 6,500 medical and scientific journals; those journals publish about 4,000 papers every day. So, those papers not surprisingly follow a bell-shaped curve, some are excellent, some are awful, most are more or less mediocre. But the point being that there is a lot of bad and misleading science published all the time. So, what to do? How can we sort all this out?
I think that scientific studies basically stand on two pillars. One is peer review, meaning that when you submit a paper, your paper is reviewed by people that are experts in your field who will recommend its publication or not. And the second — I think the stronger pillar on which it stands — is reproducibility, which is to say that if you’re right, that you’re ultimately be shown to be right when other investigators in other parts of this world or other parts of this country reproduce what you’ve found.
And it’s not even just that bad studies get published in bad journals, or so-called predatory journal; bad studies can be published in good journal. So for example, in the early 1980s there was a paper published in The New England Journal of Medicine, arguably the best clinical medical journal in the United States. They claimed that excess coffee drinking increase your risk of pancreatic cancer. The first author in that study was Brian MacMahon. He was at the Harvard School of Public Health; he was an epidemiologist and a good one. But that study was wrong. Excess coffee drinking didn’t cause an increased risk of pancreatic cancer, and we came to know that as more and more people tried to find what Brian MacMahon had found but couldn’t.
Another example would be a paper that was published in The Lancet in the late 1990s. This was published by a British intestinal surgeon named Andrew Wakefield, claiming that the combination measles, mumps, rubella vaccine caused autism. Now, The Lancet is an excellent journal; it’s the oldest, frankly, of the general medical journals. And again, this was a thin study at best, really nothing was studied; it was just a case series of eight children who had autism within a month of receiving the vaccine, proving, frankly, only that the MMR vaccine doesn’t prevent autism. And now 17 different studies have been published in seven different countries, on several different continents, involving hundreds of thousands of children, showing that that hypothesis was wrong.
So, I do think that my advice would be that when you see a study that makes a surprising or outlandish claim, wait — wait to see whether or not it’s reproduced by others, because that’s really the strongest pillar on which good science stands.
Thank you.
Related Centers and Programs: Vaccine Education Center