Some of the connections that doctors and researchers hold so dear – links between certain biomarkers like gene variations and the diseases that they are thought to predict – may not be so strong after all. According to a large new review, many of these connections no longer hold water, meaning that doctors will need to adjust the recommendations they make for their patients based on their labs and blood tests.
Researchers reviewed 35 of the most widely-cited studies in medicine, which had been published in some of the top journals. They found that less than half of the connections between specific biomarkers, like genes and levels of blood proteins, and certain diseases were supported by later follow-up studies. "We found that a large majority of these highly cited papers suggested substantially stronger effects than that found in the largest study of the same markers and outcomes," said author John Ioannidis.
They found that less than half of the connections between specific biomarkers, like genes and levels of blood proteins, and certain diseases were supported by later follow-up studies.
This means that in many cases, the research community is holding on to obsolete findings, just because they were the first or perhaps the splashiest studies on a particular subject. This is ironic, because researchers know that replication or repeatability is one of the central tenets of science — that is, a finding must be supported by follow-up studies to really hold weight.
Once a finding is published, especially if it is pioneering and published in a major journal, it tends to be cited by author after author (not to mention the media) in a kind of snowball effect. This phenomenon makes the original finding appear as gospel, even it if it not backed up by later studies.
Sometimes a connection may be reported even though the results are really just due to chance: suppose you flip a coin ten times with your left hand. Occasionally (if rarely) they’ll all come up heads. This might lead you to say that there’s a strong connection between left-handed coin-flipping and the coin landing on heads, when really it’s just luck. The same thing can happen in science, which is why more studies (i.e., more coin-flipping) must be done to either confirm or invalidate the original study.
'We need to follow the scientific method through to the end and demand replication and verification of results before accepting them as fact.'
Sometimes the explanation is not so innocent. Ioannidis says that "[r]esearchers tend to play with their data sets, and to analyze them in creative ways. We’re certainly not pointing out any one investigator with this study; it’s just the societal norm of science to operate in that fashion. But we need to follow the scientific method through to the end and demand replication and verification of results before accepting them as fact."
Ioannidis suggests that there be an ongoing way to track new research in a particular area. As each new result comes in on a specific marker and its disease, it could feed into a larger analysis so that the connection could be evaluated as it develops. Time will tell how the scientific community adjusts for these findings. In the meantime, doctors will need to be more careful about how they interpret a patient’s test results, and refer to the latest – not the biggest – studies on a particular subject.
Ioannidis is a researcher at Stanford University; the results are published in the June 2011 issue of the Journal of the American Medical Association.