What's Good Health Information?

The wide availability of health information — on television, in print, on websites and via search engines — brings with it a new set of challenges for consumers of medical and health information. Sometimes it seems as though we are bombarded with health information. A lot of it is hyped, taken out of context or boiled down to a sensational headline. The Internet— with its thousands of sites putting out information whose quality can be difficult to judge — brings its own new set of challenges for consumers of medical and health information.

Consumers need to be able to evaluate different kinds of information and understand how they are generated.

How can a person know if the information they are getting is accurate?

Obviously, the source of the information is a big factor. The National Institutes of Health offer some tips for determining the quality of information on health and medical websites and a tutorial to help you evaluate medical websites.

But the source of information is not the only issue. Consumers need to be able to evaluate different kinds of information and understand how it was generated. Suppose you hear a reporter on the evening news saying, "A recent study has shown that exercise will add twenty years to your life." Such a statement should raise a few preliminary questions in your mind.

• Who did the study? Was it done by researchers at a reputable medical school, by an advocacy group with an axe to grind or by a weight-loss business with memberships to sell?
  
• Where was the study published? Studies published in major medical journals are peer-reviewed, meaning they have been read and evaluated by professionals working in the same field. Often the editorial board of such journals will reject an article for failure to meet accepted standards or send it back to its author for further explanation or additional data. A study published in, for example, The New England Journal of Medicine or the Journal of the American Medical Association (JAMA) will have had to withstand a good bit more expert scrutiny than an article in a small magazine devoted to exercise.
  
• What kind of study was it? Take, for example, our "exercise makes you live longer" story. Like the average newspaper editor or TV producer, the average consumer of medical and health news is not very interested in un-sexy details such as how many people were studied or how the study was structured. With that mindset, it is easy to blur the logical line between, say, "people who live longer exercise" and "exercise makes you live longer."
  

Unfortunately, the devil, as the saying goes, is often in the details. Perhaps the most poorly understood fact about scientific studies is that there are only a few specific types of research that are able to prove cause and effect relationships between one thing and another. Often, all a study is really reporting is that there is a relationship, or correlation, between one thing and another. Once again, take as an example our fictional exercise story. Let's say that the researchers in this study followed a group of retirees from age 65 to age 90. The subjects were asked to fill out questionnaires about how many hours per week they exercised; researchers monitored the retirees and discovered that those who said they exercised for one hour a day lived an average of 20 years longer.

So what is wrong with saying that this study proves that exercise makes you live longer? For one thing, if you look carefully, you will see that there is a possible bias in the selection of subjects; among other differences, people who retire are wealthier than average. Second, the researchers accepted self-reported data about how much subjects exercised; because people know that they are supposed to exercise, when reporting to a medical professional or researcher they might well exaggerate how much exercise they really did. Finally, because of the way it was designed, this study can, at best, do nothing more than establish an association between exercise and longevity. In other words, it shows that the two are related, but not how they are related. Because the study did not attempt to account for the myriad large and small differences between the individuals studied — from medical history to income to ethnicity to education and economic status, there is absolutely no way it could prove that a) exercise causes b) longer life. In fact, the findings of the study are perfectly consistent with other explanations, such as:

• healthier people exercise more; they live longer because they are healthy, not because of the exercise.
• wealthier and more educated people exercise more; these same people also take better care of their health.
• depressed people under-report the amount of exercise they get and depression is associated with premature death.

Medical and other academic journals in the sciences require that their authors explain the methodology they used. This is done so the methods themselves can be evaluated and so other researchers can try to test or replicate (reproduce) the findings. If other researchers perform similar experiments and cannot come up with similar findings, it is likely that the findings of the first study were affected by some factor other than the one being studied — anything from unaccounted-for qualities of the participants to an accidental flaw in the way the study was done. Suppose one study found that eating a grapefruit every day improved muscle tone in a group of elderly men, but other researchers were unable to replicate these findings. The likelihood is that some other, unidentified factor was the real cause for the muscle mass increase in the first study.

That issue, the fact that even a carefully designed, peer-reviewed "scientific study" — the kind quoted in news articles, on television and websites — can be wrong, is the other reason why it is important to know a little about some of the basic research designs used by medical and other researchers. The rest of this article considers some of the major types of studies and the strengths and weaknesses of each.