Data dredging, bias, or confounding

PDF of this article

extra: Figures

extra: References

Email this article to a friend

Respond to this article

PubMed citation

Editorials

They can all get you into the BMJ and the Friday papers

On 4 October 2002, women who were moderate drinkers received good news: their risk of breast cancer was not raised, accordingto a report in the Lancet that was widely covered by the Britishmedia.¹ The bad news was that smoking at an early age was nowimplicated as a risk factor for breast cancer. However, afterthey had enjoyed guilt-free drinks (without cigarettes) for onlya few days, on 13 November the message was reversed: alcohol didincrease the risk of breast cancer after all, but smoking wasdeclared innocent.² The press release proclaimed "Alcohol,tobacco and breast cancer: the definitive answer." A reader wasdriven to complain in the letters page of the Guardian (14 November2002): "So let me get this rightalcohol's no good anymore, andif you smoked within five years of getting your periods, that'sbad news too. Oh no, that was a couple of weeks ago; smoking'sokay now . . . Do things stop being bad for us if we just forgetabout them for a bit, do youthink?"

This is a familiar storyso much so that in Bristol we set our medical students the exercise of examining the "health scareof the week" that appears each Friday, generally from a studyreported in the BMJ or Lancet.^w1

Observational studies propose, RCTs dispose
The widespread perception that epidemiologicalstudies generate conflicting and often meaningless findings^w2 has received support from recent randomised controlled trials,which have failed to confirm even apparently robust findings fromobservational epidemiological studies. The most topical of theserelates to hormone replacement therapy. In 1991 a meta-analysisof epidemiological results relating the use of hormone replacementtherapy to the risk of coronary heart disease concluded that ithalved the risk, and that the evidence was statistically robust(relative risk 0.50; 95% confidence interval 0.43 to 0.56) andthat "overall, the bulk of the evidence strongly supports a protectiveeffect of estrogens that is unlikely to be explained by confoundingfactors."³ Results from randomised controlled trials were,however, very disappointing, with the first large scale trialshowing no benefit, confirmed in two subsequent trials, resultingin a pooled odds ratio of 1.11 (0.96 to 1.30).⁴ The apparent cardioprotective effects of hormone replacement therapy that hadbeen found in the observational epidemiology studies were overturned. Again, women were left wondering what they shoulddo.

A similar scenario had previously been played out for the antioxidant vitamin carotene. Promising epidemiological and laboratoryfindings led to a paper published in 1981 in Nature entitled "Candietary beta-carotene materially reduce human cancer rates?"⁵Cancers related to smoking seemed particularly tractable, andby 1990 the answer for lung cancer was a clear yes: Walter Willett,reviewing the observational epidemiological data, concluded that"Available data thus strongly support the hypothesis that dietarycarotenoids reduce the risk of lung cancer."⁶ Four years latera large scale randomised controlled trial showed an 18% increase(3% to 36%) in lung cancer in those taking carotene.⁷ VitaminE and coronary heart disease provided another example of observationalstudies and randomised controlled trials failing to reach thesame conclusion.^w3

"Eating fruit halves the risk of an early death" the Independent claimed^w4 in an excited response to a study showing a strong inverse associationbetween blood vitamin C levels and mortality due to coronary heartdisease.⁸ A subsequent randomised controlled trial of a vitaminsupplement that raised blood vitamin C levels by 15.7 µmol/l foundfive year mortality due to coronary heart disease unchanged (relativerisk 1.06; 0.95 to 1.16),⁹ whereas the equivalent observationalfindings for this increase in blood vitamin C were coronary heartdisease relative risks of 0.63 (0.49 to 0.84) in women and 0.72(0.61 to 0.86) in men (see fig A on bmj.com). Again, the resultsfrom robust experiment and fallible observation are clearly non-compatible.

This litany of failure has attracted considerable popular comment. Medical journalist James Le Fanu has proposed an extremesolution to this problem: "The simple expedient of closing downmost university departments of epidemiology could both extinguishthis endlessly fertile source of anxiety mongering while simultaneouslyreleasing funds for serious research."^w5

Data dredging, biases, and confounding
It would seem wiser to attempt a better diagnosisof the problem before prescribing Le Fanu's solution. Data dredgingis thought by some to be the major problem: epidemiologists havestudies with a huge number of variables and can relate them toa large number of outcomes, with one in 20 of the associationsexamined being "statistically significant" and thus acceptablefor publication in medical journals.^w6 The misinterpretation of a P<0.05 significance test as meaningthat such findings will be spurious on only 1 in 20 occasionsunfortunately continues. When a large number of associations canbe looked at in a dataset where only a few real associations exist,a P value of 0.05 is compatible with the large majority of findingsstill being false positives.^w7 These false positive findings are the true products of data dredging, resulting from simply looking at too many possible associations.One solution here is to be much more stringent with "significance" levels, moving to P<0.001 or beyond, rather than P<0.05.^w7

Selection and information biases also need to be considered. Selection bias could produce a study database in which a givenexposure is related to a variety of characteristics that increase(or decrease) risk of disease, where such associations are notapparent in the general population. Information biases also arise.For example, some people like to complain and will, if asked,complain both about life's experiences (such as stress) and alsosubjective health outcomes (such as having chest pain). An associationbetween the two would lead to the inference that life stressorslead to angina, but in fact the two are simply related by a proclivityto complain, as evidenced by the finding that there is no associationbetween reporting life stressors and objective, as opposed tosubjective, indicators of coronary heart disease.¹⁰

By far the most likely cause of spurious association is confoundingwhere one factor that is not itself causally related todisease is associated with a range of other factors that do increasedisease risk. Women who use hormone replacement therapy may beless likely to be smokers, more likely to exercise regularly,and less likely to be poor, all of which reduce the risk of coronaryheart disease (see fig B on bmj.com). Associations reported inobservational studies but not confirmed in randomised controlledtrials tend to be of exposures that are related to many socioeconomicand behavioural measures that are in turn related to disease.As with bias, increasing the significance level provides no protectionagainst being misled by confoundedassociations.

The inadequately recognised truth is that we live in an associational worldpeople who are disadvantaged in one regard tendto be disadvantaged in other regards, since the forces that structurelife chances and experience tend to ensure that some folk getthe worst of all things. We showed this by producing a pairwisecorrelation matrix of 133 physical examination and laboratoryassay variables (8778 correlations) derived from a study of over4000 older women.^w8 This would be expected to yield 88 "significant" chance associationsat the P<0.01 level. In fact over 3000 such correlations wereobserved with a P value <0.01. In many ways it is more remarkablewhen things don't "significantly" correlate with each other thanwhen theydo.

A standard argument is that hypotheses built on good scientific understanding of pathogenesis are unlikely to be spurious,but unfortunately it is generally easy to find a biologicallyplausible mechanism to "explain" each association.^w9 Furthermore, it is seldom recognised how poorly the standardstatistical techniques "control" for confounding, given the limitedrange of confounders measured in many studies and the inevitablesubstantial degree of measurement error in assessing the potentialconfounders.^{w9 w10}

What can be done about confounding?
Where possible, associations should be replicatedin databases in which the potential confounding structure differsfrom the initial study. In different countries exposures suchas self reported stress, diet, or birth dimensions, for example,may be related in different ways to socioeconomic circumstancesand socioeconomically patterned causes of disease. Finding thesame association within different populations gives some protectionagainst being misled by confounding.

Specificity of associations between exposure and diseases is also helpful, as most diseases have only a finite number of causes.When exposures are related in a promiscuous way with a wide varietyof outcomes, confounding by socially patterned behavioural andenvironmental factors is likely. Early on in the hormone replacementtherapy debate, Diana Petitti pointed out that hormone replacementtherapy apparently protected against accidental and violent deathsin observational studies as much as against coronary heart diseaseandthat given the lack of a plausible biological link between hormonereplacement therapy and accidental or violent death, both associationsmay have been confounded.¹¹ This suggestion was later confirmedby the randomised controlled trials.⁴

Further measures include improving study design by measuring confounders better and thus allowing for a greater degree ofstatistical control. This may require carrying out more measurementson a smaller number of participants.^w11 Sensitivity analyses should be carried out to model the degreeto which measurement error in confounders could have left residualconfounding^{w12 w13}and should be a necessary part of the statistical reporting ofstudy results. A gift to epidemiology from modern genomics isthe potential for using functional genetic polymorphisms thatmimic the effects of environmental exposures to test exposure-diseaserelationships. There is very little opportunity when alleles segregateeffectivelya random processfor social and behavioural factors to confoundthe resulting polymorphism-disease associations.¹² ^w14

Finally, the findings in observational studies of individuals should be related to the differences in risk of disease observedbetween populations, and within populations over time, as onlythose exposures which fit coherently into this scheme are likelyto be important causes ofdisease.

Of course all our recommendations should be suspended once a year, to allow the Christmas issue of the BMJ to continue withits tradition of making the festive time a merrily data dredged,biased, and confounded one. Also remember that dredging, now disparaged,was the technique by which pearls were harvested from oysters.Among data dredged observations will reside new and precious associations:the only problem is deciding which ones should be gathered andused.

George Davey Smith, professor of clinical epidemiology.

(zetkin@bristol.ac.uk)

Shah Ebrahim, professor in epidemiology of ageing.

Department of Social Medicine, University of Bristol, Bristol BS8 2PR

Footnotes

Competing interests: GDS and SE are the co-editors of the International Journal of Epidemiology. Because the BMJ and othermajor weekly medical journals have cornered the market in splashingdata dredged, biased, and confounded associations across the mediathrough their press releases, the profile of quality journalsis reduced, much to the chagrin of theireditors.

Extra figures and references appear on bmj.com