Nikolai Bogduk, Newcastle Bone and Joint Institute, University of Newcastle, Newcastle, NSW, Australia.

Transcription

1 TRUTH IN MUSCULOSKELETAL MEDICINE. I: CONFIDENCE INTERVALS Nikolai Bogduk, Newcastle Boe ad Joit Istitute, Uiversity of Newcastle, Newcastle, NSW, Australia. Critical reasoig ad biostatistics is ot somethig ew. The istrumets ad cocepts were developed i the 1950s, ad have bee elaborated ad refied sice that time. What is relatively ew is their applicatio to medicie at large ad to musculoskeletal medicie i particular. Cospicuously abset i the past, ad eve to this day, has bee a appropriate respect for biostatistics ad critical reasoig i udergraduate ad postgraduate medical curricula. At best, lip service has bee paid to biostatistics but the implicatios of biostatistics have ot bee itegrated ito cliical practice. Istead, medical studets are typically taught i a way that implies that the various techiques of physical examiatio ad diagostic tests i which they are traied are reliable ad valid, ad that the treatmets that they are taught are uquestioably efficacious. This patter cotiues ito postgraduate traiig. Yet, iroically whe these tests ad treatmets are subjected to scietific scrutiy they ofte prove ot to be reliable, valid or efficacious. There is, therefore, a mismatch betwee what is taught ad the truth. This series of articles is desiged ot to costitute some sort of course of istructio i academic material that makes doctors erudite but which is immaterial to cliical practice. Rather, it is iteded to help the reader become a better cosumer of cliical iformatio, so that they ca recogise the reliable, the valid ad efficacious, ad thereby distiguish truth from mythology, assertio ad speculatio. The objective is to equip the reader with devices that allow them to check iformatio for themselves rather tha relyig o what experts say is right ad wrog. If othig else, the reader will lear what questios to ask, what iformatio to demad, before acceptig or believig a speaker or the writer of a joural article. CONFIDENCE INTERVAL OF A PROPORTION This first cocept is a preface. It does ot lead systematically ito subsequet topics but recurs i various forms i other areas of biostatistics, ad has some immediate applicatios to day to day practice. The cocept ca be itroduced by the questio: does 3 out of 10 equal 30%? Mathematicias ad philosophers may argue what they please about this questio, but i medicie the aswer is - o. The fractio - 3/10, is a proportio, ad i medicie will usually reflect the result of some sort of harvest. A ivestigator will have studied or surveyed 10 cases ad foud the idex coditio i three. They are tempted to proclaim a yield of 30%. The illegitimacy of this temptatio stems from the possibility that if the same ivestigator, or aother ivestigator, repeated the same experimet, they might ecouter a slightly differet yield - say, 4/10 or 2/10 or eve 6/1 What the is the true frequecy? The priciple at had is that there may be a correct or true proportio, that would be evidet if every patiet or every perso i the uiverse was surveyed, but this proportio will ot ecessarily be evidet if oly a small sample of the total, possible populatio is surveyed. For ay small sample a samplig error may occur. Just by accidet, the ivestigator might select a group of subjects who happe to exhibit the feature i questio somewhat more frequetly tha the true proportio or somewhat less frequetly.

2 Bogduk - Cofidece Itervals 2 I order to accommodate this possibility a statistical correctio applies 1. The formula is: p* = p p( 1 p) where, p is the observed proportio, is the umber of subjects, 1.96 is a coefficiet that geerates a 95% probability, ad p* is the rage withi which there is a 95% chace that the true proportio actually lies. Coversely there is a 5% probability that the true value lies outside this rage. If we cosider our example, p = 3/10 = 3 (1-p) = (1.0-3) = 7 = 10 p* = = = 02 to 58 ( 3)( 7) 10 This result shows that upo samplig 10 subjects ad fidig a idex coditio i 3, the prevalece of that coditio is ot ecessarily 30%. The true prevalece could be as low as 2% or as high as 58%. Uder these coditios, 30% is ot a represetative figure. The ambiguity arises because the sample size () is small. Now see the effect of icreasig the sample size. Suppose that the ivestigator studied 100 subjects istead of 10, ad foud the coditio i 3 The prevalece is still ot 30%; that figure is still oly a estimate because the ivestigator did ot survey every patiet i the uiverse. The cofidece iterval of the observed proportio must be calculated. p = 30/100 = 30 (1-p) = ( ) = 70 = 100 p* = = = 22 to 38 ( 30)( 70) 100 There is still ucertaity but it is less tha whe the sample size was 1 The true proportio could be as low as 22% or as high as 38%, but for this rage, 30% is ow a reasoable, idicative figure. Icreasig sample size decreases the size of the 95% cofidece iterval of the observed proportio. The decrease is expoetial. Whe is small the cofidece iterval is wide. As icreases the cofidece iterval arrows but ever gets to zero. The iterval would get to zero oly if everyoe i the uiverse is sampled, i.e. = ifiity. Oly you ca aswer how big the sample should be. The aswer traslates ito a questio to you - how close do you wat the estimated proportio to be to the true proportio? If you are satisfied with a 20% rage you might settle for a small sample, but if you wat to be withi 5% of the true proportio, you will

3 Bogduk - Cofidece Itervals 3 demad a larger sample. The equatio is available to you i order to calculate these aswers. This equatio is oe that you should either memorise or carry i your wallet; you ever kow whe you will eed to use it as a cosumer of ew iformatio. EXAMPLES The followig are a series of examples that illustratio the applicatio of cofidece itervals i situatios that may befall practitioers i musculoskeletal medicie. They are ot esoteric or academic applicatios but oes that should be of day to day cocer or iterest. Example 1: success rate of a ew treatmet A speaker aouces a success rate of 70% for a ew treatmet. Do you believe him? You should ot. You should first ask - what was? If = 10, the success rate is 7/1 You should calculate the cofidece iterval of this proportio. It amouts to 42% to 98%. I real-life terms this meas that if you were to repeat the experimet, i.e. adopt the treatmet, you might ecouter a result as good as 98% or as bad as 42%. You caot expect 70%; you should be prepared for 42%. If = 100, the cofidece iterval chages to 61% to 79%. I this case, 70% is a more idicative figure of what you might expect to ecouter; but be prepared for a success rate as low as 61% istead of 70%. This example shows that cofidece itervals are ot a tool used oly by research scholars. They are relevat to you as a therapist. Every time someoe advises you to adopt a ew therapy, they are effectively ivitig you to become a ivestigator ad to repeat their experimet. Therefore, you should have o illusio that reported proportios are absolute. Your experiece may be differet from that of the first ivestigator, ad the cofidece iterval formula idicates to you how differet your experiece might be. You ca also use the equatio i reverse to calculate the appropriate, if you have a certai cofidece iterval i mid. Let s say you wat to esure that the success you are prepared to accept is aywhere withi 15% of the speaker s reported success rate. O what sample size should the speaker s results be based? Uder these coditios, p* = ( )( ) = 15 ( 70)( 30) = = 0765 = = = 35.59

4 Bogduk - Cofidece Itervals 4 Thus, for you to expect a success rate i your hads of 70% + 15%, the ivestigator should provide you with a success rate of 70% based o subjects, i.e. at least 36 subjects. If the ivestigator s study is smaller tha this, you caot rely o achievig a result i your hads that falls withi 15% of 70%; your result might be cosiderably worse. The same calculatio could be repeated if you wated a tighter rage, say 5%, ad if the reported success rate were ay other figure, say 80% or 60%. The geeral formula is: where p = the reported success rate p* = the rage which you would accept, i.e. p* = p + z z = 1.96 p( 1 p) from which ca be calculated. Example 2: could it be placebo? A ivestigator audits a ew treatmet. He fids a success rate of 8/1 Is this impressive, or might it be a placebo respose? I order to aswers these questios, calculate the cofidece itervals. The cofidece iterval of 8/10 is 55% to 100%. Prima facie this looks like the result is ot a placebo, for 55% is well above the covetioal estimate of 30% for a placebo respose rate. However, oe must also calculate the cofidece iterval of the placebo respose rate. I a sample of oly 10 patiets, the cofidece iterval of 3/10 is 2% to 58%. This meas that i a sample of oly 10 patiets, a series of ivestigators could ecouter placebo rates as low 2% or as high as 58%. The figure - 58%, is higher tha the figure - 55%. Thus, the cofidece itervals of 8/10 ad 3/10 overlap. This meas that statistically, it possible for a success rate of 8/10 to overlap the possible placebo rage. Thus, prima facie, 8/10 could well be a placebo respose rate. The ivestigator would have to provide you with a larger study, with a arrower cofidece iterval of the success rate, before you ca accept that the result is ot placebo. This sort of calculatio does ot prove that a reported result is or is ot a placebo respose; that ca oly be show directly i a cotrolled study. But it does show that statistically 8/10 is ot ecessarily a impressive result. Calculatig the cofidece itervals prevets you from beig seduced ito believig that such a impressive could ot possibly be a placebo. Ivestigators watig to pla a study ca use these sorts of calculatios i reverse i order to determie what size of study is required to provide prima facie evidece that the observed success rate is ulikely to be due to a placebo effect. Such calculatios ca be used to determie if the treatmet i questio is worthy of a cotrolled trial. There is o poit expedig effort if the prima facie evidece is cosistet with a placebo effect. What size study should be coducted if the observed success rate is 75%, ad the placebo rate is assumed to be 30%?

5 Bogduk - Cofidece Itervals 5 For there to be prima facie evidece of the treatmet ot beig due to placebo, the upper cofidece limit of the placebo rate should be less tha the lower cofidece limit of the success rate, i.e ( 3)( 7) < ( 75)( 25) ṅ < ṅ < < 23 Upo squarig both sides, ( )( 1875) + + < < < > Thus, for this success rate ad expected placebo rate, the study must be based o at least 16 subjects. A smaller study would ot show a success rate greater tha the possible placebo rate. Coversely, if the placebo rate was greater, or the success rate smaller, the sample size would eed to be appropriately larger. Remember, however, that such calculatios do ot prove that the success is ot due to placebo; they provide oly prima facie evidece that it is ulikely to be due to placebo. The utility of the calculatios is ot to substitute for a cotrolled trial, but to prevet cotrolled trials beig wasted o success rates that are possibly withi the placebo rage. They also protect the cosumer from beig seduced by figures that umerically look impressive but which are based o too small a study. Example 3: populatio studies I a epidemiological study, a ivestigator samples 50 idividuals with a history of eck pai after a motor vehicle accidet ad fids that oe developed chroic eck pai. He cocludes that chroic eck pai after whiplash does ot occur. Is this deductio correct? Assume that the true prevalece of chroic eck pai after whiplash is 6%. Did the author have a large eough sample to exclude this prevalece?

6 Bogduk - Cofidece Itervals 6 For a sample of 50, the cofidece iterval of 6% is p* = ( )( ) 50 = = 0 to 125 The figure - 0, idicates that with a sample of oly 50 the ivestigator could well be studyig a populatio i which the true prevalece was 6% but would fid zero cases. Aother ivestigator, usig the same sample size might fid 12.5% of 50 = 6.25, i.e. 6 cases. Thus, the study does ot exclude a 6% prevalece. For iterest, calculate what prevalece does a sample of 50 reasoably exclude. p p( 1 p) 50 > 0 p > 071 Thus, a sample of 50 might oly exclude a prevalece of more tha 7%. However, remember that this is ot a absolute result. The cofidece iterval expresses oly a 95% chace. Thus, a sample of 50 has oly a 95% chace of excludig a prevalece of 7%. There remais a 5% chace that a prevalece of 7% would ot be excluded by a sample of 5 Aother warig is that towards the extremes, cofidece iterval calculatios come to grief. Whe the proportios approach 0% or 100%, the covetioal formula does ot apply ad certai mathematical adjustmets eed to be applied 2. Armed with this example, the reader is ivited to aalyse for themselves, as a assigmet exercise, the data ad coclusios provided by a recet study i Lithuaia o the prevalece of whiplash 3. [Hit: fid how may patiets suffered eck pai immediately after the accidet ad how may of these wet o to develop chroic symptoms. Usig these figures, calculate the prevalece that would be excluded by this sample size, of chroic eck pai arisig i patiets who suffer eck pai immediately after a accidet. Compare that fidig with the prevalece of chroic eck pai i idividuals who simply are ivolved i a accidet without sufferig eck pai.] CONCLUSION This is the first step towards icorporatig biostatistics ito everyday practice. The ability to calculate the cofidece iterval of a proportio equips the reader with a survival techique i the world of medical cosumerism. It protects the reader agaist beig hoodwiked by figures that look good but which are based o too small a sample. The cofidece iterval is oe of the devices that helps aswer the questio - was the study big eough. Istead of appealig to experts, readers ca ow use the cofidece iterval formula to aswer this questio for themselves, wheever the occasio arises. I future articles, we will address truth i diagosis ad truth i therapy.

7 Bogduk - Cofidece Itervals 7 REFERENCES 1. Armitage P, Berry G. Statistical Methods i Medical Research, 3rd ed. Oxford: Blackwell, 1994 pp Sackett DL, Hayes RB, Guyatt GH, Tugwell P. Cliical Epidemiology. A Basic Sciece for Cliical Medicie, 2d ed. Bosto: Little, Brow & Co. 1991; pp Schrader H, Obelieiee D, Bovim G, Surkiee D, Mickeviciee I, Sad T. Natural evolutio of late whiplash sydrome outside the medicolegal cotext. Lacet 1996;347: