Interval Estimates for Predictive Values in Diagnostic Testing with Three Outcomes

Transcription

1 Iterval Estimates for Predictive Values i Diagostic Testig with Three Outcomes Scott Clark, Laure Modi, Courtey Weber ad Jessica Wibor Advisor: Melida Miller Holt Abstract: I disease testig, patiets ad doctors are iterested i estimates for positive predictive value (PPV) ad egative predictive value (NPV). The PPV of a test is the probability that a patiet actually has the disease, give a positive test result. The NPV is the probability that a patiet actually does ot have the disease, give a egative test result. Here we cosider diagostic tests i which the disease state remais ucertai, so the ucertai predictive value (UPV) is also of iterest. UPV is the probability that, give a ucertai test result, follow-up testig will remai icoclusive. We derive classical Wald-type ad Bayesia iterval estimates of PPV, NPV, ad UPV. Performace of these itervals is compared through simulatio studies of iterval coverage ad width.. Itroductio The positive predictive value (PPV) ad egative predictive value (NPV) of a diagostic test are importat to both patiets ad doctors. The PPV of a test is the probability that, give a positive test result, the patiet actually has the disease. Similarly, the NPV of a test is the probability that, give a egative test result, the patiet is actually disease-free. Mercaldo et al. (007) derived Wald-type iterval estimates of both PPV ad NPV for biary diagostic tests i case-cotrol studies where the true disease prevalece is assumed kow. Stamey ad Holt (00) derived aalogous Bayesia iterval estimates that allow for ucertaity relative to disease prevalece. Here we cosider studies i which both the diagostic test uder cosideratio ad the best test available may both provide icoclusive results, such as those 34

2 reported i Khatami et al. (007). We exted the results of Mercaldo et al. (007) ad Stamey ad Holt (00) to diagostic tests with three possible outcomes: positive, egative, ad ucertai. To address the possibility of icoclusive results, we derive a estimator for what we call the ucertai predictive value (UPV). This is the probability that, give a ucertai iitial diagostic result, the truth would remai ukow uder the best test available, which we call the gold stadard for clarity. Because such follow-up procedures ca be ivasive ad costly, the UPV may be helpful i determiig whether the patiet should udergo more testig. Other authors have cosidered similar situatios, such as Feistei (990), which cosiders a similar desig with three results of the surrogate, or diagostic, test but oly two for the gold stadard test. As far as we kow, there is o other work associated with three possible outcomes for both the diagostic test ad the gold stadard, as experieced by Khatami et al. (007). To begi, Sectio describes the experimetal desig ad associated estimators developed by Mercaldo et al. (007) ad Stamey ad Holt (00). Wald-type iterval estimators for PPV, NPV, ad UPV are derived uder a expaded desig i Sectio 3.. Sectio 3. presets aalogous Bayesia iterval estimators. I Sectio 4, we provide a performace aalysis comparig coverage ad width of these itervals. Coclusios are offered i Sectio 5.. Biary Respose Studies. Desig Cosider first the study desig described i Table. Let i = represet the existece of disease ad i = represet absece of disease. Similarly, let j = represet a positive test result ad j = represet a egative result. Thus x ij represets the umber of patiets i disease state i 35

3 ad test result j. Likewise i represets the total umber of people i disease state i. Mercaldo et al. (007) ad Stamey ad Holt (00) described studies of this type that were ot desiged to reflect the true disease prevalece i the target populatio. The same situatio is cosidered here. Thus, we assume that these values are available from sources exteral to the curret study ad that p represets the kow populatio probability of disease ad p = p reflects the kow probability that a perso is disease-free. Test + Test Total Disease + x x Disease x x Table : Biary Respose Study Desig For such studies, researchers ad cliicias are iterested i estimatig the sesitivity (Se) of the test, ad the specificity (Sp), Se = P(Test + Disease + ) Sp = P(Test Disease ). Through Bayes Theorem, the sesitivity ad specificity the lead to evaluatio of the positive predictive value PPV = P(Disease + Test + ) = p Se p Se + p ( Sp) ad the egative predictive value NPV = P(Disease Test ) = p Sp. p Sp + p ( Se) The data i Table yield the followig maximum likelihood estimators (MLE): x Se = ad x Sp =. 36

4 Similarly, the false positive (Fp) ad false egative (F) rates are estimated by x Fp = ad x F =. Based o the kow prevalece values ad the estimators above, estimators for the PPV ad NPV become PPV = p Se p Se + p Fp () NPV = p Sp. () p F + p Sp Mercaldo et al. (007) derived Wald-type iterval estimators for both PPV ad NPV, based o equatios () ad (). Likewise, Stamey ad Holt (00) derived aalogous Bayesia estimators. These will be discussed i Sectios. ad.3, respectively.. Wald-Type Itervals Mercaldo et al. (007) developed Wald-type iterval estimators for PPV ad NPV by employig the delta method to determie the variaces of PPV ad NPV. Assumig that the resposes follow a biomial distributio ad based o the asymptotic ormality of the MLEs, the authors proposed a approximate iterval for PPV of the form PPV ± z var PPV, α / where z ( α /) is the (-α/) th quatile of the stadard ormal distributio ad var ( PPV) = ( p p Fp) Se F ( p p Se) Sp Fp ( p p Fp Se). 37

5 Similary, the estimator for NPV was NPV ± z var NPV α /, where z ( α /) is the (-α/) th quatile of the stadard ormal distributio ad var ( NPV) = ( p p Sp) Se F ( p p F) Sp Fp ( p p F Sp). Mercaldo et al. (007) also proposed alterate itervals based o a Agresti-type adjustmet (Agresti ad Caffo, 000). Here the adjusted estimates of Se ad Sp became ad Se = k Se + k Sp + Sp =, so that the adjusted PPV ad adjusted NPV calculatios become PPV = p Se p Se + p Fp (3) ad p Sp NPV =, p F + p Sp (4) respectively. 38

6 Mercaldo et al. (007) compared the performace of these proposed iterval methods, as well as itervals based o their logit trasformatios, usig simulatios of cofidece iterval width ad coverage probability. They cocluded that, whe it provided a solutio, the uadjusted logit trasformatio performed best. Otherwise, they recommeded the adjusted stadard iterval. Stamey ad Holt (00) foud, however, that the logit trasformed solutio ofte does ot exist. Thus, the research herei focuses o adjusted stadard Wald-type itervals..3 Bayesia Itervals Stamey ad Holt (00) derived Bayesia iterval estimates of PPV ad NPV for casecotrol studies described i Table. Because such desigs require a priori kowledge of disease prevalece, the authors cosidered two Bayesia models: oe that took disease prevalece to be kow ad oe that placed a prior distributio o prevalece. Like Mercaldo et al. (007), Stamey ad Holt (00) assumed biomial data so that x ~ biomial (, Se ) ad ~ (, ) They assumed cojugate beta priors so that x biomial Sp. Se ~ (, ) beta α β ad Se Se Sp Sp Sp ~ beta( α, β ), which yielded posterior distributios for Se ad Sp of Se d ~ beta( x + α, x + β ) ad Sp d ~ beta( x + α, x + β ) Sp Sp, (5) Se Se where = x x d (,,, ). Takig p ad p to be kow, Mote Carlo samplig from the posteriors i (5) provided estimates of Se ad Sp. Pluggig these values ito equatios () ad () yielded estimates of the posterior distributio for the uadjusted estimates of PPV ad NPV. Adjusted estimates of PPV ad NPV were similarly obtaied, usig equatios (3) ad (4). 39

7 The secod model assumed that p is i fact estimated from previous survey data or expert opiio. I this case, the value of p i () ad () follows the posterior distributio p d ~ beta ( y+ α, y+ β ) p p where the data vector is ow d = ( x, x, y,,, ) ad y represets the umber of patiets with the disease i a previous survey of patiets. Here Se, Sp, ad p (ad thus p ) are each obtaied through Mote Carlo samplig i order to estimate PPV ad NPV. The authors compared the iterval width ad true coverage probability of both Bayesia itervals to that of the adjusted stadard iterval ad the uadjusted logit iterval of Mercaldo et al. (007) through simulatio studies. They foud that the Bayesia itervals maitaied coverage close to the omial level, while the adjusted Wald-type iterval dipped far below omial for large values of Sp. Widths were geerally comparable for the methods that assumed kow prevalece, with the exceptio of large values of Sp. I that case, the Bayesia iterval was wider because it maitaied coverage closer to omial. The Bayesia procedure that assumed ukow prevalece produced itervals that were geerally wider tha the other methods, but maitaied coverage quite close to omial for all parameter values. 3. Multiomial Respose Studies 3. Desig Now cosider studies with the desig described below i Table. Patiets either have the disease, do ot have the disease, or their true state remais icoclusive, as determied by a gold stadard test. The diagostic test uder cosideratio may retur results that are positive, egative, or ucertai. Let i = represet the existece of disease, i = represet absece of 40

8 disease, ad i = 3 represet true ucertaity. Similarly, let j = represet a positive test result, j = represet a egative test result, ad j = 3 represet a icoclusive test result. Thus x ij represets the umber of patiets i disease state i ad test result j. Likewise i represets the total umber of people i disease state i. Test + Test Test 0 Total Disease + x x x 3 Disease x x x 3 Disease 0 x 3 x 3 x 33 3 Table. Multiomial Respose Study Desig The data i Table yield the followig maximum likelihood estimators (MLE) of sesitivity (Se), specificity (Sp), ad ucertaity (U): x Se =, Sp x =, ad U = x33. 3 Similarly, the false positive (Fp) ad false egative (F) rates are x x =. Fp = ad F Usig this desig, we must also cosider the probability of a ucertai test result amog patiets with the disease (Yu), the probability of a ucertai test result amog patiets without the disease (Nu), the probability of a positive test that caot be cofirmed or refuted by the gold stadard (U ), ad the probability of a egative test that caot be cofirmed or refuted by the gold stadard (U ). These values are estimated by the followig: x3 Yu =, x3 Nu =, x =, ad 3 U 3 x 3 U = 3 4

9 Agai we assume that p represets the kow populatio probability of disease; p reflects the kow probability that a perso is disease-free; ad p 3 is equal to the kow probability of udetermiable results. Just as Mercaldo et al. (007) ad Stamey ad Holt (00) assumed a biomial distributio, we assume that x = ( x, x, x 3 ), x = ( x, x, x3), ad x = ( x, x, x ) are idepedet ad follow a multiomial distributio. Thus, the probability mass fuctio for x is x x x3 (,, ) Se F Yu f x x x 3! =. x! x! x! 3 We let t = (Se, F, Yu), ad write x ~ M (, t ). Similarly, t = (Fp, Sp, Nu) ad t 3 = (U, U, U) so that x ~ M (, t ), ad x 3 ~ M ( 3, t 3 ). Applyig Bayes Theorem produces the followig estimators for PPV, NPV, ad UPV: PPV = NPV = UPV = p Se p Se + p Fp + p U 3 p Sp p F + p Sp + p U 3 p 3 U p Yu + p Nu + p U 3 (6) (7) (8) Based o the equatios (6) through (8), we exted the results of Mercaldo et al. (007) ad Stamey ad Holt (00) to situatios i which both the diagostic test beig evaluated ad the gold stadard produce icoclusive results. 4

10 3. Wald-type Itervals Cosider first the estimatio of the PPV. We derive a 00( α)% Wald-type iterval by applyig the delta method. The iterval takes the form: PPV ± z var PPV, (9) α / where z is the (-α/) th quatile of the stadard ormal distributio. Because Se, Fp, ad ( α /) U are asymptotically ormal with meas Se, Fp ad U, respectively, we take PPV to be asymptotically ormally distributed with mea PPV. To derive the variace of PPV, we must first cosider the variaces of Se, Fp, ad U. By the properties of the multiomial distributio, amely that the margial distributio of each idividual cell cout is biomial, we obtai Se Se var ( Se) =, var Fp Fp Fp =, ad var U U U =. 3 Because x, x, ad x 3 are idepedet, the covariaces of Se, Fp, ad U are equal to zero. I order to obtai the variace estimator for PPV, we apply the delta method ad use the partial derivative of PPV with respect to Se, Fp, ad U. For the full derivatio, see Appedix A. This provides a variace estimator for PPV i (9): ( p p3 U) + p p Fp var(se) + p p3 Se var(u ) + p p Se var(fp) var(ppv) = 4 ( p Se + p Fp + p3 U). Similarly, 43

11 Sp Sp var ( Sp) =, var F F F =, ad var U U U =, 3 which lead to ( 3 ) ( p p3 U) + ( p p F ) var(sp) + p p3 Sp var(u ) + p p Sp var(f) var ( NPV) = 4 p F + p Sp + p U ad the iterval estimator NPV ± z ( α /) var( NPV). (0) Lastly, Yu ( Yu) var ( Yu) =, var Nu Nu Nu =, ad var U U U =, 3 which lead to ( p ) ( ) ( ) p3 Yu + p p3 Nu var U + p p3 U var Nu + p p3 U var Yu var ( UPV) = 4 p Yu + p Nu + p U ( 3 ) ad the iterval estimator UPV ± z var UPV. () ( α /) As do Mercaldo et al. (007), we cosider a Agresti type adjustmet (000) to improve performace of the estimators for proportios ear 0 or. To do so, the data i Table is adapted to become those preseted below i Table 3. 44

12 Test + Test Test 0 Total Disease + x +k x +k x 3 +k +3k Disease x +k x +k x 3 +k +3k Disease 0 x 3 +k x 3 +k x 33 +k 3 +3k Table 3. Study Desig with Adjustmets We represet estimators based o this adjustmet with. For example, x + k Se = + 3k. For biomial proportios, Agresti ad Caffo (000) recommed usig ( z ) ( α /) k =. To our kowledge, however, o such recommedatios exist for estimatio of multiomial proportios. Several values of k were cosidered ad tested. We recommed k = 0.5 because it produced the best iterval coverage ad width i simulatio-based performace aalysis. 3.3 Bayesia Itervals Next, we derive Bayesia estimators for PPV, NPV, ad UPV. I the Bayesia model, we cosider the Dirichlet distributio because it is the cojugate prior for multiomial data. Thus, the joit prior distributio for t = (Se, F, Yu), the parameters i the distributio of x, is 3 ( α ) Γ i α 3 g( t ) Se α = Yu α F. Γ i ( = 3 α i= i ) The joit posterior for Se, F, ad Yu the becomes 3 ( α x ) Γ + g( t x) = Se F Yu 3 Γ + i i α+ x α+ x α3 + x3 i= ( α i i x = i) writte t x ~ D (α + x, α + x, α 3 + x 3 ). Similarly, the posterior distributios for t = (Fp, Sp, Nu) ad t 3 = (U, U, U) ca be writte t x ~ D (α + x, α + x, α 3 + x 3 ) ad t 3 x 3 ~ D (α 3 + x 3, α 3 + x 3, α 33 + x 33 ), respectively. For the purposes of the aalysis herei, we, 45

13 cosider oiformative Dirichlet priors with parameter values equal to 0.5. This makes the Bayesia aalysis aalogous to the Wald-type aalysis i that o iformatio is available relative to t, t, ad t 3 beyod that i Table 3. Here agai, the prevalece vector, p = (p, p, p 3 ), is assumed kow, so the posterior distributios for PPV, NPV, ad UPV ca be approximated via Mote Carlo samplig by B values from the posteriors of t, t, ad t 3. At each iteratio, the geerated values of Se (j), F (j), Yu (j), Fp (j), Sp (j), Nu (j), U (j), U (j), ad U (j) are plugged ito PPV p Se ( j) ( j) = ( j) ( j) p Se + p Fp + p3 U j, () NPV p Sp ( j) ( j) = ( j) ( j) ( j) p F + p Sp + p3 U, (3) ad UPV ( j) = p U j, (4) p Yu + p Nu + p U ( j) 3 ( j) ( j) 3 where the superscript (j) deotes that this is the j th iteratio i the Mote Carlo approximatio. The posteriors for PPV, NPV, ad UPV are the approximated usig these B values. The.5 th ad 95.5 th percetiles of the distributio the provide the 95% Bayesia credible sets. 4. Performace Aalysis This sectio provides a simulatio-based compariso of the 95% adjusted Wald-type ad Bayesia iterval estimates of PPV through both coverage probability ad iterval width. Cosider = = 3 = 5, Sp = 0.5, ad let Se vary from 0.5 to 0.80 by 0.0. For this aalysis, Yu = Nu = 0., U = 0.8, U = U = 0., Fp = 0.4, ad F varies with Se. I each boxplot, values 46

14 of p vary from 0.50 to 0.85 by 0.05, while fixig p 3 = 0.0 so that p varies from 0.05 to 0.40 by For each set of parameter values, 0,000 samples were simulated leadig to 0,000 Wald (W) ad Bayesia (B) itervals. Coverage probability ad width are estimated by averagig over the 0,000 samples. Figure compares resultig estimates of coverage probability, while Figure compares iterval width. Figure. PPV Coverage Probability Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7, 0.8 Figure. PPV Width Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7,

15 Figure idicates that the Bayesia coverage probability remais closer to the omial 0.95 whe estimatig PPV tha that of the Wald-type iterval for the parameters cosidered here, while Figure suggests that the Bayesia iterval is slightly arrower. Figures 3 ad 4 provide the coverage ad width, respectively, for estimatig NPV at the same parameter values. These plots idicate geerally the same performace patter, with the Bayesia iterval coverage closer to omial ad its width arrower. The behavior exhibited here by the Wald-type itervals is similar to that observed i Stamey ad Holt (00), because coverage deviates further from omial as Se icreases. Likewise, the width of the Bayesia iterval icreases, while coverage remais close to omial. Figure 3. NPV Coverage Probability Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7,

16 Figure 4. NPV Width Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7, 0.8 Figures 5 ad 6 below idicate that the Bayesia coverage probabilities are much closer to omial tha those of the Wald-type iterval whe estimatig UPV, although geerally the Bayesia iterval is slightly wider. Figure 5. UPV Coverage Probability Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7,

17 Figure 6. UPV Width Compariso for Sp = 0.5, Se = 0.5, 0.6, 0.7, 0.8 Several other sets of parameter values were cosidered, varyig Sp ad sample size. Those simulatio results are ot icluded for the sake of space. For varyig values of Sp the behavior demostrated above remais cosistet. At sample sizes of 50 ad above, the two procedures maitaied very similar coverage ad width. Further details about simulatio results are available from the authors. 5. Example The research herei was motivated by the work of Khatami et al. (007). Khatami ad co-authors developed a predictio model, combiig the results of various blood ad tissue markers, to determie whether or ot a cacer origiated i the ovaries. The model desigated the cacer origi as ovaries, ot ovaries, or ucertai. To determie its effectiveess, they compared its diagoses agaist the stadard histologic assessmet. Table 4 describes the data obtaied i their experimet: 50

18 Test + Test - Test 0 Total Disease Disease Disease Table 4. Ovaria Cacer Data (Khatami et al., 007) I order to calculate the iterval estimates of PPV, NPV, ad UPV, the values of p, p, ad p 3 must be determied. We take these to be kow but base them o calculatios i Khatami et al. (007), we take p to be We take p ad p 3 to be 0.6 ad 0.8, respectively. For the adjusted Wald-type iterval, we take a = 0.5. For the Bayesia credible sets, t, t, ad t 3 are each give oiformative Dirichlet (0.5, 0.5, 0.5) distributios. The resultig iterval estimates are preseted i Table 5. Method PPV Estimate 95% Iterval NPV Estimate 95% Iterval UPV Estimate 95% Iterval Wald Adjusted Wald Bayes Table 5. Iterval Estimates of PPV, NPV ad UPV from Khatami et al. (007) Evaluatig Table 5, we see that the iterval estimates are very similar. The mai differece is i the differet iterpretatios of a cofidece iterval ad a credible set. A cofidece iterval states that there is a 95% chace that the true proportio is i the iterval, while the credible set states that if we ru the test a ifiite umber of times, 95% of the time the true proportio will be i the iterval. 5

19 6. Coclusio I this paper, we derive both Wald-type ad Bayesia iterval estimates of PPV, NPV, ad UPV for diagostic procedures with three possible outcomes. We cosidered prevalece to be kow for both the Wald-type ad Bayesia aalysis. Both of these methods require kowledge of the outcome of the gold stadard. Simulatio studies idicate that the Bayesia approach usig oiformative priors typically provides better coverage ad smaller width tha that of the Wald-type iterval with a 0.5 Agresti type adjustmet. It is, therefore, our coclusio that the Bayesia method is preferred. Future research will cosider Bayesia aalyses that icorporate iformative Dirichlet priors to reflect ay available prior iformatio o values such as the sesitivity ad specificity i order to further improve performace of the Bayesia method ad Bayesia methods for estimatig PPV, NPV, ad UPV for ukow prevalece. Additioal work will also expad the secod model i Stamey ad Holt (00) to accout for possible ucertaity about disease prevalece values. ACKNOWLEDGEMENTS We wish to thak the aoymous referees for their commets that sigificatly improved the paper. Also, we wish to thak Dr. Ke Smith ad the Natioal Sciece Foudatio for their support through grat #DMS

20 Refereces: Agresti, A. ad Caffo, B. (000). Simple ad Effective Cofidece Itervals for Proportios ad Differeces of Proportios. The America Statisticia 54: Feistei, A. (990). The Iadequacy of Biary Models for the Cliical Reality of Three-Zoe Diagostic Decisios. J Cli Epidemiol 43: Khatami, Z. Cross P., Steveso M.R., Naik R. (007). Desig of a bio-mathematical predictio model usig serum tumor markers ad immuohistochemistry i peritoeal carciomatosis with ovaria ivolvemet: a pilot study. Iteratioal Joural of Gyecological Cacer 7: Mercaldo, N. D. Lau, K. G., Zhou, X. H. (007). Cofidece Itervals for predictive values with a emphasis to case-cotrol studies. Statistics i Medicie 6: Stamey, J. Holt, M. A. (00) Bayesia iterval estimatio for predictive values from casecotrol studies. Commuicatios i Statistics - Simulatio ad Computatio 39:

21 Appedix A To derive var(ppv) we employ the delta method, so that var(ppv) = ( PPV)' Σ( PPV), where [ ] PPV = (PPV) (Se), (PPV) (U ), (PPV) (Fp) ad Σ is the variace/covariace matrix of v = (Se, U, Fp). Recall that Thus, PPV = p Se ( Se) + ( Fp) + ( U ) p p p 3. var (PPV) (PPV) (Se) (Se) var(se) 0 0 (PPV) (PPV) PPV = 0 var(fp) 0 (Fp) (Fp) 0 0 var(u ) (PPV) (PPV) (U ) (U ) T ( p p Fp) + ( p p3 U) p p Se p p3 Se = ( p Se) + ( p Fp) + ( p3 U) ( p Se) + ( p Fp) + ( p3 U) ( p Se) + ( p Fp) + ( p3 U) Se( Se) ( p p Fp) ( p p3 U) ( p Se) ( p Fp) ( p3 U) + + Fp( Fp) p p Se 0 0 ( p Se) + ( p Fp) + ( p3 U) U( U ) 0 0 p p3 Se 3 ( p Se) + ( p Fp) + ( p3 U) 54

22 ( ) ( ) ( ) Se Se Fp Fp U U ( p p ) ( p p ) ( p p ) ( p p ) = ( p Se) + ( p Fp) + ( p3 U) Fp + 3 U + Se + 3 Se 3 4. Substitutig the estimators Se, Fp, ad U, yields the estimator for var ( UPV) ad var (NPV) are derived usig the same method ad thus omitted. var ( PPV ). The values of 55