ROC curves and nonrandom data

Transcription

1 Thi i the working paper verion of Cook, J. A. (2017): ROC urve and nonrandom data, Pattern Reognition Letter, 85(1), ROC urve and nonrandom data Jonathan Aaron Cook May 2016 Abtrat Thi paper how that when a laifier i evaluated with nonrandom tet data, ROC urve differ from the ROC urve that would be obtained with a random ample. To addre thi bia, thi paper introdue a proedure for plotting ROC urve that are inferred from nonrandom tet data. I provide imulation and an example with wine data to illutrate the proedure a well a the magnitude of bia that i found in tandard ROC urve generated from nonrandom tet data. Keyword: ROC urve; Claifier evaluation; Sample-eletion bia 1 Introdution In many etting, data are olleted in a nonrandom fahion. The deiion to invetigate inurane laim for fraud may be baed on a preditive model. Invetigating inurane laim i otly and it may be diffiult to alloate reoure to inpet a random ample of laim. Similarly, the Internal Revenue Servie (IRS) ue a model that predit tax-filing error to elet tax return for audit. A reommender ytem may only how the uer item that are predited to be of interet. In thee three example, data are only olleted for intane that are judged to be more likely to be poitive ae. Thi paper make two ontribution. Thi paper firt ontribution i a haraterization of the bia that reult in reeiver operating harateriti (ROC) urve when they are generated with nonrandom tet data. Nonrandom tet data an arie from uing the laifier that we want to evaluate to elet the tet data or from uing ome other laifier to elet the tet data. Thi paper how that the bia that arie in evaluating the laifier that wa ued to elet the tet data i imilar to the well-known bia for regreion with trunation on the dependent variable. When we only oberve ae whih are ored uffiiently high by the laifier, there i a type of attenuation bia for ROC urve. Thi paper alo how that the ROC urve are puhed outward for a laifier with low orrelation to the laifier that wa ued to elet the tet data. Thi bia that arie when another laifier eleted the tet data i related to the bia for linear regreion with inidental trunation. Thi paper eond ontribution i a proedure to reate ROC urve that provide a onitent etimate of the ROC urve that would be obtained with random tet data. Thi proedure infer the preditive power of the laifier baed on available data and plot the implied ROC urve. The derived ROC urve are baed on eonometri work on bivariate probit analyi (e.g. Van de Ven and Van Pragg (1981) and Poirier (1980)). A key differene between thi paper and Thi paper i under onideration at Pattern Reognition Letter Publi Company Aounting Overight Board. The PCAOB, a a matter of poliy dilaim reponibility for any private publiation or tatement by any of it Eonomi Reearh Fellow and employee. The view expreed in thi paper are the view of the author and do not neearily reflet the view of the Board, individual Board member, or taff of the PCAOB. JACook@ui.edu

2 truth predition poitive negative poitive True Fale Poitive (T P ) Poitive (F P ) negative Fale True Negative (F N) Negative (T N) total Poitive (P ) Negative (N) Table 1: Confuion matrix. prior work on eletion problem i that the problem onidered by thi paper are not regreion equation. Setion 4.2 diue intane for whih ROC urve are biaed, but the parameter of a regreion equation would not be. Thi paper make ditributional aumption that lead to maximum likelihood problem that are imilar to thoe enountered in etimating regreion equation with trunation or inidental trunation. Under the ditributional aumption ued in thi paper, a laifier expeted ROC urve i determined by two parameter. The firt parameter determine how many poitive ae there are in the population. The eond parameter i the orrelation of the laifier output with the true latent propenity to be a poitive ae. Thi paper proedure i related to the Dorfman-Alf (1969) proedure for etimating parameter of fitted ROC urve, whih alo ue maximum likelihood etimate under parametri aumption. The Dorfman-Alf proedure doe not orret for eleting tet data with a laifier. Thi paper ontribute to the literature on evaluating laifier. Reent work have hown the onnetion between ROC urve and preiion-reall urve (Davi and Goadrih 2006) and ot urve (Hernández-Orallo et al. 2013). Other work on the propertie of evaluation metri for laifier inlude Wang et al. (2013), who how that normalized diounted umulative gain (NDCG) an onitently ditinguih laifier, and Moffat (2013), who provide propertie of evaluation metri. There doe not appear to be any exiting work on evaluating laifier with nonrandom data. Thi paper doe not onider the problem of reating a laifier with nonrandom data. To reate laifier with nonrandom training data, the eonometri literature ha built on the ample-eletion orretion regreion of Hekman (1976, 1979) (ee Van de Ven and Van Pragg (1981) for a binary laifier). The redit-oring literature ha introdued rejet inferene, whih inorporate information from uneleted item, to improve laifier performane (ee, for example, Crook and Banaik 2004). In the next etion, I derive the bia in ROC urve when the laifier being evaluated wa ued to elet the tet data. Setion 3 derive a ROC urve that i onitent with nonrandom tet data. I onider two ae of nonrandom tet data: (i) Uing the laifier that we want to evaluate to elet the tet data, and (ii) Uing an unknown laifier to elet the tet data. Monte Carlo imulation and an example with wine quality data are preented in Setion 4 and 5 to illutrate thi proedure a well a the bia found in tandard ROC urve. Setion 6 onlude. 2

3 2 Claifier and ROC urve A laifier map intane to predited lae. Thi paper foue on binary laifier, whih map to two lae (e.g., poitive and negative). While ome laifier map diretly to predited lae, thi paper foue on laifier that produe a ontinuou output. Given the laifier output and a threhold, we laify all intane above the threhold a poitive and all intane below the threhold a negative. The onfuion matrix in Figure 1 define true poitive (TP), true negative (TN), poitive (P), and negative (N). We define enitivity and peifiity a Senitivity = T P P, and (1) Speifiity = T N N. (2) ROC urve, whih plot enitivity a a funtion of peifiity for all poible threhold, illutrate a laifier trade-off between true poitive and fale negative. A higher value of enitivity for a given value of peifiity indiate better performane. The area under the ROC urve (AUC) i a ommonly ued metri for evaluating a laifier performane (a deribed by Bradley (1997)). If the laifier output ha no onnetion to the true la, the expeted AUC would be.5. An exellent introdution to ROC urve i provided by Fawett (2006). 2.1 Evaluating a laifier that wa ued to elet the tet data Thi etion how that ROC urve are biaed downward for the laifier that wa ued to elet the tet data. Let u denote the ontinuou output of laifier A for eah intane i a a i. I aume that there i ome unoberved propenity to be a poitive ae and denote thi propenity a p i for eah intane i. The true laifiation of eah intane i { poitive if pi p outome i = negative otherwie, (3) where p i the threhold for an intane to be a poitive ae. A value of p = 0 indiate that half of the obervation are poitive ae. The la kew inreae with the abolute value of p. Throughout thi paper, I treat both p i and a i a (poibly orrelated) random variable. The modeler never oberve p i, only outome i. For a given threhold, we an give probabiliti definition of enitivity and peifiity: Senitivity = Prob(a i > p i > p ), and (4) Speifiity = Prob(a i < p i < p ). (5) The value in Equation (1) and (2) provide ample etimate of thee probabilitie. Thi etion onider the implet form of hooing tet data baed on the laifier output: hooing all intane with a value of a i above (for ome ontant ). We denote enitivity and peifiity onditional on eletion a Senitivity Seletion = Prob(a i > p i > p, a i > ), and (6) Speifiity Seletion = Prob(a i < p i < p, a i > ). (7) When data are hoen baed on the laifier output, the etimate in Equation (1) and (2) provide an etimate of the value in Equation (6) and (7) intead of the value in Equation (4) and (5). The following lemma will aid in proving our reult regarding the bia in tandard ROC urve for nonrandom tet data. 3

4 Lemma 1 For a fixed value of, onditioning on eletion (i) Inreae enitivity, i.e. Senitivity < Senitivity Seletion for all < <, and (ii) Dereae peifiity, i.e. Speifiity > Speifiity Seletion for all < <. All proof are provided in the appendix. For a given utoff level, eletion move enitivity and peifiity in oppoite diretion. The intuition for thi reult i that, a we fou on intane that our laifier onider more likely to be poitive ae, we will have more poitive ae in our tet data. Senitivity, whih i onditional on the number of poitive ae, i biaed downward a the relative prevalene of poitive ae inreae. Similarly, peifiity i biaed upward a the relative number of negative ae dereae. The ROC urve plot enitivity a a funtion of peifiity: Senitivity( Speifiity) =Prob(a i > p i > p ), where atifie Speifiity = Prob(a i < p i < p ). (8) Up to thi point, we have not made any ditributional aumption. To derive analytial reult about the effet of eletion on ROC urve, it i ueful to aume that p i and a i ome from a bivariate normal ditribution: ( pi a i ) ([ 0 N 2 0 ] [ 1 ρap, ρ ap 1 The multivariate normal ditribution i hoen beaue of the relative eae of working with onditional ditribution. Given that the ale of the unoberved rik i arbitrary, I define the mean and variane of p i to be zero and one. Thi i only done for notational impliity and p i an be redefined uh that i ha mean zero and variane one. The aumption that the laifier output i normally ditributed i an eaily tetable aumption. We are now ready to tate the main reult of thi etion. Propoition 2 When tet data are eleted baed on the laifier that we want to evaluate, enitivity i lower for all value of peifiity between zero and one. The aumed bivariate ditribution i a uffiient but not neeary ondition for Propoition 2. The downward bia in the ROC urve i reated by trunating the ditribution of the laifier output. Trunation aue an attenuation bia in pereived orrelation between the laifier output and the latent propenity to be a poitive ae. Thi attenuation bia aue the ROC urve to ave in. 2.2 Evaluating a laifier with tet data eleted by an another laifier I now onider the ae of uing another laifier, with output denoted a b, to elet the tet data. Thi paper foue on ituation in whih b i not oberved. Appendix B explore the ae when b i oberved. We aume that eah intane of b an be written a b i = δ X i + γ a i + ε i, where X i i a vetor of feature for ae i and ε i i a tandard normal random variable. The parameter δ i a vetor of oeffiient and γ indiate the degree to whih the laifier output ]). 4

5 Figure 1: ROC urve for a laifier that wa ued to elet the tet data. The imulation above ue ρ ap =.7 and p = 0. For data that wa eleted by the laifier, 1,000 intane are drawn from the bivariate normal ditribution and the 500 draw with the greatet value of a are hoen. wa inorporated into the eletion proe. I aume that ε i mean independent of X and α, i.e. E(ε X, α) = 0. Thi aumption allow for etimation of δ and γ by a probit regreion. The eletion rule i { Seleted if δ Xi + γ a i + ε i >. (9) Not eleted otherwie When δ = 0, γ = 1, and Var(ε) = 0, thi eletion rule redue to the ae explored in Setion 2.1. A poitive orrelation between ε and p indiate that information that ued to elet the tet data, whih i not inluded in a, i preditive of poitive ae. 3 ROC urve for nonrandom tet data Thi paper proedure for reating ROC urve that are robut to ample eletion i to infer the preditive power of the laifier (taking trunation into onideration) then draw the ROC urve that i implied by our ditributional aumption. The propoed proedure ha the following three tep. Step 1 Subtrat the mean and divide by the tandard deviation to tandardize the laifier output. The mean and tandard deviation hould be baed on all of the data, not only on the eleted intane. Step 2 Etimate p and the orrelation between the laifier output and the latent propenity to be a poitive ae, i.e. ρ ap. Step 3 Draw the ROC urve that i implied by our etimate in Step 2 and Senitivity( Speifiity) =Prob(a i > p i > p ), where atifie Speifiity = Prob(a i < p i < p ). (10) 5

6 Figure 2: ROC urve with tet data eleted by another laifier. The imulation above ue ρ ap = ρ εp =.7, γ = 0, and p = 0. For data that wa eleted by the laifier, 1,000 intane are drawn from the bivariate normal ditribution and the 500 draw with the greatet value of γ a + ε are hoen. To draw the ROC that implied by thee etimate (denoted here a p and ρ ap ), begin with a et of utoff with uffiiently large range (e.g., -4 to 4). For eah utoff [ 4, 4], we find the orreponding value of enitivity a Prob(a i > p i > p ) = and peifiity a [ 1 Φ( p ) ] 1 Prob(a i < p i < p ) = Φ( p ) 1 [ ( [ ] )] φ(a) 1 Φ p 2 ρ ap a / 1 ρ ap da (11) ( [ ] ) φ(a) Φ p 2 ρ ap a / 1 ρ ap da. (12) The ROC urve that we draw in Step 3 i a determiniti funtion of the maximum likelihood etimate from Step 2. By the funtional invariane property of maximum likelihood etimate, we know that the ROC urve drawn in Step 3 i a onitent etimate of the expeted ROC urve. The remainder of thi etion derive the maximum likelihood etimate for p and the orrelation between the laifier and the latent propenity to be a poitive ae. Thee maximum likelihood etimate, a well a Equation (11) and (12), are baed on an aumption of multivariate normality. The example in Setion 4 illutrate the performane of thi proedure for a ae when the laifier output i not normally ditributed. 6

7 3.1 Evaluating a laifier that wa ued to elet the tet data After eleting on a, the likelihood funtion for the data an be expreed a L = Φ( [p ρ ap a i ]/ 1 ρ 2 ap) 1(outome i=poitive) i Φ([p ρ ap a i ]/ 1 ρ 2 ap) 1(outomei=negative) φ(a i ), (13) where 1( ) i the indiator funtion. We an find the maximum likelihood etimate for ρ ap and p from ρ ap, p = arg max [1(outome i = poitive] ln[φ( [p ρ ap a i ]/ 1 ρ 2 ρ ap,p ap)] i + [1(outome i = negative)] ln[φ([p ρ ap a i ]/ 1 ρ 2 ap)] (14) i The maximum likelihood etimate only depend on the intane that were eleted by the laifier. The non-eleted ae do not provide any additional information. An additional benefit of thi proedure i that it provide an etimate of p. We an ue the marginal ditribution of p i to infer the perent of poitive ae in the population. 3.2 Evaluating a laifier with tet data eleted by another laifier Under the eletion rule { Seleted if δ Xi + γ a i + ε i > Not eleted otherwie, the likelihood funtion i L = Φ 2 (δ X i + γ a i, (p a i ρ ap )/ 1 ρ 2 ap ; ρ εp ) 1(outome i=poitive) i Φ 2 (δ X i + γ a i, (p a i ρ ap )/ 1 ρ 2 ap ; ρ εp ) 1(outome i=negative) Φ( [δ X i + γ a i ]) 1(outome i=na). (15) Thi i a reparameterization of the likelihood derived by Van de Ven and Van Pragg (1981). The parameter δ, γ,, ρ ap, and ρ εp an be etimated by maximizing the likelihood funtion in Equation (15). 4 Simulation Thi etion report the reult of imulation exerie for both of the proedure preented in the previou etion. The purpoe of thee imulation i to illutrate the performane of inferred ROC urve a well a the bia that arie in tandard ROC urve. 4.1 Evaluating a laifier that wa ued to elet the tet data I firt imulate the ROC urve that i obtained with random tet data. For eah Monte Carlo run, I draw 500 obervation from the ditribution ( ) ([ ] [ ]) pi 0 1 ρap N 2, 0 ρ ap 1 a i 7

8 Evaluating a laifier that wa ued to elet the tet data ρ ap =.2 ρ ap =.5 ρ ap =.7 AUC for ROC urve with a random ample (.026) (.022) (.018) AUC for tandard ROC urve with data eleted by a laifier (.027) (.025) (.025) AUC for inferred ROC urve with data eleted by a laifier (.041) (.034) (.027) Portion poitive ae in nonrandom tet data (.022) (.022) (.021) Table 2: Reult baed on 10,000 imulation. Eah imulation i baed on a ample of 500 draw. Mean value aro the imulation are preented with tandard deviation in parenthei. The parameter p i et to zero for all imulation o the portion of poitive ae in the unbiaed ae i.5. and define the outome a outome i = { poitive if pi p negative otherwie. I then alulate the AUC for A. Thi erve a an etimate of the unbiaed AUC. Next, I imulate the ROC urve that reult with eletion and with the inferred ROC urve. I draw 1,000 obervation from thi ditribution, ort them by the value of a and keep the top 500. Thi i done o that there i a high degree of eletivity, but the number of obervation ued to generate the ROC urve i the ame for both the random and nonrandom ae. I then ue the 500 nonrandom obervation to etimate ρ ap and p baed on Equation (14). Thee value are ued to generate the ROC urve baed on Senitivity( Speifiity) =Prob(a i > p i > p ), Speifiity = Prob(a i < p i < p ). where atifie I ue a value of p = 0 for all imulation, but vary the value of ρ ap aro imulation. ROC urve for one imulation are preented in Figure 1. The biaed ROC i aved in verion of the ROC that i obtained with random data. The inferred ROC urve preent a moothed out verion of the unbiaed ROC urve. I ue 100,000 Monte Carlo run for eah value of ρ ap. Thee reult are provided in Table 2. For better laifier, whih have larger value of ρ ap, the bia of the AUC of tandard ROC urve i larger. When ρ ap =.7, the AUC for tandard ROC when the data are eleted by the laifier i 13% le than the AUC with a random ample (.719 ompared with.830). The bia when ρ ap =.2 i only 6% (.553 ompared with.590). In eah ae, the average area under the inferred ROC urve mathe the average AUC that i obtained with a random ample. 4.2 Evaluating a laifier with tet data eleted by another laifier For eah Monte Carlo run, I draw 1,000 obervation from the ditribution p i 0 1 ρ ap ρ εp a i N 3 0, ρ ap 1 0 ε i 0 ρ εp 0 1 8

9 Evaluating a laifier with tet data eleted by another laifier ρ ap =.2 ρ ap =.5 ρ ap =.7 AUC for ROC urve with a random ample (.026) (.022) (.018) For no orrelation between laifier and data eleted by a highly preditive laifier, i.e. γ = 0, ρ εp =.7 AUC for tandard ROC urve with data eleted by a laifier (.028) (.022) (.011) AUC for inferred ROC urve with data eleted by a laifier (.031) (.058) (.042) For.5 orrelation between laifier and data eleted by a highly preditive laifier, i.e. γ = 1, ρ εp =.7 AUC for tandard ROC urve with data eleted by a laifier (.028) (.032) (.059) AUC for inferred ROC urve with data eleted by a laifier (.084) (.084) (.095) For.5 orrelation between laifier and data eleted by a le preditive laifier, i.e. γ = 1, ρ εp = 0 AUC for tandard ROC urve with data eleted by a laifier (.026) (.024) (.021) AUC for inferred ROC urve with data eleted by a laifier (.093) (.077) (.057) Table 3: Reult baed on 10,000 imulation. Eah imulation i baed on a ample of 500 draw. Mean value are preented with tandard deviation in parenthei. The parameter p i et to zero for all imulation. 9

10 and define the outome a before. I then ort the value by (γa i + ε i ) and keep the 500 larget. Thi i equivalent to etting δ = 0 in Equation (9). Aro imulation, I vary the orrelation between the laifier and the orrelation of the laifier that wa ued to elet the tet data with the latent propenity to be a poitive ae. The orrelation between the laifier i Cor(a i, b i ) = γ 1 + γ 2 and the orrelation between the laifier that wa ued to elet the tet data and the latent propenity to be a poitive ae i Cor(p i, b i ) = γ ρ ap + ρ εp 1 + γ 2. When there i no orrelation between the laifier (γ = 0), there i an upward bia in the AUC for tandard ROC urve. Thi i related to the tendeny of ROC urve to be overly optimiti when the data i kewed (Davi and Goadrih 2006, p. 233). The bia i larget (13%) when ρ ap = ρ εp =.7. The ae for whih γ = 0 illutrate another differene between the problem onidered by thi paper and the eonometri literature on ample-eletion bia. For a regreion, when there i no orrelation between the eletion rule and the regreor in the equation of interet, there i no bia for ordinary leat quare regreion. When there i a.5 orrelation between the laifier (i.e. γ = 1), the ROC urve i biaed downward. For mall poitive value of γ (reult not reported), there i an upward bia in ROC urve. A the orrelation between the laifier inreae, the bia beome more imilar to the trunation bia explored in Setion 4.1. The ae for whih ρ εp = 0 illutrate another differene between the problem onidered by thi paper and the eonometri literature on ample-eletion bia. For a regreion, when there i no orrelation between the tohati element in the eletion equation and the tohati element in the outome equation, there i no bia. By ontrat, the bottom panel of Table 3 how that there i a downward bia for ROC urve in thi ituation. For γ = 1 and ρ εp =.7, there i a notieable differene between the area under the inferred ROC urve and the AUC that are obtained with random tet data. While thee value are loer to the AUC obtained with random tet data than tandard ROC urve with nonrandom tet data, the tandard deviation of the AUC i muh larger for inferred ROC urve. In reult not hown, the differene between area under the inferred ROC urve and the AUC obtained with random tet data are dereaing in the ize of the ample. Appendix B how that, when the laifier that eleted the tet data i oberved, the average and tandard deviation of AUC from inferred ROC urve are equal to thoe obtained with random tet data. 5 An example with wine-quality data To provide a demontration of thi proedure with non-imulated data, I ue data on white wine quality from Cortez et al. (2009). 1 Thi dataet ontain eleven attribute for 4,898 white wine, inluding alohol ontent, itri aid, and reidual ugar. A detailed deription of thi data are provided by Cortez et al. For the meaure of wine quality, eah wine wa evaluated by expert and given a ore from zero to ten (with ten being the highet quality). Beaue we are intereted in binary predition, I define a wine with a ore of ix or higher a good wine and other wine a not good wine. I ue all eleven attribute in a random foret laifier (baed on Breiman (2001) and implemented in R uing Liaw and Wiener (2002) randomforet pakage) to predit (binary) wine 1 Thee data are available at the Univerity of California at Irvine Mahine Learning Repoitory, http: //arhive.i.ui.edu/ml/dataet.html. 10

11 Figure 3: Hitogram of the random foret output. The mean, variane, and kew are.71,.05, and -.62, repetively. quality. The random foret ontain 1,000 tree and trie three attribute at eah plit. I ue the firt 2/3 of the obervation (3,233 obervation) a the training data and the remaining 1/3 (1,665 obervation) a the tet data. I firt find the ROC urve for the random foret laifier uing the full et of tet data. The area under the ROC urve i.83. Next, let u uppoe that the wine expert do not have enough time to ore all of the wine in the tet data. Preferring to tate wine that i more likely to be good wine, the expert tate the half the tet data that the random foret laifier predited wa mot likely to be good wine. With only half of the tet data available, the area under the ROC urve fall to.60. I now perform the proedure deribed in Setion 3 with the half of the tet data predited to be good wine. Figure 3 preent a hitogram of the random foret ore. The ditribution i learly non-gauian. Thi example provide ome inight into the performane of thi paper propoed proedure when it aumption are not met. I tandardize the random foret ore and maximize Equation (9) to find the etimate ρ ap =.64 and p =.55. Figure 3 plot the ROC urve that are obtained with the full et of tet data, the half of the tet data that reeived a high ore from the random foret, and the ROC urve baed on our etimate of p and ρ ap. The ROC urve baed on our etimate of p and ρ ap loely mathe the ROC urve obtained with the full et of tet data. 6 Diuion and Conluion For tet data that i hoen by the laifier that we want to evaluate, ROC urve are biaed downward. When tet data wa eleted by another laifier, the diretion of the bia in the ROC i not lear. Thi paper preent a proedure for reating ROC urve that provide a onitent etimate of the ROC urve that would be obtained with random tet data. The proedure introdued here relie on ditributional aumption. The example in Setion 11

12 Figure 4: ROC urve for wine quality predition, a deribed in Setion 5. The area under the ROC urve that ue all of the tet data i.83. The area under the tandard and inferred ROC urve are.60 and.81, repetively. 5 violate the aumed ditribution for the laifier output and the area under the inferred ROC urve i a till a loe math to the area under the ROC urve that would be obtained with the full et of tet data. If non-gauian ditribution are preferred for laifier output, the multivariate ditribution ued in thi paper ould be written in term of opula, a ha been done for ample-eletion bia in a regreion etting (a in Li and Rahman (2011)). An advantage of thee ditributional aumption i that they introdue a new parameter whih meaure the orrelation between the laifier output and latent propenity to be a poitive ae. Alo, given that our inferred ROC urve are baed on maximum likelihood etimate, it i poible to ontrut onfidene band for the urve. Another advantage of thee ditributional aumption i that the etimation of the parameter p lead an etimate of the perent of poitive ae in the population. Thi parameter may be of interet to an organization like the IRS that ould ue p to etimate the perent of tax return that ontain error. A Proof Proof of Lemma 1. For (i): We want to how that Prob(a i > p i > p ) < Prob(a i > p i > p, a i > ). We aume that Prob(p i > p ) and Prob(a i > ) are both nonzero. If our eletion rule were negative enough, eletion would have no impat on enitivity: lim Prob(a i > p i > p, a i > ) = Prob(a i > p i > p ). 12

13 We will how that enitivity i monotonially inreaing in the eletion rule. We firt rewrite peifiity in term of the pdf of a i onditional on (p i > p ) a Prob(a i > p i > p, a i > ) = Prob(a i > p i > p ) Prob(a i > p i > p ) =, where f a p>p i pdf of a i onditional on (p i > p ). We take the derivative of peifiity onditional on eletion with repet to through a traight-forward appliation of Leibniz rule: ( ) d d = f a p>p () [ f a p>p (a i )da ] 2 > 0. i For (ii): We want to how that Prob(a i < p i < p ) > Prob(a i < p i < p, a i > ). We aume that Prob(p i < p ) and Prob(a i > ) are both nonzero. A in part (i), we begin by noting that if our eletion rule were negative enough, eletion would have no impat on peifiity: lim Prob(a i < p i < p, a i > ) = Prob(a i < p i < p ). We will how that peifiity i monotonially dereaing in the eletion rule. We firt rewrite peifiity in term of the pdf of a i onditional on (p i < p ) a Prob(a i < p i < p, a i > ) = Prob( < a i < p i < p ) Prob(a i > p i < p ) = f a p<p (a i)da i, f a p<p (a i )da i where f a p<p i pdf of a i onditional on (p i < p ). We take the derivative of peifiity onditional on eletion with repet to through a traight-forward appliation of Leibniz rule: ( d f a p<p (a ) i)da i d = f a p<p () [ f a p<p (a i )da i f a p<p (a ] i)da i [ f a p<p (a i )da ] 2 < 0. i f a p<p (a i )da i Proof of Propoition 2. Here, I how that enitivity for a given level of peifiity i a dereaing funtion of. Sine thi term approahe a point on the ROC urve a approahe negative infinity, a monotoni dereae in implie that any point on the ROC urve will be lower. I define enitivity for a given level of peifiity and eletion rule a Senitivity( Speifiity, ) =Prob(a i > p i > p, a i > ), Speifiity = Prob(a i < p i < p, a i > ), where atifie auming that Prob(p i < p a i > ), Prob(p i > p a i > ), and Prob(a i > ) are all nonzero. For a fixed level of peifiity, the effet of an inreae in on enitivity i d Senitivity d = Senitivity + Senitivity d. } {{ }} {{ d } Diret effet of on enitivity Indiret effet of hanging 13

14 Thee term are Senitivity Senitivity = f a p>p () [ ] 2 > 0, f a p>p () = < 0, and d d = Prob(a i < p i < p, a i > )/ Prob(a i < p i < p, a i > )/ = f a p<p () [ f a p<p (a i )da i ] f a p<p () [ f a p<p (a i )da i ] > 0, where the lat term follow from the ue of the impliit funtion theorem. After applying ome high-hool algebra, d Senitivity/d an be written a d Senitivity d = f a p>p ()f a p<p () [ ] [ ] f a p<p (a i )da i f a p<p () [ f a p>p ()f a p<p () [ f a p<p () [ f a p<p (a i )da i ] [ ] 2 ] [ f a p<p (a i )da i ] f a p<p (a i )da i ] [ ] 2. The denominator i learly poitive o we fou on the numerator. Given the bivariate ditribution that we aumed, the ondition for the numerator to be negative i [φ()] 2 Φ( ρ ap /(1 ρ ap ) 2 )Φ(ρ ap /(1 ρ ap ) 2 ) φ(a i )Φ( a i ρ ap /(1 ρ 2 ap))da i φ(a i )Φ(a i ρ ap /(1 ρ 2 ap))da i < [φ()] 2 Φ( ρ ap /(1 ρ ap ) 2 )Φ(ρ ap /(1 ρ ap ) 2 ) φ(a i )Φ( a i ρ ap /(1 ρ 2 ap))da i φ(a i )Φ(a i ρ ap /(1 ρ 2 ap))da i. Thi ondition hold for >. It follow that d Senitivity d whih implie that, for a fixed level of peifiity, enitivity i monotonially dereaing in eletivity. < 0, B Evaluating a laifier with tet data eleted by an oberved laifier A in the main text, I onider the ae of eletion of tet data baed another laifier, b. Intane i i eleted if b i > and not eleted otherwie. Unlike the main text, thi etion explore ituation in whih b i oberved. A before, I allow for orrelation between the output of laifier A and B, whih ould arie from uing imilar attribute to make predition: p i a i b i N , 1 ρ ap ρ bp ρ ap 1 ρ ab ρ bp ρ ab 1. 14

15 Evaluating a laifier with tet data eleted by an oberved laifier ρ ap =.2 ρ ap =.5 ρ ap =.7 AUC for ROC urve with a random ample (.026) (.022) (.018) For no orrelation between laifier and data eleted by a highly preditive laifier, i.e. ρ ab = 0, ρ bp =.7 AUC for tandard ROC urve with data eleted by a laifier (.029) (.022) (.011) AUC for inferred ROC urve with data eleted by a laifier (.023) (.021) (.015) For.5 orrelation between laifier and data eleted by a highly preditive laifier, i.e. ρ ab =.5, ρ bp =.7 AUC for tandard ROC urve with data eleted by a laifier (.027) (.027) (.021) AUC for inferred ROC urve with data eleted by a laifier (.025) (.020) (.015) For.5 orrelation between laifier and data eleted by a le preditive laifier, i.e. ρ ab =.5, ρ bp =.2 AUC for tandard ROC urve with data eleted by a laifier (.026) (.023) (.018) AUC for inferred ROC urve with data eleted by a laifier (.028) (.025) (.020) For.5 orrelation between laifier and data eleted by a nonpreditive laifier, i.e. ρ ab =.5, ρ bp = 0 AUC for tandard ROC urve with data eleted by a laifier (.025) (.022) (.016) AUC for inferred ROC urve with data eleted by a laifier (.029) (.026) (.021) Table 4: Reult baed on 10,000 imulation. Eah imulation i baed on a ample of 500 draw. Mean value are preented with tandard deviation in parenthei. The parameter p i et to zero for all imulation. 15

16 The likelihood funtion for the data an be expreed a L = i Φ( [p E(p i a i, b i )]/σ p ab ) 1(outome i=poitive) Φ([p E(p i a i, b i )]/σ p ab ) 1(outomei=negative) φ(a i, b i ), (16) where σ p ab i the tandard deviation of p onditional on a and b, 1 σ p ab 1 1 ρ 2 [(ρ ap ρ bp ρ ab )ρ ap + (ρ bp ρ ap ρ ab )ρ bp ], ab and the expetation of p i onditional on a i and b i i E(p i a i, b i ) = 1 1 ρ 2 [(ρ ap ρ bp ρ ab )a i + (ρ bp ρ ap ρ ab )b i ]. ab We etimate the parameter ρ ap, ρ ab, ρ bp, and p by maximizing the likelihood funtion in Equation (16). A in the main text, I ue a imulation tudy to examine the performane of the proedure. Table 4 preent thee reult. Not urpriingly, when the laifier that eleted the tet data i oberved, the average area under the inferred ROC urve are muh loer to the average area under the ROC urve that are baed on random ample. We alo ee that the tandard deviation of the area under the inferred ROC urve are loer to the tandard deviation of the area under the ROC urve baed on random ample. Referene Bradley, A. P. (1997): The ue of the area under the ROC urve in the evaluation of mahine learning algorithm, Pattern Reognition, 30(7), Breiman, L. (2001): Random foret, Mahine Learning, 45(1), Cortez, P., A. Cerdeira, F. Almeida, T. Mato, and J. Rei (2009): Modeling wine preferene by data mining from phyiohemial propertie, Deiion Support Sytem, 47(4), Crook, J., and J. Banaik (2004): Doe rejet inferene really improve the performane of appliation oring model?, Journal of Banking & Finane, 28(4), Davi, J., and M. Goadrih (2006): The relationhip between Preiion-Reall and ROC urve, in Proeeding of the 23rd International Conferene on Mahine Learning, pp ACM. Dorfman, D. D., and E. Alf (1969): Maximum-likelihood etimation of parameter of ignal-detetion theory and determination of onfidene interval: Rating-method data, Journal of Mathematial Pyhology, 6(3), Fawett, T. (2006): An introdution to ROC analyi, Pattern Reognition Letter, 27(8), Hekman, J. J. (1976): The ommon truture of tatitial model of trunation, ample eletion and limited dependent variable and a imple etimator for uh model, Annal of Eonomi and Soial Meaurement, 5(4),

17 (1979): Sample eletion bia a a peifiation error, Eonometria, 47(1), Hernández-Orallo, J., P. Flah, and C. Ferri (2013): ROC urve in ot pae, Mahine Learning, 93(1), Li, P., and M. Arhad Rahman (2011): Bayeian analyi of multivariate ample eletion model uing gauian opula, Advane in Eonometri, 27, 269. Liaw, A., and M. Wiener (2002): Claifiation and Regreion by randomforet, R New, 2(3), Moffat, A. (2013): Seven numeri propertie of effetivene metri, in Information Retrieval Tehnology, pp Springer. Poirier, D. J. (1980): Partial obervability in bivariate probit model, Journal of Eonometri, 12(2), Van de Ven, W. P. M. M., and B. M. S. Van Praag (1981): The demand for dedutible in private health inurane: A probit model with ample eletion, Journal of Eonometri, 17(2), Wang, Y., L. Wang, Y. Li, D. He, and T.-Y. Liu (2013): A Theoretial Analyi of NDCG Type Ranking Meaure, in Proeeding of the 26th Annual Conferene on Learning Theory. 17