How To Find Out If You Can Get A Better Esimaion From A Recipe Card

Transcription

1 1 OUT-OF-BAG ESTIMATION Leo Breiman* Saisics Deparmen Universiy of California Berkeley, CA Absrac In bagging, predicors are consruced using boosrap samples from he raining se and hen aggregaed o form a bagged predicor. Each boosrap sample leaves ou abou 37% of he examples. These lef-ou examples can be used o form accurae esimaes of imporan quaniies. For insance, hey can be used o give much improved esimaes of node probabiliies and node error raes in decision rees. Using esimaed oupus insead of he observed oupus improves accuracy in regression rees. They can also be used o give nearly opimal esimaes of generalizaion errors for bagged predicors. * Parially suppored by NSF Gran Inroducion: We assume ha here is a raining se T= {(yn,xn), n=1,...,n} and a mehod for consrucing a predicor Q(x,T) using he given raining se. The oupu variable y can eiher be a class label (classificaion) or numerical (regression). In bagging (Breiman[1996a]) a sequence of raining ses TB,1,..., TB,K are generaed of he same size as T by boosrap selecion from T. Then K predicors are consruced such ha he kh predicor Q(x,Tk,B) is based on he kh boosrap raining se. I was shown ha if hese predicors are aggregaed--averaging in regression or voing in classificaion, hen he resulan predicor can be considerably more accurae han he original predicor. Accuracy is increased if he predicion mehod is unsable, i.e. if small changes in he raining se or in he parameers used in consrucion can resul in large changes in he resuling predicor. The examples generaed in Breiman[1996a] were based on rees and subse selecion in regression, bu i is known ha neural nes are also unsable, as are oher well-known predicion mehods. Oher mehods such as neares neighbors, are sable. I urns ou ha bagging, besides is primary purpose of increasing accuracy, has valuable byproducs. Roughly 37% of he examples in he raining se T do no appear in a paricular boosrap raining se TB. Thus, o he predicor Q(x,TB) hese examples are unused es examples. Thus, if K = 100, each paricular example (y,x) in he raining se has abou 37 predicions among he Q(x,Tk,B) such ha Tk,B does no conain (y,x). The predicions for examples "ha are ou-of-he-bag" can be used o form accurae esimaes for imporan quaniies. For example, in classificaion, he ou-of-bag predicions can be used o esimae he probabiliies ha he example belongs o any one of he J possible classes. Applied o CART his gives a mehod for esimaing node probabiliies more accuraely han anyhing available o dae. Applied o regression rees, we ge an improved mehod for esimaing he expeced error in a node predicion. In regression, using he ou-of-bag esimaed values for he oupus insead of he acual raining se oupus gives more accurae rees. Simple and accurae

2 2 ou-of-bag esimaes can be given for he generalizaion error of bagged predicors. Unlike cross-validaion, hese require no addiional compuing. In his paper, we firs look a esimaes of node class probabiliies in CART (Secion 2), and hen a esimaes of mean-squared nodes errors in he regression version of CART (Secion 3). Secion 4 looks a how much accuracy is los by using esimaes ha are averaged over erminal nodes. Indicaions from synheic daa are ha he averaging can accoun for a major componen of he error. Secion 5 gives some heoreical jusificaion for he accuracy of ou-ofbag esimaes in erms of a poinwise bias-variance decomposiion. Secion 6 gives he effec of consrucing regression rees using he ou-of-bag oupu esimaes. In Secion 7 he ou-of-bag esimaes of generalizaion error for bagged predicors are defined and sudied. The Appendix gives some saisical deails. The presen work came from wo simuli. One was he dissaisfacion, over many years, bu growing sronger more recenly, wih he biased node class probabiliy and error esimaes in CART. The oher consised of wo papers. One, by Tibshirani[1996], proposes an ou-of-bag esimae as par of a mehod for esimaing generalizaion error for any classifier. The second, by Wolper and Macready [1996], looks a a number of mehods for esimaing generalizaion error for bagged regressions--among hem, a mehod using ou-of-bag predicions equivalen o he mehod we give in Secion Esimaing node class probabiliies Assume ha he raining se T consiss of independen draws from an Y,X disribuion where Y is a J-class oupu label and X is a mulivariae inpu vecor. Define p*(x) as ha probabiliy vecor wih componens p*(j x) = P(Y=j X=x). Mos classificaion algorihms use he raining se T o consruc a probabiliy predicor p R (x,t) ha oupus a nonnegaive sum-one J-vecor p R (x,t) = (p R (1 x),...,p R (J x)) and hen classify x as ha class for which p R( j x) is maximum. In many applicaions, he componens of p R as well as he classificaion is imporan. For insance, in medical survival analysis, esimaes of he survival probabiliy is imporan. In some consrucion mehods, he resubsiuion values p R (x) are inrinsically biased esimaes of p*(x). This is rue of mehods like rees or neural nes where he opimizaion over T ries o drive all componens of p R o zero excep for a single componen ha goes o one. The resuling vecors p R are poor esimaes of he rue class probabiliies p*. Wih rees, he p R esimaes are consan over each erminal node and are given by he proporion of class j examples in he erminal node. In Breiman e. al. [1984] wo mehods were proposed o improve esimaes. In his hesis, Walker[1992] showed ha he firs of he wo mehods worked reasonably well on some synheic daa. However, he mehod only esimaes maxjp*(j ), and he resuls are difficul o compare wih hose given below. To define he problem beer--here is he arge: assume again ha he raining se T consiss of independen draws from an Y,X disribuion. Assume ha a ree C(x,T) wih erminal nodes {} has already been consruced. For a given erminal node, define p*(j ) = P(Y=j X ). The vecor p*() is wha we wan o esimae. The resubsiuion probabiliy esimae p R (x, T) is consan over each erminal node and consiss of he relaive class proporions of he raining se examples in. The ou-of-bag esimae p B is goen as follows: draw 100 boosrap replicaes of T geing T1,B,...,T100,B. For each k, build he ree classifier C(x,Tk,B). For each (y,x) in T, define p B (x) as he average of he p R (x, Tk,B) over all k such ha (y,x) is no in Tk,B. Then for any erminal node, le p B () be he average of p B (x) over all x in.

3 3 2.1 Experimenal resuls We illusrae, by experimen, he improved accuracy of he ou-of-bag esimaes compared o he resubsiuion mehod using synheic and real daa. For any wo J-probabiliies p and p' denoe p p' = p( j) p'( j) j p p' 2 = ( p( j) p'( j)) 2 j Le q*() = P(X ) be he probabiliy ha an example falls ino he erminal node. For any esimae {p()] of he {p*()} define wo error measures: E 1 = q* ( ) p*() p() E 2 = ( q*() p*() p() The difference beween he wo measures is ha large differences are weighed more heavily by E2. To simplify he inerpreaion we divide E1 by J and E2 by J. Then E1 measures he absolue average error in esimaing each componen of he probabiliy vecor, while E2 measures he corresponding rms average error. We use a es se o esimae q*() as he proporion q'() of he es se falling ino node and esimae p*() by he proporions of classes p'() in hose es se examples in. This les us esimae he wo error measures. Wih E2, i is possible o derive a correcion ha adjuss for he error in using p'() insead of p*(). The correcion is generally small if he es se is large and is derived in he Appendix. Synheic Daa We give resuls for four ses of synheic daa (see Breiman[1996b]) for specific definiions): Table 1 Synheic Daa Se Summary 2 ) 1 2 Daa Se Classes Inpus Training Tes waveform wonorm hreenorm ringnorm In all cases, here were 50 ieraions wih he raining and es ses generaed anew in each ieraion and 100 replicaions in he bagging. The resuls given are he averages over he 50 ieraions: Table 2 Node Probabiliy Errors Daa Se EB1 ER1 EB1/ER1 EB2 ER2 EB2/ER2 waveform, wonorm hreenorm ringnorm The raios of errors in he 3rd and 6h columns shows ha he ou-of-bag esimaes are giving significan error reducions.

4 4 Real Daa The daa ses we used in his experimen are available in he UCI reposiory. We used some of he larger daa ses o insure es ses large enough o give adequae esimaes of q* and p*. Table 3 Daa Se Summary Daa Se Classes Inpus Training Tes breas-cancer diabees vehicle saellie dna The daa ses lised in Table 3 consised examples whose number was he oal of he es and raining se numbers. So, for example, he breas-cancer daa se had 699 examples. The las wo daa ses lised came pre-separaed ino es and raining ses. For insance, he saellie daa came as a 4435 example raining se and a 2000 example es se. These were pu ogeher o creae a single daa se wih 6435 examples. Noe ha he raining se sizes are 100 per class. There were 50 runs on each daa se. In each run, he daa se was randomly divided ino raining and es ses wih sizes as lised in Table 3, and 100 boosrap replicaes generaed wih each raining se. The resuls, averaged over he 50 runs, are given in Table 4. Table 4 Node Probabiliy Errors Daa Se EB1 ER1 EB1/ER1 EB2 ER2 EB2/ER2 breas cancer diabees vehicle saellie dna The resuls generally show a significan decrease in esimaion error when he ou-of-bag esimaes are used. 3. Esimaing node error in regression Assume here ha he raining se consiss of independen draws from he disribuion Y,X where Y is a numerical oupu, and X a mulivariae inpu. Some mehods for consrucing a predicor f(x,t) of y using he raining se also ry o consruc an esimae of he average error in he predicion--for insance by giving an esimae of he rms error in he predicion. When hese error esimaes are based on he raining se error, hey are ofen biased oward he low side. In rees, he prediced value f(x,t) is consan over each erminal node and is equal o he average y() of he raining se oupus over he node. The wih-in node error esimae e R () for is compued as he rms error over all examples in he raining se falling ino. However, since he recursive spliing in CART is based on rying o minimize his error measure, i is clearly biased low as an esimae of he rue error rae e*() defined as:

5 5. e*()=(e((y y()) 2 X )) 1/2 The ou-of-bag esimaes are goen his way: draw 100 boosrap replicaes of T geing T1,B,...,T100,B. For each k, build he ree predicor f(x,tk,b). For each (y,x) in T, define s B (x) as he average of (y - f(x,tk,b)) 2 over all k such ha (y,x) is no in Tk,B. Then for any node define e B () as he square roo of he average over all x in of s B (x). 3.1 Experimenal resuls For any esimae e() of e*(), define wo error measures: E 1 = q*() e*() e() E 2 = ( q*() (e*() e()) 2 ) 1 2 To illusrae he improved accuracy of he ou-of-bag esimaes of e*(), five daa ses are used--he same five ha were used in Breiman[1996a]. The firs hree of he daa ses are synheic daa, he las wo real. Table 5 Daa Se Summary Daa Se Inpus Training Tes Friedman # Friedman # Friedman # Ozone Boson There were 50 runs of he procedure on each daa se, and 100 boosrap baggings in each run. In he synheic daa ses, he raining and es se was freshly generaed for each run. Wih he real daa ses, for each run a differen random spli ino raining and es se was used. The values of q* and e* were esimaed using he es se. For E2 an adjusmen was used o correc for his approximaion (see Appendix). The resuls, averaged over he 50 runs, are given in Table 6. For inerpreabiliy, he error measures displayed have been divided by he sandard deviaion of he combined raining and es se. Table 4 Node Esimaion Errors Daa Se EB1 ER1 EB1/ER1 EB2 ER2 EB2/ER2 Friedman# Friedman # Freidman # Ozone Boson Again, here are significan reducions in esimaion error. 4. Error due o wihin-node variabiliy

6 6 Suppose ha we wan o esimae a funcion of he inpus h*(x). Only if we wan o say in he srucure of a single ree wih predicions consan over each erminal node, does i make sense o esimae h*(x) by some esimae of h*(). The arge funcion h*(x) may have considerable variabiliy over he region defined by a erminal node in a ree. Given a ree wih erminal nodes and esimaes h() of h*(), denoe he corresponding esimae of h by h(x,t). The squared error in esimaing h* given ha he inpus x are drawn from he disribuion of he random vecor X can be decomposed as: E X (h(x) h(x, T )) 2 = E X (h(x) h*( ) 2 X )P(X ) + (h*() h()) 2 P(X ) The firs erm in his decomposiion we call he error due o wihin-node variabiliy, he second is he node esimaion error. The relevan quesion is how large he wihin-node variabiliy error is compared o he node esimaion error. Obviously, his will be problem dependen, bu we give some evidence below ha i may be a major porion of he error. We look a he problem of esimaing he condiional probabiliies p*(x). The expression analogous o he above decomposiion is: E X p*(x) p(x,t) 2 = q*()e X ( p*(x) p*() 2 X ) + q*() p*() p() 2 (4.1) For he four synheic classificaion daa ses used in Secion 3, p*(x) can be evaluaed exacly. Therefore, replacing he expecaions over X by averages over a large es se (5000), he error E V due o wihin-node variabiliy (firs erm in (4.1) and he error E N due o node esimaion (second erm in (4.1))can be evaluaed. There are wo mehods of node esimaion--he sandard mehod leads o error E NR and he ou-of-bag mehod o error E NB. Anoher mehod of esimaing p* is by uilizing he sequence of bagged predicors. For each (y,x) in he es se, define p B (x) as he average of he p R (x, Tk,B) over all k. Then defining E B as: E B = E X p*(x) p B (X) 2, his quaniy is also evaluaed for he synheic daa by averaging over he es daa. The experimenal procedure consiss of 50 ieraions for each synheic daa se. In each ieraion, a 5000 examples es se and 100J example raining se is generaed, and he following raios evaluaed: R 1 = 100*E NR /(E NR +E V ), R 2 = 100*(E NB -E NR )/(E NR +E V ), R 3 = 100*E NB /(E NB +E V ), R 4 = 100*E B /E V. Thus, R 1 is he percen of he oal error due o node esimaion when he resubsiuion mehod of esimaing p*() is used: R 2 is he percen of reducion in he oal error when he bagging esimae p B () is used. When p B () is used, R 3 is he percen of he oal error due o node esimaion. Finally, R 4 is he raio*100 of he error using he poinwise bagging esimae of p* o he error using esimaes consan over he erminal nodes bu using he opimal node esimae p*(). Table 5 gives he resuls.

7 7 Table 5 Error Raios(%) in Esimaing Class Probabiliies Daa Se R 1 R 2 R 3 R 4 waveform wonorm hreenorm ringnorm For hese synheic daa ses, he values for R 1 shows ha wihin-node variabiliy accouns for abou wo-hirds of he error. The second column ( R 2 ) shows ha use of he bagging esimae p B () eliminaes mos of he errors due o node esimaion, bu ha he reducion is relaively modes because of he srong conribuion of wihin-node variabiliy. The resuls for R 3 show ha when p B () is used, only abou 10% of he oal error is due o node esimaion, so ha we are close o he limi of wha can be accomplished using esimaes of p* consan over nodes. The final column gives he good news ha using he poinwise bagging esimaes p B (x) gives abou 50% reducion as compared o he bes possible node esimae. I is a bi disconcering o see how much accuracy is los by he averaging of esimaes over nodes. Smyh e.al.[1996] avoid his averaging by using a kernel densiy mehod o esimae variable wihin-node densiies. However, poinwise bagging esimaes may give comparable or beer resuls. Care mus be aken in generalizing from hese synheic daa ses as I suspec hey may have more wihin-node variabiliy han ypical real daa ses. 5. Why i works--he poinwise bias-variance decomposiion Suppose here is some underlying funcion h*(x) ha we wan o esimae using he raining se T and ha we have some mehod h(x,t) for esimaing h*(x). Then for x fixed, we can wrie, using ET o denoe expecaion over replicae raining ses of he same size drawn from he same disribuion E T (h*(x) h(x,t)) 2 =(h*(x) E T h(x,t)) 2 +E T (h(x,t) E T h(x,t)) 2. This is a poinwise in x version of he now familar bias-variance decomposiion. The ineresing hing ha i shows is ha a each poin x, ETh(x,T) has lower squared error han does h(x, T)--i has zero variance bu he same bias. Tha is, averaging h(x,t) over replicae raining ses improves performance a each individual values of x. Bagging ries o ge an esimae of ETh(x,T) by averaging over he values of h(x,tk,b). Now ETh(x,T) is compued assuming x is held fixed and T is chosen in a way ha does no depend on x. Bu if x is in he raining se, hen he Tk,B ofen conain x, violaing he assumpion. A beer imiaion of ETh(x,T) would be o leave x ou of he raining se and do bagging on he deleed raining se. Bu his is exacly wha ou-of-bag esimaion does resuling in more accurae esimaes of h*(x) a every example in he raining se. When hese are averaged over any erminal node, more accurae esimaes h B () of h*()=e(h*(x) X ) resul. 6. Trees using ou-of-bag oupu esimaes. In regression, for y,x an example in he raining se, define he ou-of-bag esimae y B for he oupu y o be he average of f(x,tk,b) over all k such ha x is no in Tk,B. The ou-of-bag oupu esimaes will generally be less noisy han he original oupus. This suggess he

8 8 possibiliy of growing a ree using he y B as he oupus values for he raining se. We did his using he daa ses described in Secion 3, and followed he procedure in Breiman[1996]. Wih he real daa ses we randomly subdivided hem so ha 90% served as he raining se, and 10% as a es se. A ree was grown and pruned using he original raining se, and he 10% es se used o ge a mean-squared error esimae. Then we did 100 boosrap ieraions and compued he y B. Finally a single ree was grown using he y B as oupus and is error measured using he 10% es se. The random subdivsion was repeaed 50 imes and he es se errors averaged. Wih he hree synheic daa ses, a raining se of 200 and a es se of 2000 were freshly generaed in each of he 50 runs. The resuls are given in Table 6. Table 6. Mean Square Tes Se Error Daa Se Error--Original Oupus Error--O-B Oupus Friedman # Freidman #2* * x1000 Friedman #3** **/1000 Boson Ozone These decreases are no as dramaic as hose given by bagging. On he oher hand, hey involve predicion by a single ree, generally of abou he same size as hose grown on he original raining se. If he desire is o increase accuracy while reaining inerpreabiliy, hen using he ou-of-bag oupus does quie well. The sory in classificaion is ha using he ou-of-bag oupu esimaes gives very lile improvemen in accuracy and can acually resul in less accurae rees. The ou-of-bag oupu esimaes consis of he probabiliy vecors p B (x). CART was modified o accep probabiliy vecors as oupus in ree consrucion, and a procedure similar o ha used in regression was ried on a number of daa ses wih disappoining resuls. The problem is wo-fold. Firs, while he probabiliy vecor esimaes may be more accurae, he classificaion depends only on he locaion of he maximum componen. Second, for daa ses wih subsanial missclassificaion raes, he ou-of-bag esimaes may produce more disorion han he original class labels. 7. Ou-of-bag esimaes for bagged predicors. In his secion, we reinforce he work by Tibshirani [1996] and Wolper and Macready[1996], boh of whom proposed using ou-of-bag esimaes as an ingredien in esimaes of generalizaion error. Wolper and Macready worked on regression ype problems and proposed a number of mehods for esimaing he generalizaion error of bagged predicors. The mehod hey found ha gave bes performance is a special case of he mehod we propose. Tibshirani used ou-of-bag esimaes of variance o esimae generalizaion error for arbirary classifiers. We explore esimaes of generalizaion error for bagged predicors. For classificaion, our resuls are new. As Wolper and Macready poin ou in heir paper, cross-validaing bagged predicors may lead o large compuing effors. The ou-of-bag esimaes are efficien in ha hey can be compued in he same run ha consrucs he bagged predicor wih lile addiional effor. Our experimens below also give evidence ha hese esimaes are close o opimal. Suppose again, ha we have a raining se T consising of examples wih an oupu variable y ha can be a mulidimensional vecor wih numerical or caegorical coordinaes, and corresponding inpu x. A mehod is used o consruc a predicor f(x,t), and a given loss funcion

9 9 L(y, f) measures he error in predicing y by f. Form boosrap raining ses Tk,B, predicors f(x, Tk,B) and aggregae hese predicors in an appropriae way o form he bagged predicor fb(x). For each y,x in he raining se, aggregae he predicors only over hose k for which Tk,B does no conaining y,x. Denoe hese ou-of-bag predicors by f OB Then he ou-of-bag esimae for he generalizaion error is he average of L(y, f OB (x)) over all examples in he raining se. Denoe by e TS he es se error esimae and by e OB he ou-of-bag error esimae. In all of he runs in Secions 2 and 3, we also accumulaed he average of e TS, e OB and e TS - e OB. We can also compue he expeced value of e TS - e OB under he assumpion ha e OB is compued using a es se of he same size as he raining se and independen of he acual es se used (see Appendix). We claim ha his expeced value is a lower bound for how well we could use he raining se o esimae he generalizaion error of he bagged predicor. Our reasoning is his: given ha he raining se is used o consruc he predicor, he mos accurae esimae for is error is a es se independen of he raining se. If we used a es se of he same size as he raining se, his is as well as can be done using his number of examples. Therefore, we can judge he efficiency of any generalizaion error esimae based on he raining se by comparing is accuracy o he esimae we would ge using a es se of he size of he raining se. Table 7 conains he resuls for he classificaion runs for boh he real and synheic daa. The las column is he raio of he experimenally observed e TS - e OB o he expeced value if he raining se were an independen es se. The closer his raio is o one, he closer o opimal e OB is. Table 8 gives he corresponding resuls for regression. Table 7 Esimaes of Generalisaion Error (% Missclassificaion) Daa Se Av e TS Av e OB Av e TS - e OB Raio waveform wonorm hreenorm ringnorm breas-cancer diabees vehicle saellie dna Table 8 Esimaes of Generalizaion Error (Mean Squared Error) Daa Se Av e TS Av e OB Av e TS - e OB Raio Friedman # Friedman #2 24.7* 23.6* * x1000 Friedman #3 32.8** 30.5** 6.7** 1.06 **/1000 Boson Ozone Tables 7 and 8 show ha he ou-of-bag esimaes are remarkably accurae. On he whole, he raio values are close o one, reflecing he accuracy of he ou-of-bag esimaes of he generalizaion error of he bagged predicors. In classificaion, he ou-of-bag esimaes

10 10 appear almos unbiased, i.e. he average of e OB is almos equal o he average of e TS. Bu he esimaes in regression may be sysemaically low. The wo values slighly less han one in Table 6 we aribue o random flucuaions. The denominaor in he raio column depends on a parameer which has o be esimaed from he daa. Error in his parameer esimae may drive he raio low )see Appendix for deails). References Breiman, L. [1996a] Bagging Predicors, Machine Learning 26, No. 2, Breiman, L. [1996b] Bias, Variance, and Arcing Classifiers, submied o Annals of Saisics, fp fp.sa.berkeley.edu pub/breiman/arcall.ps Breiman, L., Friedman, J., Olshen R., and Sone, C. [1984] Classificaion and Regression Trees, Wadsworh Smyh, P.,Gray, A. and Fayyad, U.[1996} Rerofiing Tree Classifiers Using Kernel Densiy Esimaion, Proceedings of he 1995 Conference on Machine Learning, San Francisco, CA: Morgan Kaufmann, 1995, pp Tibshirani, R.[1996] Bias, Variance, and Predicion Error for Classificaion Rules, Technical Repor, Saisics Deparmen, Universiy of Torono Walker, Michael G. (1992) Probabiliy Esimaion for ClassificaionTrees and Sequence Analysis. San-CS Ph.D. Disseraion. Deparmens of Compuer Science and Medicine, Sanford Universiy,Sanford, CA. Wolper, D.H. and Macready, W.G.[1996] An Efficien Mehod o Esimae Bagging's Generalizaion Error I. Adjusmen o E2 in classificaion Appendix Wih moderae sized es ses we can ge, a bes, only noisy esimaes of p*. Using hese esimaes o compue error raes may lead o biases in he resuls. However, a simple adjusmen is possible in he E2 error crierion. For {p()} probabiliy esimaes in he erminal nodes depending only on he examples in T, E2 is defined by E2 2 = q() p*() p() 2 (2.1) Le p'() be he class proporions of es se examples in node so ha p'() is an esimae of p*(). We assume ha he examples in S and T are independen. Le N be he number of es se examples falling ino erminal node. Condiional on N, p'(j ) imes N has a binomial disribuion B(p*(j ),N). Hence, p'(j ) has expecaion p*(j ) and variance p*(j )(1-p*(j ) )/N. Wrie p' p 2 = p* p 2 + p' p* 2 +2(p* p,p' p*). Taking expecaions of boh sides wih respec o he examples in S holding N consan gives N E p' p 2 =N p* p 2 +1 p* 2 (A.1) Puing p=0 in (A.1) gives N E p' 2 =N p* 2 +1 p* 2 (A.2

11 11 Solving (A.1) and (A.2) for N p* p 2 gives Thus, we esimae he error measure E2 as: N p* p 2 =N E p' p 2 N N 1 (1 E p' 2 ) E2 2 = q'()[ p'() p() 2 (1 p' 2 ) (N() 1)] (A.3) where N() is he number of es se examples in node, q'() is N()/N, and we define he second erm in he brackes o be zero if N()=1. This erm in (A.3) is he adjusmen. In our examples, i had only a small effec on he resuls. II. Adjusmen o E2 in regression In regression, E2 is defined as he square roo of R= q*() (e*() e()) 2. Since e*() is unknown, we esimae i as he square roo of he average over all es se (y,x) falling in of (y- y()) 2 and denoe his esimae by e'(). Le R' = q*() (e'() e()) 2. Then we wan o adjus R' so ha i is an unbiased esimae of R, i.e. so ha ER' = R, when he expecaion is aken over he es se examples. To simplify his compuaion, we assume ha: i) es se oupu values in are normally disribued wih rue mean y*(). ii) q*() y*()-y() is small. Now, e'() 2 ' = (y n n y()) 2 / N() where he sum is over all es se (y,x) falling ino. By he Cenral Limi Theorem, e'() 2 = e*() 2 + Z/ N() where Z is approximaely normally disribued wih mean zero and variance equal o he variance of (Y- y()) 2 condiional on X in. I follows ha Ee'() 2 = e*() 2, and o firs order in N(), Ee'() = e*() - EZ 2 /(8e*() 3 N()). Recall ha for a normally disribued random variable U wih variance σ 2,he variance of U 2 is 2σ 4. Using assumpion i) gives EZ 2 = 2(var(Y- y())) 2.By assumpion ii) he variance of Y- y() on can be approximaed by e*() 2. Thus, Ee'()=e*()[1.25/ N()] (A.4)

12 12 Wrie. (e'() e()) 2 =(e'() e*()) 2 +(e*() e()) 2 +2(e'() e*())(e*() e()). (A.4) Taking expecaion of (A.5) wih respec o he es se and using (A.4) gives E(e'() e()) 2 E(e*() e()) 2 +e*()e()/2n() (A.5) Approximaing e*()e() by e'() 2 gives he adjused measure: 2 E 2 = q'() [(e'() e()) 2.5e'() 2 / N()] Again, he adjusmen conribues a relaively small correcion in our runs. III. Lower bound for raining se accuracy. Suppose we have a classifier Q(x) wih rue generalizaion missclassificaion rae P*. Tha is, for he disribuion of he r.v. Y,X, e*= P(Y Q(X)). Two ses of daa T'= {(y'n,x'n), n=1,...,n'} and T= {(yn,xn), n=1,...,n} are independenly drawn from he underlying disribuion of Y,X and run hrough he classifier. The firs has has classificaion error rae e' and he second e. We evaluae g(n',n) =E e'-e. In he conex of he experimens on ou-of-bag esimaion, he firs se of daa is he given es se. The second se is he raining se. The raining se is used boh o form he bagged classifier and he ou-of-bag esimae of he generalizaion error. Suppose we had an unouched es se of he same size as he raining se and used his new es se o esimae he generalizaion error. Cerainly, we would do beer han any way of using he raining se over again o do he same hing. Thus g(n',n) is a lower bound for he accuracy of generalizaion error esimaes using a raining se of N examples, when i is being compared o a es se using N' examples. Now, e is given by e= V n / N (A.6) n where V n is one if he nh case in T is missclssified and zero oherwise. By he Cenral Limi Theorem, e=e* + Z/ N, where Z is approximaely normal wih mean zero and variance P*(1- P*). Similarly, e'=e* + Z' / N' where Z' is approximaely normal mean zero also wih variance e*(1-e*) and independen of Z. So e'-e is normal wih variance s=e*(1-e*)[(1/n)+(1/n')]. The expecaion E U of a mean zero normal variable U wih variance s is 2s/π and i was his las expression ha was used as a comparison in Secion 6, wih s esimaed using e' in place of e*. In regression, he predicor is a numerical-valued funcion f(x). The rue generalizaion error is e* = E(Y-f(X)) 2. Expression (A.6) holds wih V n =(y n f (x n )) 2. Again, e=e*+ Z/ N, where Z is approximaely normal wih mean zero and variance c/ N wih c he variance of (Yf(X)) 2. Repeaing he argumen above, e'- e is a normal variable wih mean zero and variance

13 s=c[(1/n)+(1/n') so E e'-e equals 2s/π. The variance of (Y-f(X)) 2 is given by E(Y-f(X)) 4 - (E(Y-f(X)) 2 ) 2. This value is approximaed by he using he corresponding momens over he es se, leading o he evaluaion of he lower bound. 13