Logistic Regression, AdaBoost and Bregman Distances

Transcription

1 A exteded abstact of this joual submissio appeaed ipoceedigs of the Thiteeth Aual Cofeece o ComputatioalLeaig Theoy, 2000 Logistic Regessio, Adaoost ad egma istaces Michael Collis AT&T Labs Reseach Shao Laboatoy 80 Pak Aveue, Room A253 Floham Pak, NJ mcollis@eseachattcom Robet E Schapie AT&T Labs Reseach Shao Laboatoy 80 Pak Aveue, Room A203 Floham Pak, NJ schapie@eseachattcom Yoam Sige School of Compute Sciece & Egieeig Hebew ivesity, Jeusalem 9904, Isael sige@cshujiacil Octobe, 2000 Abstact We give a uified accout of boostig ad logistic egessio i which each leaig poblem is cast i tems of optimizatio of egma distaces The stikig similaity of the two poblems i this famewok allows us to desig ad aalyze algoithms fo both simultaeously, ad to easily adapt algoithms desiged fo oe poblem to the othe Fo both poblems, we give ew algoithms ad explai thei potetial advatages ove existig methods These algoithms ca be divided ito two types based o whethe the paametes ae iteatively updated sequetially oe at a time) o i paallel all at oce) We also descibe a paameteized family of algoithms which itepolates smoothly betwee these two extemes Fo all of the algoithms, we give covegece poofs usig a geeal fomalizatio of the auxiliay-fuctio poof techique As oe of ou sequetial-update algoithms is equivalet to Adaoost, this povides the fist geeal poof of covegece fo Adaoost We show that all of ou algoithms geealize easily to the multiclass case, ad we cotast the ew algoithms with iteative scalig We coclude with a few expeimetal esults with sythetic data that highlight the behavio of the old ad ewly poposed algoithms i diffeet settigs

2 Itoductio We give a uified accout of boostig ad logistic egessio i which we show that both leaig poblems ca be cast i tems of optimizatio of egma distaces I ou famewok, the two poblems become vey simila, the oly eal diffeece beig i the choice of egma distace: uomalized elative etopy fo boostig, ad biay elative etopy fo logistic egessio The similaity of the two poblems i ou famewok allows us to desig ad aalyze algoithms fo both simultaeously We ae ow able to boow methods fom the maximum-etopy liteatue fo logistic egessio ad apply them to the expoetial loss used by Adaoost, especially covegece-poof techiques Covesely, we ca ow easily adapt boostig methods to the poblem of miimizig the logistic loss used i logistic egessio The esult is a family of ew algoithms fo both poblems togethe with covegece poofs fo the ew algoithms as well as Adaoost Fo both Adaoost ad logistic egessio, we attempt to choose the paametes o weights associated with a give family of fuctios called featues o weak hypotheses Adaoost woks by sequetially updatig these paametes oe by oe, wheeas methods fo logistic egessio, most otably iteative scalig [, 2], update all paametes i paallel o each iteatio Ou fist ew algoithm is a method fo optimizig the expoetial loss usig paallel updates It seems plausible that a paallel-update method will ofte covege faste tha a sequetial-update method, povided that the umbe of featues is ot so lage as to make paallel updates ifeasible A few expeimets descibed at the ed of this pape suggest that this is the case Ou secod algoithm is a paallel-update method fo the logistic loss Although paallel-update algoithms ae well kow fo this fuctio, the updates that we deive ae ew ecause of the uified teatmet we give to the expoetial ad logistic loss fuctios, we ae able to peset ad pove the covegece of the algoithms fo these two losses simultaeously The same is tue fo the othe algoithms peseted i this pape as well We ext descibe ad aalyze sequetial-update algoithms fo the two loss fuctios Fo expoetial loss, this algoithm is equivalet to the Adaoost algoithm of Feud ad Schapie [] y viewig the algoithm i ou famewok, we ae able to pove that Adaoost coectly coveges to the miimum of the expoetial loss fuctio This is a ew esult: Although Kivie ad Wamuth [9] ad Maso et al [22] have give covegece poofs fo Adaoost, thei poofs deped o assumptios about the give miimizatio poblem which may ot hold i all cases Ou poof holds i geeal without assumptios Ou uified view leads istatly to a sequetial-update algoithm fo logistic egessio that is oly a mio modificatio of Adaoost ad which is vey simila to oe poposed by uffy ad Helmbold [4] Like Adaoost, this algoithm ca be used i cojuctio with ay classificatio algoithm, usually called the weak leaig algoithm, that ca accept a distibutio ove examples ad etu a weak hypothesis with low eo ate with espect to the distibutio Howeve, this ew algoithm povably miimizes the logistic loss athe tha the aguably less atual expoetial loss used by Adaoost A potetially impotat advatage of the ew algoithm fo logistic egessio is that the weights that it places o examples ae bouded i This suggests that it may be possible to use the ew algoithm i a settig i which the boostig algoithm selects examples to peset to the weak leaig algoithm by filteig a steam of examples such as a vey lage dataset) As poited out by Wataabe [27] ad omigo ad Wataabe [3], this is ot possible with Adaoost sice its weights may become extemely lage They povide a modificatio of Adaoost fo this pupose i which the weights ae tucated at Ou ew algoithm may be a viable ad cleae alteative We ext descibe a paameteized family of iteative algoithms that icludes both paallel- ad sequetialupdate algoithms ad that also itepolates smoothly betwee the two extemes The covegece poof that we give holds fo this etie family of algoithms 2

3 Although most of this pape cosides oly the biay case i which thee ae just two possible labels associated with each example, it tus out that the multiclass case equies o additioal wok That is, all of the algoithms ad covegece poofs that we give fo the biay case tu out to be diectly applicable to the multiclass case without modificatio Fo compaiso, we also descibe the geealized iteative scalig algoithm of aoch ad Ratcliff [] I edeivig this pocedue i ou settig, we ae able to elax oe of the mai assumptios usually equied by this algoithm The pape is ogaized as follows: Sectio 2 descibes the boostig ad logistic egessio models as they ae usually fomulated Sectio 3 gives backgoud o optimizatio usig egma distaces, ad Sectio 4 the descibes how boostig ad logistic egessio ca be cast withi this famewok Sectio 5 gives ou paallel-update algoithms ad poofs of thei covegece, while Sectio gives the sequetialupdate algoithms ad covegece poofs The paameteized family of iteative algoithms is descibed i Sectio 7 The extesio to multiclass poblems is give i Sectio 8 I Sectio 9, we cotast ou methods with iteative scalig I Sectio 0, we give some iitial expeimets that demostate the qualitative behavio of the vaious vaiats i diffeet settigs Pevious wok Vaiats of ou sequetial-update algoithms fit ito the geeal family of acig algoithms peseted by eima [4, 3], as well as Maso et al s Ayoost family of algoithms [22] The ifomatio-geometic view that we take also shows that some of the algoithms we study, icludig Adaoost, fit ito a family of algoithms descibed i 97 by egma [2] fo satisfyig a set of costaits Ou wok is based diectly o the geeal settig of Laffety, ella Pieta ad ella Pieta [2] i which oe attempts to solve optimizatio poblems based o geeal egma distaces They gave a method fo deivig ad aalyzig paallel-update algoithms i this settig though the use of auxilliay fuctios All of ou algoithms ad covegece poofs ae based o this method Ou wok builds o seveal pevious papes which have compaed boostig appoaches to logistic egessio Fiedma, Hastie ad Tibshiai [7] fist oted the similaity betwee the boostig ad logistic egessio loss fuctios, ad deived the sequetial-update algoithm Logitoost fo the logistic loss Howeve, ulike ou algoithm, theis equies that the weak leae solve least-squaes poblems athe tha classificatio poblems Aothe sequetial-update algoithm fo a diffeet but elated poblem was poposed by Cesa-iachi, Kogh ad Wamuth [] uffy ad Helmbold [4] gave coditios ude which a loss fuctio gives a boostig algoithm They showed that miimizig logistic loss does lead to a boostig algoithm i the PAC sese, which suggests that ou algoithm fo this poblem, which is vey close to theis, may tu out also to have the PAC boostig popety Laffety [20] wet futhe i studyig the elatioship betwee logistic egessio ad the expoetial loss though the use of a family of egma distaces Howeve, the settig descibed i his pape appaetly caot be exteded to pecisely iclude the expoetial loss The use of egma distaces that we descibe has impotat diffeeces leadig to a atual teatmet of the expoetial loss ad a ew view of logistic egessio Ou wok builds heavily o that of Kivie ad Wamuth [9] who, alog with Laffety, wee the fist to make a coectio betwee Adaoost ad ifomatio geomety They showed that the update used Moe specifically, egma [2] descibes optimizatio methods based o egma distaces whee oe costait is satisfied at each iteatio, fo example, a method whee the costait which makes the most impact o the objective fuctio is geedily chose at each iteatio The simplest vesio of Adaoost, which assumes weak hypotheses with values i, is a algoithm of this type if we assume that the weak leae is always able to choose the weak hypothesis with miimum weighted eo 3

4 by Adaoost is a fom of etopy pojectio Howeve, the egma distace that they used diffeed slightly fom the oe that we have chose omalized elative etopy athe tha uomalized elative etopy) so that Adaoost s fit i this model was ot quite complete; i paticula, thei covegece poof depeded o assumptios that do ot hold i geeal Kivie ad Wamuth also descibed updates fo geeal egma distaces icludig, as oe of thei examples, the egma distace that we use to captue logistic egessio 2 oostig, logistic models ad loss fuctios!###$ ' Let & o istace space ) 0!243, ad each label +*-,/ ### We assume that we ae also give a set of eal-valued fuctios o ), 5 be a set of taiig examples whee each istace & belogs to a domai 57 Followig covetio i the Maximum-Etopy liteatue, we call these fuctios featues; i the boostig liteatue, these would be called weak o base hypotheses Note that, i the temiology of the latte liteatue, these featues coespod to the etie space of base hypotheses athe tha meely the base hypotheses that wee peviously foud by the weak leae We study the poblem of appoximatig the s usig a liea combiatio of featues That is, we ae iteested i the poblem of fidig a vecto of paametes 8 *-9 such that :0;<= >!@ A 5 & is a good appoximatio of How we measue the goodess of such a appoximatio vaies with the task that we have i mid Fo classificatio poblems, it is atual to ty to match the sig of :0;< to, that is, to attempt to miimize EF 4 C:0; = whee GH is if G is tue ad othewise Although miimizatio of the umbe of classificatio eos may be a wothwhile goal, i its most geeal fom, the poblem is itactable see, fo istace, [8]) It is theefoe ofte advatageous to istead miimize some othe oegative loss fuctio Fo istace, the boostig algoithm Adaoost [, 24] is based o the expoetial loss IJ K! :0;< It ca be veified that Eq ) is uppe bouded by Eq 2); howeve, the latte loss is much easie to wok with as demostated by Adaoost iefly, o each of a seies of ouds, Adaoost uses a oacle o suboutie called the weak leaig algoithm to pick oe featue weak hypothesis) 5, ad the associated paamete A is the updated It has bee oted by eima [3, 4] ad vaious late authos [7, 22, 23, 24] that both of these steps ae doe i such a way as to appoximately) cause the geatest decease i the expoetial loss I this pape, we show fo the fist time that Adaoost is i fact a povably effective method fo fidig paametes 8 which miimize the expoetial loss assumig the weak leae always chooses the best 5 ) We also give a A etiely ew algoithm fo miimizig expoetial loss i which, o each oud, all of the paametes ae updated i paallel athe tha oe at a time Ou hope is that this paallel-update algoithm will be faste tha the sequetial-update algoithm; see Sectio 0 fo pelimiay expeimets i this egad Istead of usig :0; as a classificatio ule, we might istead postulate that the s wee geeated stochastically as a fuctio of the s ad attempt to use :0;< to estimate the pobability of the associated ) 2) 4

5 Ž ƒ label A well-studied way of doig this is to pass :0; though a logistic fuctio, that is, to use the estimate MON L 2QP RTSV=WYX0Z\[^] The likelihood of the labels occuig i the sample the is _ IJ K C:0;<& Maximizig this likelihood the is equivalet to miimizig the log loss of this model `ba R IJcK d! :0;<= Geealized ad impoved iteative scalig [, 2] ae popula paallel-update methods fo miimizig this loss I this pape, we give a alteative paallel-update algoithm which we compae to iteative scalig techiques i pelimiay expeimets i Sectio 0 3 egma-distace optimizatio I this sectio, we give backgoud o optimizatio usig egma distaces This will fom the uifyig basis fo ou study of boostig ad logistic egessio The paticula set-up that we follow is take pimaily fom Laffety, ella Pieta ad ella Pieta [2] 9 be a cotiuously diffeetiable ad stictly covex fuctio defied o a closed, covex Let egfihkj set hml 9 p The egma distace associated with e is defied fo o * h to be qs +! tovu xeyto ey {z ey to Fo istace, whe q is the uomalized) elative etopy Co u e}to ~ `ba ~ `ba ~ ƒ ~ # ~ It ca be show that, i geeal, evey egma distace is oegative ad is equal to zeo if ad oly if its two agumets ae equal Thee is a atual optimizatio poblem that ca be associated with a egma distace, amely, to fid the vecto o * h p that is closest to a give vecto * h subject to a set of liea costaits These costaits ae specified by a ˆŠ Œ matix ad a vecto o Ž * h The vectos o satisfyig these costaits ae those fo which ow w o w Thus, the poblem is to fid N y a q p ^ c $ Co u whee gšco * hmf o œž o ž 3) 4) 5) 5

6 * Ÿ Ÿ Ÿ u o The covex dual of this poblem gives a alteative fomulatio Hee, the poblem is to fid the vecto of a paticula fom that isp closest to a give vecto o Ž The fom of such vectos is defied via the Legede tasfom, witte Ÿ : p + N a c 0 q The Legede tasfom is a fuctio which maps h 9 equivalet to z e}tÿ p + Co u z ey j h sig calculus, this ca be see to be q Fo istace, whe is uomalized elative etopy, it ca be veified usig calculus that p + S < Note that, i ode fo Eq ) to have a solutio i all cases, we eed to assume that z e is a bijective oe-to-oe ad oto) mappig fom the iteio of h to 9 We make this assumptio fo the emaide of the pape Fom Eq ), ad the bijective popety of z e, it ca be show that the tasfom has the followig useful additive popety: p ª«+ p ª! tÿ Ÿ 8) Fo a give ˆ matix p ad vecto * h, we coside vectos obtaied by takig the Legede tasfom of a liea combiatio of colums of, Ÿ p 0 ±8 R with the vecto p, that is, vectos i the set ²P 8 *³9 The dual optimizatio poblem ow ca be stated to be the poblem of fidig whee is the closue of N y a / µ q The emakable fact about these two optimizatio poblems is that thei solutios ae the same, ad, moeove, this solutio tus out to be the uique poit at the itesectio of ad We take the statemet of this theoem fom Laffety, ella Pieta ad ella Pieta [2] The esult appeas to be due to Csiszá [8, 9] ad Topsoe [2] A poof fo the case of omalized) elative etopy is give by ella Pieta, ella Pieta ad Laffety [2] See also Csiszá s suvey aticle [0] as well as Ceso ad Zeios s book [5] p=¹ o p,,, h, e qs,, a uique * h p ¹ º q 2 tovu q p=¹0 Co u q p=¹ 3 N a q / µ Ž o»u p=¹ 4 N a c $ q p tovu Theoem Let Ž satisfyig: Ž ovu 3 qs ad be as above Assume ovu Ž p=¹ fo ay o * Moeove, ay oe of these fou popeties detemies p ¹ uiquely ad p * p ) 7) 9) The thee exists This theoem will be extemely useful i povig the covegece of the algoithms descibed below We will show i the ext sectio how boostig ad logistic egessio ca be viewed as optimizatio poblems of the type give i pat 3 º of the theoem The, to pove optimality, we oly eed to show that ou algoithms covege to a poit i

7 ƒ 4 oostig ad logistic egessio evisited We etu ow to the boostig ad logistic egessio poblems outlied i Sectio 2, ad show how these ca be cast i the fom of the optimizatio poblems outlied above Recall that fo boostig, ou goal is to fid 8 such that IJ A 5 ¾À = is miimized, o, moe pecisely, ### if the miimum is ot attaied at a fiite 8, the we seek a pocedue fo fidig a sequece 8 8 which causes this fuctio to covege to its ifimum Fo shothad, we call this the ExpLoss poblem To view this poblem i the fom give i Sectio 3, we let okãâ Ž p,»ä the all s ad all s vectos) We let ÅÆ 5 &, fom which it follows that C ±8 Ç A C5 & The space hk 9 Fially, we take e to be as i p + q Eq 4) so that is the uomalized elative etopy As oted ealie, i this case, Ÿ is as give i Eq 7) I paticula, this meas that Ê Ë p ÉÈ *³9 ÍÌ IJ K ¼½ A!@ t5 ¾À = 8 * 9 =Î Ï Ð ÌÌÌÌÌ Futhemoe, it is tivial to see that Ñ CÂÒu ) p 0 so that ÂÓu{Ÿ Ô8 Ñ is equal to Eq 0) Thus, miimizig p CÂÒu p ove * is equivalet to miimizig Eq 0) y Theoem, this is equivalet to fidig * satisfyig the costaits fo Ö 0###^ ÕÅ{ t5 = Logistic egessio ca be educed to a optimizatio poblem of this fom i ealy the same way Recall that hee ou goal is to fid 8 o a sequece of 8 s) which miimize `ba ¼½ R IJ K ¼½ Fo shothad, we call this the LogLoss poblem We defie Ž p ^Ø0Ù/ vecto is still costat, but ow is defied to A 5 ¾ÀQ¾À 0) 2) 3) o ad exactly as fo expoetial loss The Ä, ad the space h is ow esticted to be These ae mio diffeeces, howeve The oly impotat diffeece is i the choice of the fuctio e, amely, eyto The esultig egma distace is ÑÛ Üo»u ~ ~ `ba ~ `ba ƒ ~ ~ `Úa ~ ~! `ba ƒ ~ / 7

8 > P * Ÿ Paametes: hml 9 efihkj p 9 satisfyig Assumptios ad 2 * h qs p E such that ÜÂÒu 0 Iput: Matix * ÑÝi whee, fo all Þ,!@ Å{ Pcß ### Output: 8 8 such that ` Ú qã p àúá â CÂÒu{Ÿ ±8 à a7ä qs ; 9å Let 8 xâ 0dÙc### Fo æo ç p à xÿ ç Fo Ö2 : p 0 ±8 à 0###^ : ç pdate paametes: 8 à x8 à T à àté àté ô àté Õê ëìîíï Zñðò ó Õê ëìîíï Zñðò ó Ù `ba³õ àté àté ö CÂÒu{Ÿ àté àté P Å P P Å{ P p ±8 = Figue : The paallel-update optimizatio algoithm Tivially, so that ÑÛ CÂÒu Fo this choice of e, it ca be veified usig calculus that whee øo ove p * costaits i Eq 2) p + `Úa! 4) S < S < 5) Ê Ë p È Ì ø ¼½ t5 ¾À} & 8 * 9 òî Ï Ð ùts [ g Thus, ÌÌÌÌÌ ÑÛ p 0 ÂÓu{Ÿ Ô8 ÑÛ is equal to Eq 3) so miimizig ÜÂÒu is equivalet to miimizig Eq 3) As befoe, this is the same as fidig * 5 Paallel optimizatio methods satisfyig the I this sectio, we descibe a ew algoithm fo the ExpLoss ad LogLoss poblems usig a iteative method i which all weights A ae updated o each iteatio The algoithm is show i Fig The algoithm ca 8

9 , p Ÿ Ÿ ÿ 3 P be used with ay fuctio e satisfyig cetai coditios descibed below; i paticula, we will see that it ca be used with the choices of e give i Sectio 4 Thus, this is eally a sigle algoithm that ca be used fo both loss-miimizatio poblems by settig the paametes appopiately Note that, without of geeality, we assume i this sectio that fo all istaces Þ, > Å{ Pcú The algoithm is vey simple O each iteatio, the vecto à is computed as show ad added to the paamete vecto 8 à We assume fo all ou algoithms that the iputs ae such that ifiite-valued updates eve occu This algoithm is ew fo both miimizatio poblems Optimizatio methods fo ExpLoss, otably Adaoost, have geeally ivolved updates of oe featue at a time Paallel-update methods fo LogLoss ae well kow see, fo example, [, 2]) Howeve, ou updates take a diffeet fom fom the usual updates deived fo logistic models; we discuss p the diffeeces i Sectio 9 A useful poit is that the distibutio à is a simple fuctio of the pevious distibutio p à y Eq 8), This gives p à à Ÿ fo ExpLoss ad LogLoss espectively p 0 mc8 à T à tÿ p à p 4 ±8 à! à! Ê Ë àté é IJ KÇû >!@ ô àté Å{ ü ÉÈ àté 4ý àté IJ K û >!@ ô àté Å ü àté Õþ à ) 7) We will pove ext that the algoithm give i Fig coveges to optimality fo eithe loss We pove this abstactly fo ay matix p ad vecto, ad fo ay fuctio e satisfyig the followig assumptios: Assumptio Fo ay *³9, p * h, Assumptio 2 Fo ay ÿ q ÜÂÒu{Ÿ, the set p +! * h q P q CÂÓu CÂÓu ù ù S < ^! is bouded We will show late that the choices of e give i Sectio 4 satisfy these assumptios which will allow us to pove covegece fo ExpLoss ad LogLoss To pove covegece, we use the auxiliay-fuctio techique of ella Pieta, ella Pieta ad Laffety [2] Vey oughly, the idea of the poof is to deive a oegative lowe boud called a auxiliay fuctio o how much the loss deceases o each iteatio Sice the loss eve iceases ad is lowe bouded by zeo, the auxiliay fuctio must covege to zeo The fial step is to show that whe the auxiliay fuctio is zeo, the costaits defiig the set must be satisfied, ad theefoe, by Theoem, we must have coveged to optimality p p ### Moe fomally, we defie a auxiliay fuctio fo a sequece ad matix to be a cotiuous fuctioôf hkj 9 satisfyig the two coditios: ad q Ü u p à 2 q p CÂÓu p àe  p àe 8) 9) 9

10 , p q p q L efoe povig covegece of specific algoithms, we pove the followig lemma which shows, oughly, that if a sequece has a auxiliay fuctio, the the sequece coveges to the optimum poit p=¹ Thus, povig covegece of a specific algoithm educes to simply fidig a auxiliay fuctio Lemma 2 Let be a p p ### auxiliay fuctio fo ad matix p Assume the à s lie i a compact subspace of whee is as i Eq 9); i paticula, this will be the case if Assumptio 2 holds ad q E CÂÓu The ` b p àúá â à p=¹ N a q = / µ ÜÂÒu Poof: q y qs p coditio 8), ÜÂÓu à p is a oiceasig sequece As is the case fo all egma distaces, ÜÂÓu à is also bouded below by zeo Theefoe, the sequece of diffeeces q p CÂÒu à q p ÜÂ u à p à must covege to zeo y coditio 8), this meas that must also covege to zeo ecause we assume that p L p the à p s lie i a compact space, the sequece of à s must have a subsequece covegig to some poit * h y cotiuity of pl p+, we have p L Theefoe, * by p L coditio 9), whee is as i Eq 5) O the othe had, is the limit of a sequece of poits i so * p L xº L p=¹ Thus, * so by Theoem p p=¹ p This agumet ad the uiqueess of show that the à p=¹ s have oly a sigle limit poit Suppose thatp the###b3 p=¹ q p=¹ etie sequece did ot covege to The we could fid a ope set cotaiig such that cotais ifiitely may p=¹ poits ad theefoe has a limit poit which must be i the closed ad so must be diffeet fom This, we have aleady agued, is impossible Theefoe, the set h p ¹ etie sequece coveges to We ca ow apply this lemma to pove the covegece of the algoithm of Fig Theoem 3 Let e q p ^2 ž ### satisfy Assumptios ad 2, ad assume that CÂÒu 8 8O p p ### ad be geeated by the algoithm of Fig The ` b p àúá â à N E a qs µ CÂÓu Let the sequeces whee is as i Eq 9) That is, Poof: Let so that àté p à ` Ú àúá â q ad is a auxiliay fuctio fo p CÂÒu{Ÿ p àté p 0 ±8 à p à!@ a7ä ; 9å q ê ëõìîíï Zñðò ó ê ëõìîíï Zñðò ó CÂÒu{Ÿ P Å{ P P Å{ P We claim that the fuctio ƒ p 0 ±8 ### Clealy, is cotiuous ad opositive = 0

11 û Let ^ a tå{ q We ca uppe boud the chage i  u q p ÜÂÒu à q p CÂÒu à q p Ü u{ÿ à àté @ àté IJ K{¼½ P Å P àté S ó àté p à p o oud æ by2 à à q p CÂÓu à ô àté Å{ ¾À ô àté ^ P Å{ P ¾À S ó ó àté S ó àté ^ àté p à! as follows: 20) 2) 22) àté ü 23) Eqs 20) ad 2) follow fom Eq ) ad Assumptio, espectively Eq 22) uses the fact that, fo ay with > ~ ú, we have s ad fo ~ IJcK{¼½ ~ ¾À IJ K{¼½ ~ S [ ¼½ ~ Œ Ñ ¼½ ~ ¾À ~ ¾ÀQ¾À ~ S [ by Jese s iequality applied S [ to the covex fuctio Eq 23) uses the defiitios of Eq 24) uses ou choice of à ideed, à was chose specifically to miimize Eq 23)) If the fo all Ö,, that is, P Å{ P Å{ p p ### Thus, is a auxiliay fuctio fo ^ àté ad 24) 25) àté, ad The theoem ow follows immediately fom Lemma 2 To apply this theoem to the ExpLoss ad LogLoss poblems, we oly eed to veify that Assumptios ad 2 ae satisfied Statig with Assumptio, fo ExpLoss, we have Fo LogLoss, sû CÂÓu{Ÿ CÂÓuŸ p +d p +dd sû CÂÓu CÂÓu S < `ba ƒ p + tÿ `ba S < S <

12 Ù ÿ u Paametes: hml 9 efihkj p 9 satisfyig Assumptios ad 2 * h qs p E such that ÜÂÒu 0 Iput: Matix * ÑÝi ### Output: 8 8 such that ` Ú q p 0 àúá â CÂÒu{Ÿ ±8 à a7ä ; 9 å Let 8 xâ 0dÙc### Fo æo : ç p à p 0 xÿ ±8 à ç Ö à N 4J Ì àté ÅÆ Ì ç à ÌÌÌÌ ÌÌÌÌ àté tåæ ç! à àté ç# à `Úa ƒ à à à à ç ô àté à if $ Ö TÖ à othewise ç pdate paametes: 8 à x8 à T à q CÂÒu{Ÿ p 0 ±8 = Figue 2: The sequetial-update optimizatio algoithm The fist ad secod equalities use Eqs 4) ad 5), espectively The fial iequality uses all Assumptio 2 holds tivially fo LogLoss sice h qs is bouded Fo ExpLoss, if CÂÓu ÿ the which clealy defies a bouded subset of 9 S [ fo Sequetial algoithms I this sectio, we descibe aothe algoithm fo the same miimizatio poblems descibed i Sectio 4 Howeve, ulike the algoithm of Sectio 5, the oe that we peset ow oly updates the weight of oe featue at a time While the paallel-update algoithm may give faste covegece whe thee ae ot too may featues, the sequetial-update algoithm ca be used whe thee ae a vey lage umbe of featues usig a oacle fo selectig which featue to update ext Fo istace, Adaoost, which is essetially equivalet to the sequetial-update algoithm fo ExpLoss, uses a assumed weak leaig algoithm to select a weak hypothesis, ie, oe of the featues The sequetial algoithm that we peset fo LogLoss ca be used i exactly the same way The algoithm is show i Fig 2 2

13 ƒ Ù õ ö Ù Theoem 4 Give the assumptios of Theoem 3, the algoithm of Fig 2 coveges to optimality i the sese of Theoem 3 Poof: Fo this theoem, we use the auxiliay fuctio 2 õ &' ' ö 4J Å This fuctio is clealy cotiuous ad opositive We have that q CÂÒu p à q Ü u p à àté ¼½<IJ K{¼½!@ àté IJ K à Å ô àté ÅÆ ¾À ƒ àté Å{ Ù S ) à à S ) à à S ) à à à p à ^ Å{ Ù ¾À S ) 2) 27) à 28) S ) [ whee Eq 27) uses the covexity of, ad Eq 29) uses ou choice of à as befoe, we chose à to miimize the boud i Eq 28)) If the 4J Ì tåæ Ì so > ÌÌÌÌ ÌÌÌÌ ÅÆ fo all Ö p p ### Thus, is a auxiliay fuctio fo ad the theoem follows immediately fom Lemma 2 As metioed above, this algoithm is essetially equivalet to Adaoost, specifically, the vesio of Adaoost fist peseted by Feud ad Schapie [] I Adaoost, o each iteatio, a distibutio * à ove the taiig examples is computed ad the weak leae seeks a weak hypothesis with low eo with espect to this distibutio The algoithm ### peseted i this sectio assumes that the space of weak hypotheses cosists of the featues 5 5<, ad that the weak leae ^Ø0Ù always succeeds i selectig the featue with lowest eo o, moe accuately, with eo fathest fom ) Taslatig to ou otatio, the weight * à Þ assiged to example by Adaoost is exactly equal to àté Ø à, ad the weighted eo of the æ -th weak hypothesis is equal to à Ù à Theoem 4 the is the fist poof that Adaoost always coveges to the miimum of the expoetial Â, this theoem also tells loss assumig a idealized ` b p ¹,+ weak leae of the fom above) Note that whe us the exact fom of * à Howeve, we do ot kow what the limitig behavio of * à p=¹ is whe gâ, o do we kow about the limitig behavio of the paametes 8 à p ¹ whethe o ot  ) We have also peseted i this sectio a ew algoithm fo logistic egessio I fact, this algoithm is the same as oe give by uffy ad Helmbold [4] except fo the choice of à I pactical tems, vey little wok would be equied to alte a existig leaig system based o Adaoost p so that it uses logistic loss athe tha expoetial loss the oly diffeece is i the mae i which à is computed fom 8 à Thus, 29) 3

14 - 3 3 Paametes: hml 9 efihkj p 9 satisfyig Assumptios ad 2 * h qs p E such that ÜÂÒu l 9 Iput: Matix *³9 ÑÝi satisfyig the coditio that if we defie -/ - P2,0 * ÞRf P 43 ÅÆ P ß43 the Ö 0«* fo which 7 ### 3 Output: 8 8 such that ` Ú qã p àúá â CÂÒu{Ÿ ±8 à a7ä qs ; 9å Let 8 xâ 0dÙc### Fo æo ç p à xÿ ç Fo Ö2 : p 0 ±8 à 0###^ : ƒ ç 0 à N 4J 90 ;:=< àté!@ ç 2 ô Ö f àté àté 8 àté ç pdate paametes: 8 à x8 à T à àté àté 8 àté àté Õê ëìîíï Zñðò ó Õê ëìîíï Zñðò ó Ù `ba³õ àté àté ö CÂÒu{Ÿ àté àté P Å P P Å{ P p ±8 = Figue 3: A paameteized family of iteative optimizatio algoithms we could easily covet ay system such as SLIPPER [7], oostexte [25] o alteatig tees [5] to use logistic loss We ca eve do this fo systems based o cofidece-ated boostig [24] i which à ad Ö à ae chose togethe o each oud to miimize Eq 2) athe tha a appoximatio of this expessio as used i the algoithm of Fig 2 Note that the poof of Theoem 4 ca easily be modified to pove the covegece of such a algoithm usig the same auxiliay fuctio) 7 A paameteized family of iteative algoithms I pevious sectios, we descibed sepaate paallel-update ad sequetial-update algoithms I this sectio, we descibe a paameteized family of algoithms that icludes the paallel-update algoithm of Sectio 5 as well as a sequetial-update algoithm that is diffeet fom the oe i Sectio This family of algoithms also icludes othe algoithms that may be moe appopiate tha eithe i cetai situatios as we explai below 4

15 0 û P The algoithm, which is show i Fig 3, is simila to the paallel-update algoithm of Fig O each oud, the quatities àté ad àté ae computed as befoe, ad the vecto > à is computed as à was computed i Fig Now, howeve, this vecto > à is ot added diectly to 8 à Istead, aothe vecto 0 à is selected which povides a scalig of the featues This vecto is chose to maximize a measue of pogess while esticted to belog to the set - The allowed fom of these scalig vectos is give by the set -, a paamete of the algoithm; - is the estictio of - to those vectos 0 satisfyig the costait that fo all Þ,!@ P Å Piß0 The paallel-update algoithm of Fig is obtaied by choosig -, Ä ad assumig that > P Å Pi fo all Þ Equivaletly, we ca make o such assumptio, ad choose -, ÿ0ä ÿ c3 ) We ca obtai a sequetial-update algoithm by choosig - compoet equal to ad all othes equal to ), ad assumig that Å{ 0!2 * fo all Þ the becomes ô 8 àté àté Ö2TÖ à whee Ö à N 4J Ì Aothe iteestig case is whe we assume that ÌÌÌ > Å which esues that -/,0 *³9 else àté P7P P 0 ú P P 43 ã to be the set of uit vectos ie, with oe Ö The update àté Ì ÌÌÌ fo all Þ It is the atual to choose The the maximizatio ove -/ ca be solved aalytically givig the update ô 8 àté àté AP P C P P whee A àté àté ü This idea geealizes easily to the case i which > ÅE ad P P P P F fo ay dual oms ~ ad ) A fial case is whe we do ot estict the scalig vectos at all, ie, we choose - 9 I this case, the maximizatio poblem that must be solved to choose each 0 à is a liea pogammig poblem with vaiables ad ˆ costaits We ow pove the covegece of this etie family of algoithms Theoem 5 Give the assumptios of Theoem 3, the algoithm of Fig 3 coveges to optimality i the sese of Theoem 3 Poof: We use the auxiliay fuctio whee ad chage i qs ÜÂÓu q ÜÂÒu 2 4J 90 ƒ p ae as i Theoem 3 This fuctio is cotiuous ad opositive We ca boud the à usig the same techique give i Theoem 3: p à q CÂÒu p à àté IJ K ¼½!@ ô àté Å{ ¾À 5

16 3 Þ 3 ˆ Fially, if2 the Sice fo evey Ö thee exists 0 * 4J 90 ;:=< -!@ ƒ with 3 7 Applyig Lemma 2 completes the theoem 8 àté IJ K ¼½ àté û àté ƒ, this implies!@ àté P Å{ P àté S HKó àté àté 8 àté P Å{ P ¾À S HIóJ ó àté S HIó àté ^ àté p à! àté ü fo all Ö, ie, > Å I this sectio, we show how all of ou esults ca be exteded to the multiclass case ecause of the geeality of the pecedig esults, we will see that o ew algoithms eed be devised ad o ew covegece poofs eed be poved fo this case Rathe, all of the pecedig algoithms ad poofs ca be diectly applied to the multiclass case I the multiclass case, the label set L has cadiality M Each featue is of the fom 5 fc)m NL j 9 I logistic egessio, we use a model MON L P S W X Z [ é O ] S W X Z\[ >QP GR ép ] R O S W X Z\[ ép ]t=w X Z [ é O ] 30) < whee :0;< >!@ A 5 < The loss o a taiig set the is `ba O We tasfom this ito ou famewok as follows: Let S W X4Z\[ ép ]C=W X4Z\[ é O Õ] R WV^ P s WV 3V3/, Þ * L T, The vectos o p ^,, etc that we wok with ae i 9YX That is, they ae M by pais i Let ~ Z deote O ~ ép The covex fuctio e that we use fo this case is eyto which is defied ove the space ~ O ép `ba ~ ép h š o *³9 X P2 ÞOf=Z ~ ~ Z ß `ba ~ Z 3) ˆ -dimesioal ad ae idexed

17 X X X M X The esultig egma distace is Clealy, It ca be show that qs to u tÿ q O ÜÂÒu p + Z ép ] Assumptio ca be veified by otig that q Ü u{ÿ Now let Å Z ép ] é 5 q p 0 & CÂÓu{Ÿ Ô8 ~ ép `ba³õ ~ ép ép ö Z `ba ~ Z! Z S < [\ ép > O S < [\ ép `ba ƒ ~ Z Z # p q Z p ö ¼½ Z S O Z S O S Z ] 5 +d `ba³õ ÜÂÓu tÿ `ba < [\ ép PT < ]\ ¼½ ép PT < ]\ ^! ép ép WV^ &, ad let ^Ø Ä Pluggig i these defiitios gives that is equal to Eq 3) Thus, the algoithms of Sectios 5, ad 7 ca all be used to solve this miimizatio poblem, ad the coespodig covegece poofs ae also diectly applicable Thee ae seveal multiclass vesios of Adaoost AdaoostM2 [] a special case of AdaoostMR [24]), is based o the loss fuctio Z ép ] IJ K WV^ C:0;<= :0;<= Fo this loss, we ca use a simila set up except fo the choice of e We istead use eyto Z ép ] fo o * h 9 X I fact, this is actually the same e used fo biay) Adaoost We have meely chaged the idex set to Thus, as befoe, q CÂÒu ép Z ép ] ad tÿ ~ ép `ba ~ ép p + ép S < ]\ ép Choosig p as we did fo multiclass logistic egessio ad ±Ä q, we have that Ü u{ÿ ±8 is equal to the loss i Eq 33) We ca thus use the pecedig algoithms to solve this multiclass poblem as well I paticula, the sequetial-update algoithm gives AdaoostM2! p 0 32) 33) 7

18 L Ž 2 Þ Ö 2 Þ, f ˆ 3 f ö AdaoostMH [24] is aothe multiclass vesio of Adaoost Fo AdaoostMH, we eplace by the idex set 0###^ ad fo each example Þ V ad label * L, we defie 2 Ž ép $ ^L if V V + if The loss fuctio fo AdaoostMH is We ow let Å Z ép ] é ép 5 WV & this multiclass vesio of Adaoost P ;R IJ K Ž 9 A compaiso to iteative scalig WV! ép :0;< 34) ad use agai the same e as i biay Adaoost with p gä to obtai I this sectio, we descibe the geealized iteative scalig GIS) pocedue of aoch ad Ratcliff [] fo compaiso to ou algoithms We lagely follow the desciptio of GIS give by ege, ella Pieta ad ella Pieta [] fo the multiclass case To make the compaiso as stak as possible, we peset GIS i ou otatio ad pove its covegece usig the methods developed i pevious sectios I doig so, we ae also able to elax oe of the key assumptios taditioally used i studyig GIS We adopt the otatio ad set-up used fo multiclass logistic egessio i Sectio 8 To ou kowledge, thee is o aalog of GIS fo the expoetial loss so we oly coside the case of logistic loss) We also exted this otatio by defiig é O Z so that ép is ow defied fo all V * L Moeove, it ca be veified that ép MON V/P = p p 0 as defied i Eq 30) if xÿ ±8 I GIS, the followig assumptios egadig the featues ae usually made: WV fñ5 WV & ad 2 WV 5 WV & I this sectio, we pove that GIS coveges with the secod coditio eplaced by a milde oe, amely, that 5 WV ùß & Sice, i the multiclass case, a costat ca be added to all featues 5 without chagig the model o loss fuctio, ad sice the featues ca be scaled by ay costat, the two assumptios we coside clealy ca be made to hold without loss of geeality The impoved iteative scalig algoithm of ella Pieta, ella Pieta ad Laffety [2] also equies oly these milde assumptios but is much moe complicated to implemet, equiig a umeical seach such as Newto-Raphso) fo each featue o each iteatio GIS woks much like the paallel-update algoithm of Sectio 5 with e, p ad as defied fo multiclass logistic egessio i Sectio 8 The oly diffeece is i the computatio of the vecto of updates à, fo which GIS equies diect access to the featues 5 Specifically, i GIS, à is defied to be ô àté `ba³õ _ ` p à 8

19 õ X ' ö ` whee _ ` 5 & P GR ép 5 WV! & Clealy, these updates ae quite diffeet fom the updates descibed i this pape withi ou famewok as follows: sig otatio fom Sectios 5 ad 8, we ca efomulate ` ` ép 5 WV & P GR 5 & ép 5 WV^ & P GR _ ép Å Z Z ép ] ép ] é _! 5 & We ca ow pove the covegece of these updates usig the usual auxiliay fuctio method Theoem Let e, optimality i the sese of Theoem 3 Poof: We will show that ad p be as above The the modified GIS algoithm descibed above coveges to 2 ###^!###^ ` ' d u _ `ba _ `!@ ` _ p p ### is a auxilliay fuctio fo the vectos computed by GIS Clealy, òx oegativity popeties of uomalized elative etopy imply that2 _ ` fo all Ö Fom p Eq 35), _ ` if ad oly if implies that the costaits q p =d CÂÒu{Ÿ q E CÂÒu 35) ö 3) is cotiuous, ad the usual with equality if ad oly if Thus, xâ as i the poof of Theoem 3 All that emais to be show is that 37) whee ô `ba³õa_ ` We itoduce the otatio V h2!@ ô 5 = WV^! 9

20 ö ö ö ad the ewite the gai as follows usig Eq 32): q CÂÓu{Ÿ p H q CÂÓu `ba ¼½! é O O ép IJ KQ¼½ `ba h S CZ O Ú] ¼½ é O O ép!@ ô Å Z ép ] é ¾À ¾À S > ócb å Úó ð7d [\fe[ó ¾Àg 38) Pluggig i defiitios, the fist tem of Eq 38) ca be witte as h2!@!@ `ba³õa_ ` ih _ `ba³õ _ ` Next we deive a uppe boud o the secod tem of Eq 38): `ba S ÜZ O Ú] ¼½ é O O ép S > ókb å Úó ðld [\meó ¾À 5 & `ba ¼½ é O S CZ O Ú] S ÜZ O ép P ] ¾À `ba ¼½ S ÜZ ép P ] ¾À P GR ¼½ S CZ ép P ] ¾À P GR ép IJ K ¼½ P GR!@ 5 WV^ ô & ¾À ép 5 WV S Úó ^ & P GR P õ _ ` _ 5 ` R WV õ _ ` `ba Eq 40) follows fom the log boud of the 5 s Eq 43) follows fom ou defiitio of the update P ;R ép 5 WV = j 39) 40) 4) 42) ö 43) 44) Eq 42) uses Eq 25) ad ou assumptio o the fom 20

21 Fially, combiig Eqs 3), 38), 39) ad 44) gives Eq 37) completig the poof It is clea that the diffeeces betwee GIS ad the updates give i this pape stem fom Eq 38), which `ba `ba is deived fom po û Sq ü, with o œh2 o `ba the Þ th tem i the sum This choice of o `ba effectively meas that the log boud is take at a diffeet poit po û S q ü S q po S q I this moe geeal case, the boud is exact at ad theeby vaies the updates 0 Expeimets ; hece, vayig o vaies whee the boud is take, I this sectio, we biefly descibe some expeimets usig sythetic data These expeimets ae pelimiay ad ae oly iteded to suggest the possibility of these algoithms havig pactical value Moe systematic expeimets ae clealy eeded usig both eal-wold ad sythetic data, ad compaig the ew algoithms to othe commoly used pocedues I ou expeimets, we geeated adom data ad classified it usig a vey oisy hypeplae Moe specifically, i the 2-class ª case, we fist geeated a adom hypeplae i 00-dimesioal space epeseted by a vecto *9 chose uifomly at adom fom the uit sphee) We the choseiv^ 000 poits *{9 I the case of eal-valued featues, each poit was omally distibuted #sutcâ I the0!243 case of oolea featues, each poit was chose uifomly at adom fom the oolea hypecube,/ We ext assiged a label to each poit depedig o whethe it fell above o below the a ªž Afte each label was chose, we petubed each poit I the chose hypeplae, ie, u {z v^ case of eal-valued featues, we did this by addig a adom amout w to whee wxsytœcâ Fo oolea featues, we flipped each coodiate of idepedetly with pobability ~} Note that both of these foms of petubatio have the effect of causig the labels of poits ea the sepaatig hypeplae to be moe oisy tha poits that ae fathe fom it The featues wee idetified with coodiates of Fo eal-valued featues, we also coducted a simila ###^ª expeimet ivolvig te classes athe tha two, each chose uifomly at adom fom the ª I this case, we geeated te adom hypeplaes uit sphee, ad classified each poit by Nd 4J O ª O pio to petubig ) Fially, i some of the expeimets, we limited each weight vecto to deped o just 4 of the 00 possible featues I the fist set of expeimets, we tested the algoithms to see how effective they ae at miimizig the logistic loss o the taiig data We a the paallel-update algoithm of Sectio 5 deoted pa i the figues), as well as the sequetial-update algoithm that is a special case of the paameteized family descibed i Sectio 7 deoted seq ) Fially, we a the iteative scalig algoithm descibed i Sectio 9 is ) We did ot u the sequetial-update algoithm of Sectio sice, i pelimiay expeimets, it seemed to cosistetly pefom wose tha the sequetial-update algoithm of Sectio 7) As oted i Sectio 9, GIS equies that all featues be oegative Give featues that do ot satisfy this costait, oe ca subtact a costat ÿ fom each featue 5 without chagig the model i Eq 30); thus, oe ca use a ew set of featues < 7 whee 5 ÿ x5 a ép 5 WV^! & The ew featues defie a idetical model to that of the old featues, because the esult of the chage is that the deomiato ad umeato i Eq 30) ae both multiplied by the same costat, IJ KÇû > A ÿ ü ÿ ) 2

22 few elevat featues may elevat featues 095 taiig loss taiig loss is seq pa ealvalued featues, 2 classes taiig loss taiig loss ealvalued featues, 0 classes taiig loss taiig loss boolea featues, 2 classes Figue 4: The taiig logistic loss o data geeated by a oisy hypeplaes by vaious methods 22

23 few elevat featues may elevat featues test eo test eo log seq exp seq log pa exp pa ealvalued featues, 2 classes test eo 09 test eo ealvalued featues, 0 classes test eo 045 test eo boolea featues, 2 classes Figue 5: The test misclassificatio eo o data geeated by oisy hypeplaes 23

24 whee A slightly less obvious appoach is to choose a featue tasfomatio 5 < ÿ x5 a P < 5 ÿ Like the fome appoach, this causes 5 to be oegative without affectig the model of Eq 30) both demoiato ad umeato of Eq 30) ae ow multiplied by IJcK û > A ÿ ü ) Note that, i eithe case, the costats ÿ o ÿ ) ae of o cosequece duig testig ad so ca be igoed oce taiig is complete I a pelimiay vesio of this pape, we did expeimets usig oly the fome appoach ad foud that GIS pefomed uifomly ad cosideably wose tha ay of the othe algoithms tested Afte the publicatio of that vesio, we tied the latte method of makig the featues oegative ad obtaied much bette pefomace All of the expeimets i the cuet pape, theefoe, use this latte appoach The esults of the fist set of expeimets ae show i Fig 4 Each plot of this figue shows the logistic loss o the taiig set fo each of the thee methods as a fuctio of the umbe of iteatios The loss has bee omalized to be whe 8ÔÂ ) Each plot coespods to a diffeet vaiatio o geeatig the data, as descibed above Whe thee ae oly a small umbe of elevat featues, the sequetial-update algoithms seems to have a clea advatage, but whe thee ae may elevat featues, oe of the methods seems to be best acoss-the-boad Of couse, all methods evetually covege to the same level of loss I the secod expeimet, we tested how effective the ew competitos of Adaoost ae at miimizig the test misclassificatio eo Fo this expeimet, we used the same paallel- ad sequetial-update algoithms deoted pa ad seq ), ad i both cases, we used vaiats based o expoetial loss exp ) ad logistic loss log ) Fig 5 shows a plot of the classificatio eo o a sepaate test set of 2000 examples Whe thee ae few elevat featues, all of the methods ovefit o this data, pehaps because of the high-level of oise With may elevat featues, thee is ot a vey lage diffeece i the pefomace of the expoetial ad logistic vaiats of the algoithms, but the paallel-update vaiats clealy do much bette ealy o; they seem to go ight to the solutio Ackowledgmets May thaks to Mafed Wamuth fo fist teachig us about egma distaces ad fo may commets o a ealie daft Thaks also to Nigel uffy, avid Helmbold, Raj Iye ad Joh Laffety fo helpful discussios ad suggestios Some of this eseach was doe while Yoam Sige was at AT&T Labs Refeeces [] Adam L ege, Stephe A ella Pieta, ad Vicet J ella Pieta A maximum etopy appoach to atual laguage pocessig Computatioal Liguistics, 22):39 7, 99 [2] L M egma The elaxatio method of fidig the commo poit of covex sets ad its applicatio to the solutio of poblems i covex pogammig SSR Computatioal Mathematics ad Mathematical Physics, 7):200 27, 97 [3] Leo eima Acig the edge Techical Repot 48, Statistics epatmet, ivesity of Califoia at ekeley, 997 Appeaed i Poceedigs of the Thiteeth Aual Cofeece o Computatioal Leaig Theoy, 2000 WV^! 24

25 [4] Leo eima Pedictio games ad acig classifies Techical Repot 504, Statistics epatmet, ivesity of Califoia at ekeley, 997 [5] Yai Ceso ad Stavos A Zeios Paallel Optimizatio: Theoy, Algoithms, ad Applicatios Oxfod ivesity Pess, 997 [] Nicolò Cesa-iachi, Ades Kogh, ad Mafed K Wamuth ouds o appoximate steepest descet fo likelihood maximizatio i expoetial families IEEE Tasactios o Ifomatio Theoy, 404):25 220, July 994 [7] William W Cohe ad Yoam Sige A simple, fast, ad effective ule leae I Poceedigs of the Sixteeth Natioal Cofeece o Atificial Itelligece, 999 [8] I Csiszá I-divegece geomety of pobability distibutios ad miimizatio poblems The Aals of Pobability, 3):4 58, 975 [9] I Csiszá Saov popety, geealized I-pojectio ad a coditioal limit theoem Aals of Pobability, 2:78 793, 984 [0] I Csiszá Maxet, mathematics, ad ifomatio theoy I Poceedigs of the Fifteeth Iteatioal Wokshop o Maximum Etopy ad ayesia Methods, pages 35 50, 995 [] J N aoch ad Ratcliff Geealized iteative scalig fo log-liea models The Aals of Mathematical Statistics, 435): , 972 [2] Stephe ella Pieta, Vicet ella Pieta, ad Joh Laffety Iducig featues of adom fields IEEE Tasactios Patte Aalysis ad Machie Itelligece, 94): 3, Apil 997 [3] Calos omigo ad Osamu Wataabe Scalig up a boostig-based leae via adaptive samplig I Poceedigs of the Fouth Pacific-Asia Cofeece o Kowledge iscovey ad ata Miig, 2000 [4] Nigel uffy ad avid Helmbold Potetial boostes I Advaces i Neual Ifomatio Pocessig Systems, 999 [5] Yoav Feud ad Llew Maso The alteatig decisio tee leaig algoithm I Machie Leaig: Poceedigs of the Sixteeth Iteatioal Cofeece, pages 24 33, 999 [] Yoav Feud ad Robet E Schapie A decisio-theoetic geealizatio of o-lie leaig ad a applicatio to boostig Joual of Compute ad System Scieces, 55):9 39, August 997 [7] Jeome Fiedma, Tevo Hastie, ad Robet Tibshiai Additive logistic egessio: a statistical view of boostig The Aals of Statistics, to appea [8] Klaus- Höffge ad Has- Simo Robust taiability of sigle euos I Poceedigs of the Fifth Aual ACM Wokshop o Computatioal Leaig Theoy, pages , July 992 [9] Jyki Kivie ad Mafed K Wamuth oostig as etopy pojectio I Poceedigs of the Twelfth Aual Cofeece o Computatioal Leaig Theoy, pages 34 44, 999 [20] Joh Laffety Additive models, boostig ad ifeece fo geealized divegeces I Poceedigs of the Twelfth Aual Cofeece o Computatioal Leaig Theoy, pages 25 33, 999 [2] Joh Laffety, Stephe ella Pieta, ad Vicet ella Pieta Statistical leaig algoithms based o egma distaces I Poceedigs of the Caadia Wokshop o Ifomatio Theoy, 997 [22] Llew Maso, Joatha axte, Pete atlett, ad Macus Fea Fuctioal gadiet techiques fo combiig hypotheses I Advaces i Lage Magi Classifies MIT Pess, 999 [23] G Rätsch, T Ooda, ad K-R Mülle Soft magis fo Adaoost Machie Leaig, to appea [24] Robet E Schapie ad Yoam Sige Impoved boostig algoithms usig cofidece-ated pedictios Machie Leaig, 373):297 33, ecembe 999 [25] Robet E Schapie ad Yoam Sige oostexte: A boostig-based system fo text categoizatio Machie Leaig, 392/3):35 8, May/Jue

26 [2] F Topsoe Ifomatio theoetical optimizatio techiques Kybeetika, 5:7 7, 979 [27] Osamu Wataabe Fom computatioal leaig theoy to discovey sciece I Poceedigs of the 2th Iteatioal Colloquium o Automata, Laguages ad Pogammig, pages 34 48, 999 2