Supervised Rank Aggregation

Transcription

1 Sessio: Search Quaity ad Precisio Supervised Rak Aggregatio Yu-Tig Liu,*, Tie-Ya Liu, Tao Qi,3*, Zhi-Mig Ma 4, ad Hag Li Microsoft Research Asia 4F, Sigma Ceter, No. 49, Zhichu Road, Haidia District, Beijig, 00080, Chia {tyiu, Schoo of Sciece, Beijig Jiaotog Uiversity, Beijig 00044, Chia 3 Departmet of Eectroic Egieerig, Tsighua Uiversity, Beijig 00084, Chia 4 Academy of Math ad Systems Sciece, Chiese Academy of Sciece, Beijig 00080, Chia ABSTRACT This paper is cocered with rak aggregatio, the task of combiig the rakig resuts of idividua rakers at meta-search. Previousy, rak aggregatio was performed maiy by meas of usupervised earig. To further ehace rakig accuracies, we propose empoyig supervised earig to perform the task, usig abeed data. We refer to the approach as Supervised Rak Aggregatio. We set up a geera framework for coductig Supervised Rak Aggregatio, i which earig is formaized a optimizatio which miimizes disagreemets betwee rakig resuts ad the abeed data. As case study, we focus o Markov Chai based rak aggregatio i this paper. The optimizatio for Markov Chai based methods is ot a covex optimizatio probem, however, ad thus is hard to sove. We prove that we ca trasform the optimizatio probem ito that of Semidefiite Programmig ad sove it efficiety. Experimeta resuts o meta-searches show that Supervised Rak Aggregatio ca sigificaty outperform existig usupervised methods. Categories ad Subject Descriptors H.3.3 [Iformatio Storage ad Retrieva]: Iformatio Search ad Retrieva Retrieva modes. H.3.4 [Iformatio Systems Appicatio]: Systems ad Software- performace evauatio (efficiecy ad effectiveess). Geera Terms Agorithms, Experimetatio, Theory Keywords Rak aggregatio, supervised earig, Markov Chai, Semidefiite programmig. INTRODUCTION Rak aggregatio is to combie rakig resuts of etities from mutipe rakig fuctios i order to geerate a better oe. The idividua rakig fuctios are referred to as base rakers, or simpy rakers, hereafter. Rak aggregatio ca be cassified ito two categories []. I the first category, the etities i idividua rakig ists are assiged scores ad the rak aggregatio fuctio is assumed to use the scores (deoted as score-based aggregatio) [][8][8]. I the secod category, oy the orders of the etities i idividua rakig ists are used by the aggregatio fuctio (deoted as * This work was coducted whe the first ad the third authors were iters at Microsoft Research Asia Copyright is hed by the Iteratioa Word Wide Web Coferece Committee (IW3C). Distributio of these papers is imited to cassroom use, ad persoa use by others. WWW 007, May 8-, 007, Baff, Aberta, Caada. ACM /07/0005. order-based aggregatio). We focus o order-based aggregatio i this paper. Order-based aggregatio is empoyed at meta-search, for exampe, i which oy order (rak) iformatio from idividua search egies is avaiabe. Previousy order-based aggregatio was maiy addressed with the usupervised earig approach, i the sese that o traiig data is utiized; methods ike Borda Cout [][7][7], media rak aggregatio [9], geetic agorithm [4], fuzzy ogic based rak aggregatio [], Markov Chai based rak aggregatio [7] ad so o were proposed. Oe exceptio is Borda Fuse [] which aso makes use of traiig data. However, it is differet from the supervised earig method we propose i this paper. We argue that i order to improve the accuracy of rak aggregatio, it is better to empoy a supervised earig approach i which we trai a order-based aggregatio fuctio withi a optimizatio framework usig abeed data. At meta search, for exampe, abeed data ca be documets ad their reevacies to give queries. The key factors, thus, are (a) to assume that oy order iformatio from idividua rakers is avaiabe, (b) to use abeed data, ad (c) to trai the aggregatio fuctio withi a optimizatio framework. I this paper, we refer to the approach as Supervised Rak Aggregatio. There are severa advatages for takig the supervised earig approach. First, we ca everage the use of iformatio existig i abeed traiig data. Secod, we ca appy existig optimizatio techiques to the probem. Third, it becomes easier to make domai or user adaptatio. Certaiy, it aso has a disadvatage, that is, abeed data is eeded ad creatig such data ca be costy. This is, however, a shortcomig for ay supervised earig method ad we ca eave it as future research topic. I this paper, we first give a geera framework for coductig Supervised Rak Aggregatio. We show that we ca defie supervised earig methods correspodig to the existig usupervised methods, such as Borda Cout ad Markov Chai based methods by expoitig the framework. The we maiy ivestigate the supervised versios of Markov Chai based methods i this paper, because previous work shows that their usupervised couterparts are superior [4]. It turs out, however, that the optimizatio probems for the Markov Chai based methods are hard, because they are ot covex optimizatio probems. We are abe to deveop a method for the optimizatio of oe Markov Chai based method, caed Supervised MC. Specificay, we prove that we ca trasform the optimizatio probem ito that of Semidefiite Programmig. As a resut, we ca efficiety sove the issue. (We pa to appy the same techique to the other Markov Chai methods i the future.) Experimeta resuts o meta-searches show that Supervised Rak Aggregatio (i.e., Supervised MC ) ca achieve better performaces tha existig methods. 48

2 The rest of this paper is orgaized as foows. I Sectio, we itroduce reated work. I Sectio 3, we propose a geera framework ad specific methods for Supervised Rak Aggregatio. I Sectio 4, we propose a optimizatio agorithm for the method of Supervised MC. Experimeta resuts are reported i Sectio 5. Cocusios ad future work are give i the ast sectio.. RELATED WORK The origi of research o rak aggregatio ca be traced back to the eighteeth cetury, whe it was studied i socia choice theory ad appied ito poitica eectios [5]. I recet years, rak aggregatio gets spotight agai i may ew appicatios, such as geome database costructio [6], documet fiterig [3], database middeware costructio [0], spam webpage detectio [7], meta-search [][7][7][4], word associatio fidig [7], mutipe search [], ad simiarity search [9]. There are two types of rak aggregatio: score-based ad orderbased. I the former the aggregatio fuctio takes score iformatio from the idividua base rakers as iput, whie i the atter it oy utiizes order iformatio. Order-based aggregatio fits we with meta-search, as i meta-search oy order iformatio from base rakers is avaiabe; this is aso the mai focus of the research i this paper. Existig methods for order-based aggregatio icudes, for exampe, Borda Cout [][7][7], media rak aggregatio [9], geetic agorithm [4], fuzzy ogic based rak aggregatio method [] ad Markov Chai based rak aggregatio [7]. Borda Cout raks etities based o their positios i the rakig ists. For exampe, the etities are sorted accordig to the umber of etities that are raked beow them i a the rakig ists. Media rak aggregatio sorts the etities based o the medias of their raks i a the rakig ists. Markov Chai based rak aggregatio assumes that there exists a Markov Chai o the etities ad the order reatios betwee etities i the rakig ists represets the trasitios i Markov Chai. The statioary distributio of the Markov Chai is utiized to rak the etities. Dwork et a [7] proposed four methods (deoted as MC, MC, MC 3, ad MC 4 ) to costruct the trasitio probabiity matrix of the Markov Chai. The usupervised methods described above impicity coduct majority votig i their fia rakig decisios. That is to say, these methods treat a the rakig ists equay ad give high raks to those etities raked high by most of the rakers. This assumptio may ot hod i practice, however. For exampe, i meta-search, rakig ists are geerated by differet search egies with differet capacities ad accuracies. It is ot reasoabe to treat the resuts of the search egies equay. To dea with the probem, Asam et a [] proposed Borda Fuse, which ca be viewed as weighted Borda Cout for meta-search. Specificay, differet rakers are assiged differet weights, whie the weights are traied separatey by usig abeed traiig data. For exampe, the weights ca be cacuated based o the MAP (Mea Average Precisio) scores of the base rakers. Experimeta resuts show that Borda Fuse ideed improves upo Borda Cout. The probem with Borda Fuse is that the weights of the rakig ist are cacuated idepedety ad by usig heuristics. It is aso ot cear whether the same idea ca be appied to other methods. We ote that order-based rak aggregatio i meta-search is simiar to reevace rakig i documet retrieva, but there are Sessio: Search Quaity ad Precisio some cear differeces. Therefore, the methods proposed for reevace rakig may ot be directy appicabe to order-based rak aggregatio. I reevace rakig, a typica approach is to empoy a iear combiatio mode of the features to rak documets. Oe ca aso empoy a supervised earig method to trai the mode. Each feature ca be viewed as a raker ad the fia rakig mode ca be viewed as a aggregatio fuctio. However, this fia rakig mode is more cose to that of scorebased aggregatio, ot that of order-based aggregatio. How to appy a score-based method to order-based aggregatio is sti a ope probem, ad is out of the scope of this paper. 3. SUPERVISED RANK AGGREGATION I this sectio, we first itroduce a geera optimizatio framework for order-based rak aggregatio. We the defie Supervised Rak Aggregatio methods withi the framework. We first give some defiitios ad otatios. Give a set of etities S, et V be a subset of S ad assume that there is a tota order amog the etities i V. τ is caed a rakig ist with respect to S, if τ is a ist of the etities i V maitaiig the same tota order reatio, i.e.,τ = d,, d m, if d > > d m, d i V,i =,, m, where > deotes the reatio ad m deotes the size of V. If V equas S, τ is caed a fu ist, otherwise, it is caed a partia ist. A specia case of partia ist is a top-t ist, for which the first t th etities are ordered i the ist. 3. Optimizatio Framework The goa of rak aggregatio is to assig a rea-vaued score to each of the etities by aggregatig a the rakig ists give by the base rakers, ad the sort the etities accordig to their scores. Without oss of geeraity, hereafter we assume that it is i the descedig order. Let τ,, τ deote the rakig ists with respect to S ad deotes the umber of etities i S. We defie the aggregatio fuctio as Ψ: τ,, τ x, where x deotes the fia score vector of a etities. That is, if x = Ψ τ,, τ, the a the etities are raked by the scores i x. For exampe i Borda Cout, x is caed Borda score, which is cacuated as, x = Ψ τ,, τ = k= x (k) (3..) where x (k) (k) x T, x (k) i i = # j i >τk j, ad i > τk j i=,, meas that etity i is raked higher tha etity j i rakig ist τ k. We assume that the aggregatio fuctio Ψ is parameterized by a parameter vector α. I a supervised earig approach to rak aggregatio, we try to ear the optima vaues of the parameters by usig abeed traiig data. Typicay traiig data may icude groud truth idicatig pairwise prefereces of which etities shoud be raked higher tha the others. I the earig, we actuay maage to fid the aggregatio fuctio that miimizes the disagreemets betwee the groud truth ad the output of the aggregatio fuctio. We represet the agreemet betwee the output ist of a aggregatio fuctio ad the groud truth by usig a iequaity Hx < 0 where x deotes the output of the fuctio ad H deotes a matrix represetig the pairwise preferece reatioship betwee etities. For exampe, suppose that the scores produced by the aggregatio fuctio are x = x, x, x 3, x 4 T, ad the groud truth idicates 48

3 that etity shoud be raked higher tha etity, ad etity 4 shoud be raked higher tha etity 3. The, the iequaity becomes: Hx < 0, where H = By usig such a matrix, we ca brig ay form of groud truth ito our framework, ad do ot eed assume a tota order existig over a the etities i the traiig set. There is o guaratee that there exists a parameter vector α that satisfies a the pairwise costraits i the groud truth. That is, disagreemets may exist. We itroduce sack variabe t to represet the differeces (errors), To reduce traiig errors is equivaet to miimize the orm of t. Thus we ca formaize Supervised Rak Aggregatio as the foowig optimizatio probem. mi x,α,t t T t s. t. x = Ψ τ,, τ ; α α C (3..) where α deotes the parameter vector ad C deotes a feasibe regio for α. The dimesio of matrix H equas the umber of pairs idicatig pairwise prefereces i the traiig data. The objective t T t actuay deotes the empirica oss i the traiig data. Whe empirica oss is 0, the aggregatio fuctio Ψ satisfies a the pairwise costraits. With differet ways of istatiatig ad optimizig the aggregatio fuctio, we come to differet methods for rak aggregatio. 3. Methods We show that we ca defie Supervised Rak Aggregatio methods withi the framework. I this paper we oy cosider the case i which the aggregatio fuctio is defied as a iear mode of base rakers. Eve the mode is simpe; it is powerfu eough for accompishig the tasks i this paper. () Borda Fuse May rak aggregatio methods are i fact based o majority votig. Borda Cout [][7][7] is such a method ad the major assumptio withi it is that a the base rakers are equay importat. As discussed above, it is more reasoabe to give differet weights to differet rakers. I other words, we ca cosider usig Borda Fuse x = Ψ τ,, τ = k= α k x (k) Note that Borda Fuse cotais Borda Cout as its specia case. With the optimizatio framework i (3..), we ca defie Supervised Borda Fuse. Specificay we formaize it as the foowig optimizatio probem: mi x,α,t t T t k= α k x (k) k= α k s. t. x = =, α k 0, k =,, where x (k) is the same as that i (3..). Note that the parameter vector is comprised of weights of rakig ists ad is to be optimized as we. () Markov Chai based methods May other rak aggregatio methods are based o Markov Chai. It is advatageous to empoy the Markov Chai mode i rak aggregatio, particuary whe the base rakers oy output partia ists [8]. Experimeta resuts show that the Markov Chai based methods outperform other methods [4]. That is why we focus o Markov Chai based approach i this paper. Dwork et a [7] proposed four Markov Chai based modes for rak aggregatio, referred to as MC, MC, MC 3, ad MC 4. The four modes correspod to four differet heuristic rues for costructig the trasitio probabiity matrix i Markov Chai. Let us take MC as exampe. The trasitios i Markov Chai are defied as foows. If the curret state is i, the we first seect a rakig ist τ k uiformy radomy from the rakig ists τ,, τ that cotai state i, the seect state j uiformy radomy from the set of states that are raked ot ower tha state i i τ k, ad defie j as the ext state. For a fu ist or top-t ist, it is ot difficut to verify that the trasitio matrix is arithmetic mea of trasitio probabiity matrices produced from idividua rakig ists, referred to as base-trasitio matrices. Let P k p deote the ij kth base trasitio matrix produced by rakig ist τ k, i which each k eemet p ij correspods to the coditioa probabiity of state j give state i i rakig ist τ k. The fia trasitio matrix P is defied as P = where m = # j j > τk i or j = i. (k) k= P k p ij k = m, j > τ k i or j = i 0, otherwise (3..) The score vector x ca the be computed by sovig x = P T x, with costraits =, x i > 0, i =,,. i= x i I Supervised MC we assig weightig coefficiets to the base matrices P k : P = k= α k P k Formay, Supervised MC is defied as foows. mi x,α,t t T t k= α k s. t. x = P T k x i= x i =, x i > 0, i =,, k= α k =, α k 0, k =,, (3..) Simiary, we ca costruct the supervised versios of MC, MC 3, ad MC 4. The oy differeces ie i the structures of the trasitio probabiity matrices. a) Supervised MC : The trasitio matrix of MC ca be writte as P = diag where Q q ij = q ij Sessio: Search Quaity ad Precisio j = q j k= Q k,, j = q j (k), Q k q ij (k), j > τk i or j = i =. 0, otherwise Q with We ca derive Supervised MC by assigig weightig coefficiets to Q, ad obtai the foowig optimizatio probem. 483

4 mi x,α,t t T t s. t. x = i= x i k= α k b) Supervised MC 3 : k= α k Q k T diag j = q j =, x i > 0, i =,, =, α k 0, k =,,,, j = q j The formuatio of MC 3 is simiar to that of MC, except the defiitio of p ij k :, j > τ k i k p ij = m, j = i, ad m = # j j > τk i. 0, otherwise Therefore, we ca defie Supervised MC 3 i a simiar way as we defie Supervised MC. c) Supervised MC 4 : MC 4 is simiar to MC, except that the foowig two facts differ. (i) (ii) The defiitio of Q: Q q ij = (k), j > τk i with q ij = 0, otherwise. The defiitio of P: P p ij, with x k= Q k,, q ij > k p ij = m, j = i, ad m = # j q ij >. 0, otherwise Therefore, we ca obtai Supervised MC 4, simiar to Supervised MC. I summary, with the use of the optimizatio framework, we ca itroduce ew supervised aggregatio methods, correspodig to most of the existig usupervised rak aggregatio methods. The key factor is that weights are assiged to the rakig ists ad they are aso traied withi the optimizatio framework. The questio ext is how to coduct the optimizatios. For some forms of fuctio Ψ i (3..), the optimizatio is hard to sove, such as those i the Markov chai based methods. We kow of o existig optimizatio techiques which ca be straightforwardy appied, because they are ot covex optimizatio probems. I our work we are abe to fid a optimizatio soutio for Supervised MC o the basis of Semidefiite Programmig (SDP), as wi be expaied beow. 4. AN OPTIMIZATION SOLUTION I this sectio, we describe our soutio to the optimizatio probem for Supervised MC as i (3..). We thik that simiar techiques ca aso be appied to other Markov Chai based methods, but eave it as future work. Our method for Supervised MC cosists of three steps: ) We modify the objective ad costraits i (3..) to make the feasibe regio covex. ) We further trasform the optimizatio probem ito a quadratic optimizatio probem by empoyig the boud optimizatio techique. 3) Fiay, we trasform the quadratic optimizatio probem ito a Semidefiite Programmig probem. Let us eaborate o the three steps i more detais. Theoretica justificatios of the trasformatios are give i a emma ad a propositio. The first costrait i (3..) represets a eigevector probem. Oe ca easiy verify that the feasibe regio of the optimizatio probem is ot covex. I geera such a probem is hard to sove. We reformuate the origia optimizatio probem by puttig the first costrait ito the objective fuctio: mi x,α,t t T t + α k P T k= k x x s. t. i= x i =, x i > 0, i =,, k= α k =, α k 0, k =,, where deote the -orm of a vector. (4.) The, the feasibe regio becomes covex ad the objective fuctio becomes oe cosistig of two parts. The first part t T t correspods to traiig errors, ad the secod part α k P T k= k x x correspods to a approximatio of the statioary distributio. The secod part of the objective fuctio is ot covex. We try to miimize a differetiabe ad covex upper boud of it. Lemma gives the upper boud usig the properties of -orm. Lemma : Let Ξ = ξ i T i=,, = Ξ α T Ax, where A = Proof: See Appedix. () p α kp T k= k x x, we have () p () p The optimizatio probem the becomes () p mi x,α,t t T t + α T Ax s. t. i= x i =, x i > 0, i =,, k= α k =, α k 0, k =,,. (4.) By defiig β = (α,, α, x,, x, t,, t m ) T, where m is the umber of rows i matrix H, ad omittig the costat i the objective fuctio which is irreevat to the optimizatio, probem (4.) becomes with H 0 = Sessio: Search Quaity ad Precisio mi β β T H 0 β s. t. H β 0 H β = e H 3 β < 0 0 A 0 A T I R (++m ) (++m ) (4.3) H = I I m R +m ++m (4.4) H = e T e T 0 R ++m H 3 = 0 I 0 +m ++m R 0 H I m where I i is idetity matrix of size i, ad e i is vector with size i i which a the eemets are oe. The optimizatio i (4.3) is a optimizatio probem with quadratic objective fuctio ad iear costraits. The remaiig issue is that the Hessia matrix H 0 is ot positive defiite ad thus the objective fuctio is ot covex. I this situatio, if we empoy a method ike Gradiet Decet, the soutio wi be 484

5 sesitive to the iitia vaues, ad wi ikey to become ocay optima. To cope with it, we further trasform the optimizatio probem ito a Semidefiite Programmig (SDP), with the theoretica support from Propositio. Propositio : Optimizatio probem (4.3) is equivaet to the foowig Semidefiite Programmig probem, max λ,γ γ s. t. λ 0 H 0 + λ 0 D UT U ΛT e λ 0 γ 0 Where U = Λ T H + Λ T H + Λ 3 T H 3, ad λ = (λ 0, Λ T, Λ T, Λ 3 T ) T. Proof: See Appedix. (4.5) Fiay we ca sove the optimizatio probem usig the techiques of SDP, for exampe, the iterior-poit method SDPA [30] proposed i []. Our Supervised MC agorithm ca be summarized as foows. Supervised MC : Iput: rakig ists τ,, τ Output: weightig parameter α Agorithm: a) Costruct base trasitio matrices P,, P accordig to equatio (3..). b) Create matrix A as show i Lemma. c) Create matrices H 0, H, H, H 3 as show i equatio (4.4). d) Costruct matrix U as show i Propositio. e) Ca SDP too [30] to sove probem (4.5) ad get soutio λ. f) Compute β by equatio (8..3). g) Output the first eemets of β as parameter α. 5. EXPERIMENTS I this sectio, we report the experimeta resuts o meta-search usig our method based o Supervised Rak Aggregatio ad existig methods. Our first experimet was coducted with TREC dataset, ad the secod was with data from rea web search egies. 5. TREC Data TREC datasets were used i may previous works o rak aggregatio [][8][9][0][4], i which heuristic modes were used as base rakers. This motivated us to coduct our experimets with TREC dataset as we. We seected the OHSUMED dataset used i the fiterig track of TREC 000. The OHSUMED dataset is a coectio of 348,566 documets ad 06 queries. The groud truth is provided by the TREC committee with three eves of reevace judgmets: defiitey reevat, possiby reevat, ad ot reevat to the query. Based o these judgmets, we ca costruct pairwise costraits for the traiig of Supervised MC. SDP is a hot research fied i recet years [9], ad may fast iterative agorithms have bee deveoped [6][][3][30]. Sessio: Search Quaity ad Precisio I our experimet, we used 30 rakig modes (features) [] as base rakers. These icude term frequecy, iverse documet frequecy, documet egth, BM5 score [5], ad their combiatios. Tabe. Resuts of differet methods for meta-search with OHSUMED data Supervised Borda- Borda MC MC MC MC 3 MC 4 Cout Fuse P@ P@ P@ P@ P@ P@ P@ P@ P@ P@ MAP Tabe. Resuts of differet methods for meta-search with OHSUMED data Supervised Borda- Borda MC MC MC MC 3 MC 4 Cout Fuse N@ N@ N@ N@ N@ N@ N@ N@ N@ N@ Next, we coducted rak aggregatio usig our method. For compariso, we aso impemeted ad tested other rak aggregatio methods, icudig MC, MC, MC 3, MC 4, Borda Cout, ad Borda Fuse. The experimets were performed through 4-fod cross vaidatio. We radomy spit the query set ito four subsets, used the first two of them for traiig, the third for vaidatio, ad the fourth for testig, ad rotated this process four times to create four data sets. The we took the average performace over the four trias as the fia resut for each method. We used three measures i our experimets for rakig accuracy evauatios: Precisio [3], Mea Average Precisio (MAP) [3] ad Normaized Discout Cumuative Gai (NDCG) [4][5]. Whe evauatig the performaces i terms of precisio, we regarded both defiitey reevat ad possibe reevat as positive, ad ot reevat as egative. Tabe shows the resuts i terms of precisio at (P@) ad MAP, ad Tabe shows the resuts i terms of DNCG at (N@) for a the methods. From the resuts, we ca see that Supervised MC outperforms a the other methods, suggestig that it is better to empoy Supervised Rak Aggregatio proposed i this paper. 485

6 Sessio: Search Quaity ad Precisio 5. Web Search Data We aso tried to appy the rak aggregatio methods directy to meta-search o the web. 5.. Experimeta Resuts We radomy samped 500 queries from the query og of a commercia search egie, as query set. Tabe 3 shows some exampe queries. Tabe 3. Sampe queries used i meta-search Queries Atavista, Astroomy Picture of the day, BBC, cadiac, daiy atio, deta deta, famiy guy, fox theater, Googe, group heath, habitat for humaity, hotmai, Image Etertaimet, imdb, jacksovie ews, jetbue, kofax, aredo morig times, iberty uiversity, michae Jorda, Microsoft, atioa zoo, NCAA footba, ohio departmet of educatio, phiips, prime outets, souther baptist covetio, Superbow, tacoma ews tribue, texas departmet of pubic safety, Tuesday Morig, uca, uiversity of Teessee, veetia, etc. Tabe 4. Resuts of differet methods for meta-search with data from web search egies Tabe 5. Resuts of differet methods for meta-search with OHSUMED data Supervised Borda- Borda MC MC MC MC 3 MC 4 Cout Fuse N@ N@ N@ N@ N@ N@ N@ N@ N@ N@ Discussios We ivestigated why our proposed supervised method (Supervised MC ) outperforms the baseie methods. Figure shows MAP ad the weight to each search egie assiged by our method i the first tria of the cross-vaidatio experimet. (The resuts from the other trias have the same tedecies). Supervised Borda- Borda MC MC MC MC 3 MC 4 Cout Fuse P@ P@ P@ P@ P@ P@ P@ P@ P@ P@ MAP MAP Weight SE SE SE3 SE4 SE5 SE6 Figure. MAP ad weights of search egies Next, we submitted the queries to six commercia web search egies, ad coected the top-00 rakig ists of the queries retured by the search egies. We combied the resuts together ad eimiated the dupicate pages. O average there were 36 uique pages per query. The overap amog the rakig ists of the search egies was sma: there were o average 4 pages per query occurrig i a the rakig ists. The we asked huma aotators to make reevace judgmets o the pages. The reevace judgmets were biary: reevat or irreevat. Three aotators made judgmets, ad majority votig was fiay coducted o the resuts. We the coducted meta-search o the data through 4-fod cross vaidatio (i the same way as i Sectio 5.. We appied our proposed method, ad used MC, MC, MC 3, MC 4, Borda Cout, ad Borda Fuse as baseies. Tabe 4 shows the resuts i terms of P@ ad MAP. From the resuts, we ca see that our proposed method achieves the best resuts i terms of both MAP ad P@. Agai this verifies the effectiveess of our proposed method for rak aggregatio. Tabe 5 shows the experimet resuts i terms of NDCG@. From Figure, we have the foowig observatios. (a) The weights of search egies are differet from each other. This vaidates the correctess of our assumptio that rakers shoud have differet weights. (b) The weights of search egies do ot ecessariy correate with their MAP vaues. Athough the fourth search egie achieves the best MAP ad obtais the argest weight at the same time, for the other egies, MAP ad weight do ot correate. For exampe, the first search egie has a higher MAP tha the secod, but it has much smaer weight tha the secod. Our expaatio to this is as foows. The weights of search egies ot oy deped o their performaces, but aso deped o the correatios amog search egies. If a search egie highy correates to the others, its weight (ifuece) wi be reduced withi the geera optimizatio framework. To verify the correctess of this expaatio, we cacuate the correatio coefficiet betwee each pair of the six egies usig the foowig formua, ad preset the resuts i Tabe 6. (Note that the correatio is symmetric.) 486

7 # query cor SEi, SEj = # u,v u> SE i v ad u> SE j v or (u < SE i v ad u< SE j v) query # u,v u,v SEi ad (u,v SEj ) where SEi deotes the i-th search egie, u > SEi v meas that documet u is raked higher tha v by SEi for a give query, ad u, v SEi meas that documets u ad v are retured by search egie SEi. Tabe 6. Correatio amog search egies SE SE SE3 SE4 SE5 SE6 SE SE SE SE SE SE6 From Tabe 6, we ca see that the first search egie highy correates to the forth ad the sixth search egies, ad therefore its weight is suppressed by the arge weights of the two egies. I cotrast, the secod search egie oy weaky correates to the other egies, ad thus it retais a arge weight. The observatio ca aso give expaatio to other resuts i the experimets. From Tabe 5, oe may see a iterestig pheomeo. Borda Fuse, as a supervised method, performs eve worse tha the usupervised methods. As expaied, Borda Fuse assumes that the weight of each base raker oy depeds o its accuracy, ad it egects the correatio amog base rakers. It seems that this is ot appropriate ayway. Therefore, it appears better to perform rak aggregatio usig a optimizatio framework as we do. 6. CONCLUSIONS I this paper, we have proposed a ew approach to rak aggregatio: Supervised Rak Aggregatio. Our method is maiy desiged for meta-search ad is uique i that (a) takes order iformatio from base rakers, (b) it makes use of abeed traiig data, ad (c) it trais the fia rakig fuctio withi a sige optimizatio framework. We have set up a geera framework for empoyig the approach. Specificay, we have formaized the earig probem as that of optimizatio. We propose a efficiet agorithm to sove the optimizatio for oe of the typica rak aggregatio settigs, amey the Markov chai based method. We have compared the performaces of our proposed method with those of existig methods o meta-search. The resuts show that the proposed method ca outperform the existig methods. The cotributios of this paper icude ) proposa o empoyig the supervised earig approach for rak aggregatio; ) formuatio of the supervised earig approach as a optimizatio probem; 3) deveopmet of a optimizatio agorithm form the Markov Chai based earig method; ad 4) empirica verificatio of the effectiveess of the proposed approach. As future work, we pa to appy the techiques used i this paper to other supervised earig methods, ad to appy the methods to other appicatios such as simiarity search ad geome iformatics. ACKNOWLEDGEMENTS Sessio: Search Quaity ad Precisio The authors woud ike to thak Wei-Yig Ma at MSRA for his suggestios ad commets o this work. They are aso gratefu to Shisheg Li at USTC for his heps i the experimets. The authors woud aso ike to thak the aoymous reviewers for their vauabe commets o the paper. REFERENCES [] Ahmad N. ad Beg M. M. S. Fuzzy Logic Based Rak Aggregatio Methods for the Word Wide Web, I Proceedigs of the Iteratioa Coferece o Artificia Iteigece i Egieerig ad Techoogy, Maaysia, 00, [] Asam, J. A. ad Motague, M. Modes for Metasearch. I Proceedigs of the 4th Aua Iteratioa ACM SIGIR Coferece o Research ad Deveopmet i Iformatio Retrieva. ACM Press, New York, 00, [3] Baeza-Yates, R. ad Ribeiro-Neto, B. Moder Iformatio Retrieva. Addiso Wesey, 999. [4] Beg, M. M. S. Parae Rak Aggregatio for the Word Wide Web. Word Wide Web. Kuwer Academic Pubishers, vo 6, issue, 5-. March 004. [5] Borda, J. C. Mémoire sur es éectios au scruti. Histoire de Acad emie Royae des Scieces, 78 [6] Boyd, S. ad Vedeberghe, L. Covex Optimizatio. Cambridge, U. K. Cambridge Uiv. Press 003. [7] Dwork, C., Kumar, R., Naor, M., ad Sivakumar, D. Rak Aggregatio Methods for the Web. I Proceedigs of the 0th Iteratioa Word Wide Web Coferece. 00, [8] Dwork, C., Kumar, R., Naor, M., ad Sivakumar, D. Rak Aggregatio revisited. 00. Mauscript. [9] Fagi, R., Kumar, R., ad Sivakumar, D. Efficiet Simiarity Search ad Cassificatio via Rak Aggregatio. I Proceedigs of the 003 ACM SIGMOD Iteratioa Coferece o Maagemet of Data. Sa Diego, 003, [0] Fagi, R., Lotem, A., ad Naor, M. Optima Aggregatio Agorithm for Middeware. I Proceedigs of the Twetieth ACM SIGMOD-SIGACT-SIGART Symposium o Pricipes of Database Systems. Sata Barbara, Caiforia, Uited States, 00, 0-3. [] Fox, E. A. ad Shaw, J. A. Combiatio of Mutipe Searches. I Proceedigs of the Secod Text Retrieva Coferece, 994. [] Fujisawa, K., Fukuda, M., Kojima, M., ad Nakata, K. Numerica Evauatio of the SDPA (SemiDefiite Programmig Agorithm). High Performace Optimizatio, Kuwer Academic Press, 67-30, 000. [3] Hu, D. A., Pederse, J. O., ad Schütze, H. Method Combiatio for Documet Fiterig. I Proceedigs of the 9th Aua Iteratioa ACM SIGIR Coferece o Research ad Deveopmet i Iformatio Retrieva. ACM Press, 996,

8 [4] Jarvei, K.ad Kekaaie, J. IR Evauatio Methods for Retrievig Highy Reevat Documets. I Proceedigs of the 3rd Aua Iteratioa ACM SIGIR Coferece o Research ad Deveopmet i Iformatio Retrieva. ACM Press, 000, [5] Jarvei, K.. ad Kekaaie, J. Cumuated Gai-Based Evauatio of IR Techiques. ACM Trasactios o Iformatio Systems, 00. [6] Kerk, E. Aspects of Semidefiite Programmig: Iterior Poit Agorithms ad Seected Appicatios. Appied Optimizatio Series, Voume 65. Kuwer Academic Pubishers, March 00, 300 pp. [7] Lam, K. W. ad Leug, C. H. Rak Aggregatio for Metasearch Egies. I Proceedigs of the 3th Iteratioa Word Wide Web Coferece [8] Mamatha, R.., Rath, T., ad Feg, F. Modeig Score Distributios for Combiig the Outputs of Search Egies. I Proceedigs of the 4th Aua Iteratioa ACM SIGIR Coferece o Research ad Deveopmet i Iformatio Retrieva. ACM Press, 00, New York. [9] Mamatha, R. ad Sever, H. A Forma Approach to Score Normaizatio for Meta-search. I Proceedigs of HLT 0, 00, [0] Motague, M. ad Asam, J. A. Reevace Score Normaizatio for Meta-search. I Proceedigs of the 0th Coferece o Iformatio ad Kowedge Maagemet. Atata, GA, 00, [] Moteiro, R. D. C. First- ad Secod-Order Methods for Semidefiite Programmig. Georgia Tech, Jauary 003. [] Naapati, R. Discrimiative Modes for Iformatio Retrieva. I Proceedigs of the 7th Aua Iteratioa ACM SIGIR Coferece o Research ad Deveopmet i Iformatio retrieva. ACM Press, 004, [3] Pardaos, P.M. ad Wokowicz, H. Topics i Semidefiite ad Iterior Poit Methods. Fieds Istitute Commuicatios 8, AMS, Providece, Rhode Isad, 998. [4] Rada, M. E. ad Straccia, U. Web metasearch: Rak vs. Score based Rak Aggregatio Methods. I Proceedigs of the 003 ACM Symposium o Appied Computig, March 09-, 003, Meboure, Forida. [5] Robertso, S. E. Overview of the Okapi Projects. Joura of Documetatio, Vo. 53, No., 997, pp [6] Sese, J. ad Morishita, S. Rak Aggregatio Method for Bioogica Databases. Geome Iformatics, : , 00. [7] Va Erp M. ad Schomaker, L. Variats of the Borda Cout Method for Combiig Raked Cassifier Hypotheses. I Proceedigs of the 7th Iteratioa Workshop o Frotiers i Hadwritig Recogitio. Amsterdam, 000, [8] Vogt, C. ad Cottre, G. W. Fusio via a Liear Combiatio of Scores. Iformatio Retrieva, v..3, p.5-73, October 999. [9] Semidefiite Programmig. /~hemberg/semidef.htm. [30] SDPA Oie for Your Future. APPENDIX Proof of Lemma Lemma : Let Ξ = ξ i T i=,, = Ξ α T Ax, where A = Proof: Defie P = () p α kp T k= k x x, we have () p () p () p k= α k P k ad F = f ij = P T. It is cear that (k) f ij = k= α k p ji. Usig f i to deote the i th row of F, we ca rewrite Ξ as or, Ξ = ξ T i = Fx x i=,, ξ i = f i x x i = f ii x i +. j =,j i f ij x j (8..) Because the trasitio probabiity matrix has idetica rows, for the right-had side of equatio (8..), f ii x i is o-positive ad the others are o-egative. Therefore, we ca get a upper boud of ξ i as foows by usig the properties of -orm ξ i f ii x i + j =,j i f ij x j = f ii x i + j = f ij x j Appyig the resut to each eemet i Ξ yieds Ξ Cosiderig that ad i= f ii x i + i= j = f ij x j i= x i = ad i= f ij =, we obtai i= j = f ij x j = j = x j i= f ij = j = x j = i= f ii x i = i= x i i= f ii x i = i= f ii x i If further cosiderig f ii = Ξ k= α k p ii (k), we obtai i= f ii x i = i= α k p ii (k) k= x i By usig matrix form to represet this iequaity, we evetuay have Ξ α T () p () p () p Proof of Propositio () p x = α T Ax. Propositio : The optimizatio probem i (4.3) is equivaet to the foowig Semidefiite Programmig probem, max λ,γ γ s. t. λ 0 H 0 + λ 0 D UT U ΛT e λ 0 γ 0 where U = Λ T H + Λ T H + Λ 3 T H 3, ad λ = (λ 0, Λ T, Λ T, Λ 3 T ) T. Proof: (8..) Cosiderig that β = (α,, α, x,, x, t,, t m ) T, we aways have β T Dβ = α T α + x T x where D = diag(e T +, 0 T m ). Sessio: Search Quaity ad Precisio α k= k + i= x i = 488

9 It is cear that if we add this redudat costrait to the optimizatio probem (4.3), its optima soutio wi ot chage because the feasibe regio has ot chaged. I this way, we ca trasform the optimizatio probem (4.3) ito the foowig quadraticay costraied quadratic optimizatio (QCQP) probem. The Lagragia of (8..) is L λ, β mi β β T H 0 β s. t. β T Dβ H β 0 H β = e H 3 β < 0 (8..) = β T H 0 β + λ 0 β T Dβ + Λ T H β + Λ T (H β e ) + Λ 3 T H 3 β = β T (H 0 + λ 0 D)β + Uβ Λ T e λ 0 where Λ = λ, λ,, λ +m T, Λ = λ +m+, λ +m + T, Λ 3 = λ +m +3, λ +m+4,, λ ++m + T, λ = (λ 0, Λ T, Λ T, Λ 3 T ) T, ad U = Λ T H + Λ T H + Λ 3 T H 3. Accordig to the optimizatio theory [6], if the ifimum of L λ, β with respect to β exists, oe ca trasform the miimizatio of the objective fuctio i the prima probem (8..) to the maximizatio of the dua fuctio g λ = if β L λ, β. The coditio for this trasformatio is existece of the ifimum of L λ, β. We wi discuss this coditio i the foowig three cases. ) If (H 0 + λ 0 D) is positive-defiite, the fuctio L λ, β is a covex quadratic fuctio of β. Therefore, we ca fid the ifimum from the optimaity coditio: which yieds β L λ, β = (H 0 + λ 0 D)β + U T = 0 β = (H 0 + λ 0 D) U T (8..3) If (H 0 + λ 0 D) is strict positive semidefiite, usig the pseudo-iverse i [6], we ca get a reaxatio o the above coditio. That is if U ra(h 0 + λ 0 D), we ca get the dua fuctio as foows, g λ = Λ T e λ 0 4 U(H 0 + λ 0 D) U T ) Otherwise fuctio L λ, β has o ower boud, thus the probem (8..) has o soutio. With the above discussios, we cocude if ad oy if (H 0 + λ 0 D) 0, L λ, β has a ifimum ad the correspodig optimizatio probem (8..) ca be trasformed to its dua probem 3. As a resut, we ca sove the dua probem i (8..4) ad get the soutio for (8..). max λ g(λ) s. t. λ 0 (H 0 + λ 0 D) 0 (8..4) Oe ca aso fid that g λ is the Schur compemet [6] of H 0 + λ 0 D (H 0 + λ 0 D)i the matrix UT U ΛT. I this e λ 0 situatio, (8..4) ca be further formuated as a Semidefiite Programmig (SDP) probem with respect to variabes γ ad λ. max λ,γ γ s. t. λ 0 H 0 + λ 0 D Sessio: Search Quaity ad Precisio UT U ΛT e λ 0 γ 0 (8..5) For probem (8..), there exists a β that makes the foowig two iequaities true: β T Dβ <, H β < 0, ad H 3 β < 0. That is, probem (8..) is stricty feasibe, ad thus the optima vaues of (8..) ad its Lagrage dua probem (8..4) are equivaet [6]. Reca that optimizatio probem (4.3) ad (8..) are equivaet; accordig to the strog duaity theorem [6], probem (4.3) is equivaet to the SDP probem (8..5). Accordigy, we get the dua fuctio g λ = Λ T e λ 0 4 U(H 0 + λ 0 D) U T which is a cocave quadratic fuctio of. M is the pseudo-iverse of matrix M [6]. 3 M 0 meas that matrix M is semi-positive defiite [6]. 489