Ranking Optimization with Constraints

Transcription

1 Rakig Opimizaio wih Cosrais Fagzhao Wu, Ju Xu, Hag Li, Xi Jiag Tsighua Naioal Laboraory for Iformaio Sciece ad Techology, Deparme of Elecroic Egieerig, Tsighua Uiversiy, Beijig, Chia Noah s Ark Lab, Huawei Techologies Co. Ld., Sha Ti, Hog Kog wufagzhao@gmail.com, juxu@ic.ac.c, {hagli.hl, jiag.xi}@huawei.com ABSTRACT This paper addresses he problem of pos-processig of rakig i search, referred o as pos rakig. Alhough impora, o research seems o have bee coduced o he problem, paricularly wih a pricipled approach, ad i pracice ad-hoc ways of performig he ask are beig adoped. This paper formalizes he problem as cosraied opimizaio i which he cosrais represe he pos-processig rules ad he objecive fucio represes he rade-off bewee adherece o he origial rakig ad saisfacio of he rules. The opimizaio amous o refiig he origial rakig resul based o he rules. We furher propose a specific probabilisic implemeaio of he geeral formalizaio o he basis of he Bradley-Terry model, which is heoreically soud, effecive, ad efficie. Our experimeal resuls, usig bechmark daases ad eerprise search daase, show ha he proposed mehod works much beer ha several baselie mehods of uilizig rules. Caegories ad Subjec Descripors H.3.3 [Iformaio Sorage ad Rerieval]: Iformaio Search ad Rerieval Rerieval models Keywords Pos Rakig; Rakig Opimizaio; Bradley-Terry Model 1. INTRODUCTION Rece years have observed a sigifica progress i research ad developme o learig o rak, i.e., creaio of rakig models i search usig machie learig echiques. Now, i becomes a commo pracice o exploi he learig echologies o cosruc he basic rakig model of a search sysem. This paper is cocered wih pos processig of rakig, which we call pos rakig. Pos rakig Currely affiliaed wih Isiue of Compuig Techology, Chiese Academy of Scieces. Permissio o make digial or hard copies of all or par of his work for persoal or classroom use is graed wihou fee provided ha copies are o made or disribued for profi or commercial advaage ad ha copies bear his oice ad he full ciaio o he firs page. Copyrighs for compoes of his work owed by ohers ha ACM mus be hoored. Absracig wih credi is permied. To copy oherwise, or republish, o pos o servers or o redisribue o liss, requires prior specific permissio ad/or a fee. Reques permissios from permissios@acm.org. CIKM 14, November 03-07, 2014, Shaghai, Chia. Copyrigh 2014 ACM /14/11...$ hp://dx.doi.org/ / is ormally coduced a web search egies i ad-hoc maers. The paper aims o provide a pricipled approach o pos rakig, which does o seem o have bee seriously sudied previously. I pracice, here are may siuaios i which oe was o furher wis he search resuls give by he basic rakig model, i.e., o coduc pos rakig. Pos rakig is widely adoped i pracice, uder he ames of re-rakig, fial rakig, ec. For example, he query is abou a ho opic ad oe was o boos a webpage abou he opic from ews chaels o he op hree posiios, o maer how he rakig model does (oe ha i is usually hard o add such corol io a learig o rak model). I aoher example, a web page is repored o be likely a spam page, ad a immediae acio is required o demoe he posiio of he page, wihou chage of he rakig model. (See more examples i Secio 3.) Pos rakig eeds o be carried ou o oly from he viewpoi of ehacig search qualiy, bu also due o operaioal, commercial, ad eve poliical reasos. There are oher siuaios i which pos rakig appears o be ecessary, such as diversificaio of search resul [6, 27, 7], coex aware rakig [29], persoalized rakig [25, 26, 23]. Therefore, pos rakig is a ecessary ad impora process for search. Pos rakig has he followig characerisics. (1) I is usually query depede, user depede, or coex depede. (2) The effecs of i may o be achieved by usig he basic rakig model, because i is usually difficul, cosly, or eve impossible o impleme i he basic rakig model. (3) I does o eed learig or raiig. The key challege for pos rakig lies i he difficuly of formalizig he problem i a heoreically soud, effecive, ad efficie way. The origial search resul migh be very differe from he rules of pos processig, ad he rules migh also be coradicory o each oher. Thus, i is o easy o icorporae he complicaed corols io a sigle framework. Moreover, he process eeds o be coduced olie ad hus mus be very efficie. This paper proposes formalizig rakig opimizaio as a cosraied opimizaio problem. Give he rakig resul of a query by he rakig model, we perform re-rakig o he resul, by miimizig a objecive fucio uder a umber of cosrais, where he cosrais represe he rules which we wa o use for pos processig, for example, boos a page o he op k posiio, ad he objecive fucio represes he rade-off bewee adherece o he origial rakig ad saisfacio of he cosrais.

2 As he firs sudy, we propose a mehod for rakig opimizaio. Our mehod adops he Bradley-Terry model [2] for calculaig he probabiliy of a rakig lis. I realizes he opimizaio problem as miimizig he egaive log codiioal probabiliy of he origial rakig lis ad he egaive log codiioal probabiliy of he cosrais give a Bradley-Terry model. The opimizaio problem has a simple form ad is guaraeed o have a global opimal soluio. Our mehod employs gradie dece o fid he opimal soluio, wih liear order ime complexiy. I is hus a heoreically soud, effecive, ad efficie mehod. We have coduced experimes usig he LETOR bechmark daases [14, 22] ad a daase from a eerprise search egie. We ake as baselies several mehods which adjus rakig resuls wih heurisic rules. Experimeal resuls idicae ha our mehod cosisely ad sigificaly ouperform he baselie mehods i erms of NDCG o all daases, idicaig ha i is beer o employ our mehod i pos rakig. The coribuios of he paper iclude (1) formalizaio of rakig opimizaio, (2) proposal of a mehod of rakig opimizaio, (3) empirical sudy of rakig opimizaio. The res of he paper is orgaized as follows. Afer a iroducio o relaed work i Secio 2, we describe he formulaio of rakig opimizaio wih cosrais i Secio 3. We describe he proposed mehod based o Bradley-Terry model i Secio 4. Experimeal resuls ad discussios are give i Secio 5. Secio 6 cocludes his paper ad gives fuure work. 2. RELATED WORK Cosrucio of rakig model is oe of he key problems i IR. Give a query, he rakig model assigs relevace scores o he rerieved documes ad sors he documes based o he scores, ad hus i plays a impora role i search. Tradiioally, a rakig model is defied based o a small umber of facors, e.g., erm frequecy, iversed docume frequecy, ad docume legh. BM25 [24] ad LMIR (Laguage Models for Iformaio Rerieval) [21, 12] are such models. Recely, machie learig echiques are applied o cosrucio of rakig model usually usig a large umber of feaures ad a large amou labeled raiig daa, referred o as learig o rak. Mehods of learig o rak are caegorized as poiwise [18, 13], pairwise [9, 8, 3, 4], ad liswise [5, 30, 31, 32, 28] mehods, depedig o he loss fucios used. I his paper, we adop he liswise mehod of [28] o rai he basic rakig model. As explaied above, basic rakig is o eough, ad pos rakig is ecessary i may cases. I pracice, pos rakig is coduced by usig heurisic rules, ad here has o bee research o pos rakig iself, as far as we kow. There are several oher rakig issues, which ca be addressed hrough pos rakig as well, such as search resul diversificaio, persoalized rakig, ad coex aware rakig. I rece years, search resul diversificaio arises as a ho opic i IR, i which he search sysem reurs a lis of documes which are o oly releva o he query bu also cover may subopics of he query. A commo pracice for diversificaio is o see diversificaio as a pos rakig sep afer he rakig lis based o relevace is creaed [6, 27, 7]. This is because i is usually hard o model relevace ad diversiy i a sigle framework. Persoalized search may also be realized i pos rakig, for example, i wha is called clie side re-rakig [25, 26, 23]. Specifically, he rakig resul by he basic rakig model is se o he clie side ad re-rakig of he resul based o he user s ieres is coduced. Oe advaage of he approach is ha re-rakig is carried ou eirely o he clie side ad he privacy of user ca be proeced. Xiag e al. [29] have proposed coex aware rakig. For example, if he user has clicked a URL i he previous search i he same sessio, he i is very likely she will o click he same URL agai whe i appears i he resul of he curre search. Tha meas ha ideally he URL should be demoed i he curre search resul. Probabilisic models for rakig have bee sudied i s- aisics ad relaed fields from may years ago. The mos popular oes iclude Placke-Luce model [20, 15], Mallows model [16], ad Bradley-Terry model [2, 10]. Placke-Luce model is a sagewise geeraive model, which decomposes he process of geeraig a permuaio of objecs io sequeial sages. Mallows model is a disace-based model, which defies he probabiliy of a permuaio accordig o is disace o a ceroid permuaio. Bradley-Terry model calculaes he probabiliy of a permuaio by pairwise comparisos. See [17] for a review o he opic. The Placke- Luce model has bee uilized i learig o rak [5], ad he Bradley-Terry model is uilized i rakig opimizaio i his paper. 3. RANKING OPTIMIZATION WITH CONSTRAINTS Le us describe he process of pos rakig. Give a query, a rakig lis of documes is firs creaed by he basic rakig model, presumably buil by learig o rak. Pos rakig may be he coduced, depedig o he query, user, or coex, which meas a refieme of he origial rakig lis, i which some documes are moved up ad some are moved dow. The origial rakig lis is creaed maily from he viewpoi of relevace bewee query ad documes. The refied rakig lis is furher creaed from he viewpoi of qualiy, diversiy, persoalizaio, coexualizaio, ad so o. The acios of pos processig ca be realized by usig heurisic rules, which is a commo approach i pracice. A rule ca be if he query is i a lis of ermiologies, he always have he Wikipedia page of he ermiology o he op oe posiio. Aoher rule ca be if he documes are rerieved by boh he origial query ad refied queries, he have a leas oe docume rerieved by he origial query raked a he op hree posiios (o reduce he risk of opic shif). Ye aoher rule ca be if he query is presideial elecio debaes, he make all he op webpages of differe rouds of he debaes grouped ogeher i he rakig lis. How o use rules for pos rakig is o a rivial issue. Firs, he rules may o be hard rules ad hey oly represe a guidelie for refiig he iiial rakig lis. For example, a rule may be he docume should be raked a op hree posiios, i which o specific posiio is decided. Secod, muliple rules migh be applied a he same ime, ad he rules migh be coradicory o each oher. Third, differe orders of applicaios of rules migh yield differe fial rakig resuls ad hus he order of applicaios eeds also be cosidered. Fially, he rules usually oly affec a

3 Figure 1: Illusraio of rakig opimizaio wih cosrais. small umber of documes, i is impora o make a balace bewee applicaio of rules ad preservaio of he origial rakig lis. I his paper, we formalize pos rakig as cosraied opimizaio, referred o as rakig opimizaio wih cosrais. The cosrais represe he rules for pos rakig, defied as fucios over ses of permuaios. The objecive fucio represes he rade-off bewee adherece o he origial rakig lis ad saisfacio of he cosrais. The cosrais are i fac sof cosrais i he formulaio. Pos rakig is aurally performed by coducig he opimizaio problem. Therefore, he issues described above ca all be aurally solved i he framework. Suppose ha σ deoes he origial rakig lis, C deoes he se of cosrais, ad π deoes he fial rakig lis of pos rakig. The opimizaio ca be wrie as follows mi L(σ, π) + λ R(C, π), (1) π Ω N where L deoes agreeme bewee σ ad π, R deoes saisficaio of C by π, λ deoes he radeoff coefficie, ad Ω N deoes he se of all permuaios (rakig liss) o he N documes i he curre search. I he opimizaio process, we eed o fid he opimal fial rakig π. Suppose he se of cosrais C is defied as C = {c i ( )}, c i : Ω N {0, 1}. If c i (π) = 1, he permuaio π violaes cosrai c i, oherwise, c i(π) = 0. The subse of permuaios which do o violae he cosrais are good cadidaes for he fial rakig lis. We ca defie several ypes of cosrais. Top-k cosrai: A docume mus be a op k posiios. No-op-k cosrai: A docume cao be a op k posiios. Cluserig cosrai: Two or more documes should be raked ogeher. Diversiy cosrai: Two or more documes should o be raked ogeher. Figure 1 illusraes he problem of rakig opimizaio wih cosrais (oe ha he figure is oly for illusraio purposes; he se of permuaios forms a discree se, o a Euclidea space.). Ω N is he se of all possible permuaios for N documes. Ω C is he subse of permuaios saisfyig he cosrais i C. σ is he origial permuaio give by he basic rakig model. I rakig opimizaio, we aim o fid he opimal permuaio (rakig lis) π which is as close o σ as possible ad i he meaime as wihi Ω C as possible. 4. OUR METHOD I his secio, we propose a mehod of rakig opimizaio o he basis of he Bradley-Terry model. The mehod oly makes use of he op k cosrai ad he o op k cosrai. We leave o fuure work he sudy of addig oher cosrais o he mehod. 4.1 Probabilisic Approach We cosider a probabilisic approach o rakig opimizaio. We assume ha here exiss a probabilisic rakig model M which gives rise o he rakig lis π, by π = arg max τ P (τ M) ad we icorporae M io he opimizaio problem (1) o obai mi L(σ, π, M) + λ R(C, π, M). π Ω N,M Tha is o say, we ur he opimizaio problem (1) wih respec o π io a opimizaio problem wih respec o boh π ad M. Le L(σ, π, M) = log P (σ M) ad R(C, π, M) = log P (C M), where P (σ M) is he probabiliy of geeraig he permuaio σ give M ad P (C M) is he probabiliy of geeraig all of he cosrais i C give M. We firs solve ad he solve mi log P (σ M) λ log P (C M), (2) M π = arg max π Ω N P (π M). (3) where π deoes he opimal rakig lis. The ierpreaio of he mehod is as follows. Give he rakig lis σ by he basic rakig model as well as he se of cosrais C, we wa o firs fid a probabiliy model M ha ca bes explai he rakig lis as well as he cosrais (i.e., he produc of he probabiliies P (σ M) ad P (C M) is he larges, wih a rade-off coefficie). Afer M is deermied, we wa o fid he bes rakig lis give by M (i.e., he probabiliy P (π M) is he larges). I his paper, we choose he Bradley-Terry model for calculaio of P (σ M) ad P (C M). 4.2 Usig Bradley-Terry Model The Bradley-Terry model represes he probabiliy disribuio of permuaio of N documes (i geeral iems) by makig compariso amog all pairs of documes. I assumes ha he rakig model M is parameerized wih a se of N scores Θ = (θ 1,, θ N ), each correspodig o a docume. Furhermore, he parameers are assumed o be posiive ad sum o oe, i.e., θ i > 0 for i = 1,, N ad N i=1 θ i = 1. I Bradley-Terry model, he probabiliy of a preferece pair (i, j) (docume i be raked higher ha j) is defied as p ij = P {(i, j)} = θ i. θ i + θ j Give he permuaio σ, he Bradley-Terry model defies he probabiliy P (σ M) as beig proporioal o he produc of probabiliies of i raked higher ha j for all preferece pairs (i, j): P (σ M) (i,j):σ(i)<σ(j) p ij = (i,j):σ(i)<σ(j) θ i θ i + θ j. Give he cosrai se C, he Bradley-Terry model defies he probabiliy P (C M) based o he preferece pairs derived from C: P (C M) θ i p ij =, θ i + θ j c C (i,j) P c c C (i,j) P c

4 where P c is he se of preferece pairs derived from he cosrai c. The mehods for derivig preferece pairs deped o he ypes of cosrais. Here, we give mehods for he opk cosrai ad o-op-k cosrai. Give a op-k cosrai c, he se of preferece pairs P c is defied as P c = {(i, j) j : σ(j) > k}, where i is he docume o promoe i cosrai c. Similarly, Give a o-op-k cosrai c, he se of preferece pairs is P c = {(j, i) j : σ(j) k}, where i is he docume o demoe i cosrai c. We oe ha he op-k ad o-op-k cosrais ca oly be defied o he origial rakig lis σ, because oly afer he rakig lis is give oe ca perform pos rakig o i, i.e., impose cosrais o i. Thus, he op-k ad o-op-k cosrais resric he posiios of a docume i he origial rakig σ. Thus, he opimizaio i Equaio (2) becomes mi Θ θ i f(θ) = log ρ c θ i log θ i + θ j θ i + θ j (i,j):σ(i)<σ(j) c C (i,j) P c (4) N subjec o i :θ i > 0, θ i = 1, i=1 where ρ c > 0 is he weigh for cosrai c. Noe ha parameer λ i Equaio (2) has bee merged o parameers ρ c s i Equaio (4). The fial rakig π, he, ca be obaied via he maximizaio i Equaio (3). Wih he use of Bradley-Terry model, i ca be simplified o sorig of he documes i descedig order of scores i Θ. 4.3 Opimizaio I is easy o demosrae ha f(θ) = f(α Θ) for ay θ α > 0 (oe ha log i αθ θ i +θ j = log i αθ i +αθ j ). Thus we ca ge rid of he cosrai N i=1 θ i = 1 i Equaio (4). This is because for ay soluio we ca ormalize he θ i s by dividig hem wih N i=1 θi. The ormalized soluio will saisfy he cosrai ad keep f uchaged. Furhermore, he cosrais of θ i > 0 ca be discarded by replacig θ i wih e s i for i = 1,, N. Thus, he opimizaio problem i (4) becomes he followig ucosraied opimizaio problem: mi S f(s) = (i,j):σ(i)<σ(j) (log(e s i + e s j ) s i ) + ρ c (log(e si + e sj ) s i), c C (i,j) P c where S = {s 1, s 2,, s N } is he se of parameers. The objecive fucio f(s) i Equaio (5) is covex, as saed i Theorem 4.1. Theorem 4.1. f(s) is a covex fucio. Proof of he heorem ca be foud i Appedix. Theorem 4.1 idicaes ha he objecive fucio f ca be efficiely opimized wih gradie desce. The gradie w.r.. S ca (5) Algorihm 1 Rakig Opimizaio Algorihm Require: Iiial rakig σ, cosrais C, ad shrikage rae 0 < α < 1 1: S (0) radom values 2: 1 3: repea 4: S = f S S=S ( 1) {Equaio (6)} 5: γ 1 {search opimal sep size usig backrackig} 6: while f(s ( 1) γ S) > f(s ( 1) ) γ S 2 2 do 7: γ αγ 8: ed while 9: S () S ( 1) γ S {Equaio (7)} 10: : uil covergece 12: reur Θ = { es 1 Z be wrie as f = { df S df = ds i + c C ρ c j:σ(j)<σ(i) j:(j,i) P c,, es N Z ds 1,, }, where Z = N =1 es df ds N }, where df ds i e s i e s i + e s j j:σ(i)<σ(j) s e i e s i + e s j j:(i,j) P c e s j is defied as e s i + e s j s e j e s i + e s j. Thus, he updaig crierio for gradie desce is S () = S ( 1) γ () f S, (7) S=S 1 where is he ieraio umber, γ () is he opimal sep size a he i-h ieraio which is deermied by backrackig. Algorihm 1 shows he pseudo code of he opimizaio algorihm. The fial docume rakig π, he, is obaied by sorig wih he scores i Θ reured i Algorihm Aalysis Covergece We aalyze he covergecy of Algorihm 1 ad have he followig heorem: Theorem 4.2. Algorihm 1 coverges i fiie seps ad he covergece rae is O( 1 ), where ϵ > 0 is he olerace. ϵ Proof of he heorem ca be foud i Appedix. Theorem 4.2 implies ha Algorihm 1 ca reur a opimal rakig model i a reasoably shor ime, which makes i possible o apply he algorihm olie Iuiive Explaaio The gradie i Equaio (6) has a iuiive explaaio. The gradie cosiss of wo pars, oe based o he origial rakig lis σ ad he oher based o he cosrai se C. Boh pars coribue o he gradie ad are calculaed o he basis of preferece pairs. Give a preferece pair (i, j), here will be a force ha pushes he preferred docume i upward, by subracig a posiive erm es j e s i +e s j = 1 e s i sj +1 o he gradie df ds i (6) (oe ha i gradie desce s i is updaed wih egaive gradie). The erm also idicaes ha he force is relaed o he

5 j Figure 2: Iuiive explaaio of he gradie. Give a preferece pair (i, j), docume i will be pushed upward (gree solid arrow) ad docume j will be pushed dowward (red dashed arrow). The sreghs of he forces are ideical. Table 1: Saisics of daases. daase # queries #documes #relevace levels MQ MQ OHSUMED Gov Eerprise differece bewee he scores. A small s i s j, which meas he curre scores s i ad s j do o agree wih he preferece pair, leads o a srog force o promoe docume i. I he same ime, he u-preferred docume j will be pushed dowward, by addig he same erm o gradie df ds j. Figure 2 illusraes he forces ha respecively push he docume i ad docume j upward ad dowward. Give all of he preferece pairs derived from σ ad from C, he overall forces ha promoe (or demoe) a docume i are joily deermied by all he preferece pairs relaed o docume i. 5. EXPERIMENTS We coduced experimes o es he performaces of our mehod for pos rakig (rakig opimizaio). 5.1 Experime Seig We kow of o exisece of public daa available for pos rakig. As approximaio, we used relevace daases for he experimes. Tha is, we assume ha we kow ha some documes are releva (i.e., should be raked high) ad boos he raks of he documes i pos rakig. We made use of he followig subses of he LETOR bechmark daase: MQ2007, MQ2008, OHSUMED ad.gov, as well as a daase of eerprise search, deoed as Eerprise. Table 1 gives he saisics of he four LETOR daases [14, 22] ad he Eerprise daase. The Eerprise daase cosiss of 183 queries; each query is associaed wih abou 30 documes. I oal, he daase coais 5464 query-docume pairs. Each query-docume pair is assiged wih a label represeig relevace a hree levels: Good, Fair, or Bad. The daase is spli io raiig daa (130 queries) ad es daa (53 queries). The rakig models were raied usig [28], which is a sae-of-he-ar mehod i learig o rak. The sadard feaures i LETOR daases were uilized ad we also defied 18 feaures for Eerprise, icludig BM25 [24] ad word level edi disace 1, ec. Two ypes of cosrais were esed i our experimes: he op-k cosrais ad o-op-k cosrais. For each es query, we geeraed oe op-k (k = 1, 3, 5) cosrai ad oe o-op-k (k = 5, 10) cosrai based o he labels 1 hp://e.wikipedia.org/wiki/edi disace i of documes wih respec o he query 2. Specifically, we sored he documes accordig o heir labels (o obai a perfec rakig) ad radomly seleced oe docume i from he op k posiios. The we creaed a cosrai c which saes ha he seleced docume i should be raked o op k posiios i he fial rakig. Similarly, we also creaed a o-op-k cosrai by radomly selec a docume j from posiios afer k i he perfec rakig. For simpliciy, we assumed ha all op-k (ad all o-opk) cosrais are equally impora ad ake he values of ρ (ad ρ ). Thus, he rakig opimizaio of Equaio (5) becomes mi f(s) = (log(e s i + e s j ) s i) S (i,j):σ(i)<σ(j) +ρ (i,j) P (log(e si + e sj ) s i) +ρ (i,j) P (log(e si + e sj ) s i ). As for baselie mehods, we use he followig four heurisics for modifyig he origial rakig: For he op-k cosrai, always raks he seleced docume o op oe posiio. For he oop-k cosrai, i always raks he seleced docume o he boom posiio of he lis. For he op-k cosrai, always raks he seleced docume o he middle of he op k posiios. For he o-op-k cosrai, i always raks he seleced docume o he middle of he remaiig lis afer k. For he op-k cosrai, always raks he docume o he posiio of k. For he o-op-k cosrai, always raks he docume o he posiio of k + 1. The above hree baselies do o cosider he posiio of he docume i he origial lis. promoes he docume seleced by he op-k cosrai o he posiio of k pos, where pos is he N rak of he docume i he origial rakig lis ad is he ceilig fucio. Therefore, he docume will be raked higher if is origial posiio is also high. Similarly, he baselie demoes he docume s- eleced by he o-op-k cosrai o he posiio of k + pos(1 k ). N As evaluaio measures, Normalized Discoued Cumulaive Gai (NDCG) [11] a posiios 1, 3, ad 5 were used. 5.2 Experimeal Resuls Resuls o LETOR I all he four daases i LETOR, he queries ad associaed documes were spli o 5 subses, ad 5-fold cross validaios were coduced. The performaces repored here are he averages over 5 rials. For each daase, we used he raiig daa o lear he basic rakig model, he validaio daa o ue he parameers, ad he es daa o perform rakig opimizaio. There are wo parameers ρ ad ρ ued wih he validae 2 Rakig opimizaio ca be coduced or o coduced depedig o queries. Here for experimeaio purpose, rakig opimizaio is assumed o be carried ou for all queries. (8)

6 (a) op-3, o-op-5, ρ =10, ρ =0 (b) op-3, o-op-10, ρ =10, ρ =10 (c) op-5, o-op-10, ρ =10, ρ =10 Figure 3: Rakig accuracies o MQ2008 daase. (a) op-3, o-op-5, ρ =100, ρ =10 (b) op-3, o-op-10, ρ =100, ρ =10 (c) op-5, o-op-10, ρ =100, ρ =10 Figure 4: Rakig accuracies o MQ2007 daase (a) op-3, o-op-5, ρ =200, ρ =0 (b) op-3, o-op-10, ρ =200, ρ =10 (c) op-5, o-op-10, ρ =200, ρ =30 Figure 5: Rakig accuracies o OHSUMED daase. (a) op-3, o-op-5, ρ =50, ρ =50 (b) op-3, o-op-10, ρ =50, ρ =50 (c) op-5, o-op-10, ρ =50, ρ =50 Figure 6: Rakig accuracies o.gov daase.

7 1 1 1 (a) op-3, o-op-5, ρ =60, ρ =40 (b) op-3, o-op-10, ρ =60, ρ =40 (c) op-5, o-op-10, ρ =60, ρ =40 Figure 7: Rakig accuracies o Eerprise daase. Table 2: Average ime (i millisecods) of rakig opimizaio i seig of (op-5, o-op-10). MQ2008 MQ2007 OHSUMED.Gov Eerprise ime se. The performaces o he es ses are hose based o he bes performig parameers. We esed all he six combiaios of op-k cosrai (k = 1, 3, 5) ad o-op-k cosrai (k = 5, 10) o all of he four daases. Figures 3, 4, 5, ad 6 repor he experimeal resuls o MQ2008, MQ2007, OHSUMED, ad.gov, respecively. Our rakig opimizaio mehod is deoed as i he figures. The bes performig parameers of ρ ad ρ are also show. The performaces of he combiaios (op-3, o-op-5), (op-3, o-op-10), ad (op-5, o-op-10) are repored. From he resuls, we ca see ha our mehod ouperforms he baselies as well as he basic rakig model of o all daases. The experimeal resuls for (op- 3, o-op-5) ad (op-3, o-op-10) are very similar. This is because all he mehods focus o he op of rakig lis. The chages of k o he o-op-k cosrai have less impac o he fial rakig lis. Our mehod ad he baselies perform equally well, whe k is 1 for he op-k cosrai. This is because he op-1 cosrai meas promoig he docume o he op oe posiio, ad our mehod ca fucio as a hard rule i such case ad produce he same rakig resul as he baselies (we omi he resul because of space limiaio). We will have discussios o he effec of differe k values i Secio 5.3. We coduced sigifica ess (-es) o he improvemes of our mehod over all he baselies. The resuls idicae ha all he improvemes are saisically sigifica (p-value < 0.05), excep (op-5, o-op-10) over o MQ2008 i erms of NDCG@5, (op-3, o-op-5) ad (op-3, o-op-10) over o OHSUMED ad.gov i erms of NDCG@3 ad NDCG@5. Noe ha whe op-1 cosrai is adoped, our mehod performs equally well wih he baselies. Table 2 repors average ruig ime of rakig opimizaio by our mehod (wih op-5 ad o-op-10 cosrais) i millisecods o a Lapop PC wih 2.4GHZ CPU ad 4G- B memory. We ca see ha our mehod rus very fas, eve o a upowerful machie. We also observed ha for mos queries he algorihm coverges wihi 10 ieraios. Similar experimeal resuls were also observed i oher experimes. The resuls empirically verify he coclusio of Theorem Resuls o Eerprise I he experime, we uilized he raiig daa o lear he basic rakig model ad o ue he parameers ρ ad ρ. The es daa was uilized o perform rakig opimizaio. We esed all he six combiaios of op-k cosrai (k = 1, 3, 5) ad o-op-k cosrai (k = 5, 10) o he Eerprise daase. The performaces of he combiaios (op-3, o-op-5), (op-3, o-op-10), ad (op-5, oop-10) are repored i Figure 8. Our rakig opimizaio mehod is deoed as i he figures. From he resuls, we ca see ha our mehod ouperforms he baselies as well as he basic rakig model of o all daases. The experimeal resuls for (op-3, o-op-5) ad (op-3, o-op-10) are very similar, as i he experimes o LETOR daases. We coduced -es o he improvemes of our mehod over he baselies i erms of NDCG@1, NDCG@3 ad NDCG@5. The improvemes are saisically sigifica. Agai, our mehod ad he baselies perform equally well whe he op-1 cosrai is adoped. We omi he resul because of space limiaio. We also esed he ruig ime of our mehod of rakig opimizaio, i he seig of op-5 ad o-op-10 cosrais. The las colum of Table 2 repors average ime of our mehod i millisecods. We ca see ha our mehod rus very fas. Similar experimeal resuls were obaied for oher combiaios of op-k ad o-op-k cosrais. 5.3 Discussios We firs ivesigaed he reasos ha our mehod of rakig opimizaio ouperforms he baselie mehods. We foud ha i geeral our mehod ca really work beer ha he baselies o make a good comprise bewee usig he origial rakig ad usig he cosrais. Here, we use he resul of MQ2008 daase wih regard o wo queries o illusrae why our mehod is superior o he baselies. Figure 8(a) shows he origial rakig by ad he fial rakigs by differe mehods wih regard o oe query. The empy blocks, grid blocks, ad filled blocks represe o releva documes, parially releva documes, ad releva documes, respecively. The documes seleced by op-k cosrais ad o-op-k cosrais are marked wih ad, respecively. From he resuls, we ca see ha our mehod of rakig opimiza-

8 5 5 5 (a) Example 1 (b) Example k (a) Top-k cosrai Figure 8: Example rakigs from MQ2008. io ca really promoe he releva docume (marked wih ) ad demoe he o releva documes (marked wih ). I his case, our mehod ouperforms he baselies of,, ad. The resuls wih regard o he oher query are repored i Figure 8(b), which is a oisy case i which he docume seleced by he op-k cosrai is acually o releva (oises may also exis i he rules i pracice). Our mehod of rakig opimizaio also akes io accou he posiio of he docume i he origial rakig lis give by ad hus does o promoe i oo much. I oher words, our mehod ca make a good rade-off bewee he cosrais ad he origial rakig. O he oher had, he baselies of ad do o have such cosideraio ad cao rak he docume saisfacorily. Therefore, our mehod is more capable for pos rakig ha he baselies. We furher coduced experimes o see he impac of differe k values o differe mehods. For he op-k cosrai, larger k meas a sofer corol o he fial rakig lis. Whe k = 1, he cosrai becomes a hard rule. From he resuls repored i Figure 9(a), we ca see ha our mehod of rakig opimizaio always works beer ha or as well as he baselies for all k values. Whe k ges close o oe, our mehod will perform similarly as he baselies; whe k ges larger, he improvemes of our mehod over he baselies will also be larger. The resuls idicae ha our mehod is more robus ha he baselies. We also oe ha he baselie of hurs he basic rakig resuls whe k ges large. This is because is very sesiive o he oise i he cosrais (rules). The resuls idicae ha i is risky o direcly apply o pos rakig hough i ouperforms he oher baselies i mos of he experimes. O he oher had, for he o-op-k cosrai, smaller k meas a sofer corol o he fial rakig lis. Whe k = N 1, he cosrai becomes a hard rule. From he resuls repored i Figure 9(b), we ca see ha our mehod of rakig opimizaio always works beer ha or as well as he baselies for all k values. Whe k ges close o oe, our mehod will perform similarly as he baselies, because he o-op-k cosrai has a sof corol o he fial rakig lis. Whe k ges larger, he improvemes of our mehod over he baselies will also become larger. There is a peak k (b) No-op-k cosrai Figure 9: Performaces of rakig opimizaio wih respec o differe k values i erms of NDCG@1. for performace of our mehod aroud k = 5. The performace will drop afer k = 5. This is because he o-op-k cosrai impacs more o he ail of he rakig lis, ad a larger k will have less impac o he op of he rakig lis, which is more impora i rakig evaluaio. We also ivesigaed he ifluece of differe ypes of cosrais. Specifically, we esed performaces of our mehod of rak opimizaio wihou he cosrais ( oly), wih op-5 cosrai oly, wih o-op-10 cosrai oly, ad wih boh ypes of cosrais. Figure 10 repors he resuls. From he figure we ca see ha he wo ypes of cosrais ca idividually improve he rakig performaces if hey are adoped. The performaces ca be furher improved if hey are used simulaeously. The resuls idicae ha our mehod of rakig opimizaio ca leverage muliple ypes of cosrais wihi oe framework. The op-5 cosrai ouperforms he o-op-10 cosrai, because he evaluaio measure of NDCG emphasizes he imporace of op rakig, which is also he focus of he op-k-cosrai. Fially, we evaluaed how sesiive our mehod of rakig opimizaio is o he parameer seigs. I he experime, wo parameers ρ ad ρ were esed, which are weighs of he op-k cosrai ad o-op-k cosrai, respecively.

9 5 5 5 Oly op k Oly o op k Boh Figure 10: Performaces of rakig opimizaio wih differe cosrai ypes. NDCG NDCG NDCG@1 NDCG@3 NDCG@ weigh (a) ρ NDCG@1 NDCG@3 NDCG@ weigh (b) ρ Figure 11: Performaces of rakig opimizaio wih respec o differe parameer seigs. We chaged oe parameer ad fixed he oher o is opimal value. Figure 11(a) ad Figure 11(b) show he performaces of rakig opimizaio of our mehod i erms of NDCG a he posiios of 1, 3, ad 5. From he resuls, we ca see ha our mehod is o sesiive o he parameer seigs, ad hus is quie robus. 6. CONCLUSION I his paper, we have sudied he problem of pos processig of rakig i search, which we call pos rakig. Pracices o pos rakig i realiy ed o be heurisic ad we have, perhaps for he firs ime, formalized he problem as a opimizaio problem. I he formulaio, we represe he pos processig rules as cosrais ad maage o miimize he objecive fucio deoig he rade-off bewee he origial rakig ad he cosrais. As a resul, oe ca perform pos rakig hrough solvig he opimizaio issue. We have also give a specific probabilisic implemeaio of he opimizaio formulaio. Barley-Terry model is employed o calculaig he probabiliy of rakig lis. The objecive fucio is defied based o he codiioal probabiliy of he origial rakig ad he codiioal probabiliy of he cosrais give he model. The opimizaio amous o miimizaio of he sum of he egaive log probabiliies. We have compared he performaces of our mehod wih several baselies i experimes usig a umber of daases icludig bechmark daases. The baselie mehods represe pracical mehods of usig rules. The resuls show ha i is always beer o employ our mehod i pos rakig ha he baselies. There are sill may ope quesios wih regard o rakig opimizaio. We pla o coduc more research o he problem i he fuure. The ope quesios iclude (1) wheher here exiss a more geeral framework for rakig opimizaio, (2) how o defie ad icorporae oher ypes of cosrais io he framework, (3) how o aurally add diversificaio of resuls, ec. io he framework, (4) wheher here are more effecive ad efficie mehods for he ask. 7. REFERENCES [1] S. Boyd ad L. Vadeberghe. Covex Opimizaio. Cambridge Uiversiy Press, New York, USA, [2] R. A. Bradley ad M. E. Terry. The rak aalysis of icomplee block desigs I. The mehod of paired comparisos. Biomerika, 39: , [3] C. Burges, T. Shaked, E. Reshaw, A. Lazier, M. Deeds, N. Hamilo, ad G. Hulleder. Learig o rak usig gradie desce. I Proceedigs of he 22Nd Ieraioal Coferece o Machie Learig, ICML 05, pages 89 96, [4] Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Huag, ad H.-W. Ho. Adapig rakig svm o docume rerieval. I Proceedigs of he 29h Aual Ieraioal ACM SIGIR Coferece, SIGIR 06, pages , [5] Z. Cao, T. Qi, T.-Y. Liu, M.-F. Tsai, ad H. Li. Learig o rak: from pairwise approach o liswise approach. I ICML 07: Proceedigs of he 24h ieraioal coferece o Machie learig, pages , [6] J. Carboell ad J. Goldsei. The use of mmr, diversiy-based rerakig for reorderig documes ad producig summaries. I Proceedigs of he 21s Aual Ieraioal ACM SIGIR Coferece, SIGIR 98, pages , [7] Z. Dou, S. Hu, K. Che, R. Sog, ad J.-R. We. Muli-dimesioal search resul diversificaio. I

10 Proceedigs of he Fourh ACM Ieraioal Coferece o Web Search ad Daa Miig, WSDM 11, pages , [8] Y. Freud, R. Iyer, R. E. Schapire, ad Y. Siger. A efficie boosig algorihm for combiig prefereces. J. Mach. Lear. Res., 4: , Dec [9] R. Herbrich, T. Graepel, ad K. Obermayer. Large margi rak boudaries for ordial regressio. I Advaces i Large Margi Classifiers, pages , Cambridge, MA, MIT Press. [10] R. Huer. Mm algorihms for geeralized bradley-erry models. The Aals of Saisics, 32:2004, [11] K. Järveli ad J. Kekäläie. Cumulaed gai-based evaluaio of ir echiques. ACM Tras. If. Sys., 20(4): , Oc [12] J. Laffery ad C. Zhai. Docume laguage models, query models, ad risk miimizaio for iformaio rerieval. I Proceedigs of he 24h Aual Ieraioal ACM SIGIR Coferece, SIGIR 01, pages , [13] P. Li, C. J. C. Burges, ad Q. Wu. Mcrak: Learig o rak usig muliple classificaio ad gradie boosig. I NIPS, [14] T.-Y. Liu, J. Xu, T. Qi, W. Xiog, ad H. Li. Leor: Bechmark daase for research o learig o rak for iformaio rerieval. I Proceedigs of SIGIR 2007 workshop o learig o rak for iformaio rerieval, pages 3 10, [15] R. D. Luce. Idividual Choice Behavior: A heoreical aalysis. Wiley, [16] C. L. Mallows. No-ull rakig models. i. Biomerika, 44(1-2): , Jue [17] J. I. Marde. Aalyzig ad Modelig Rak Daa. Chapma & Hall, [18] R. Nallapai. Discrimiaive models for iformaio rerieval. I Proceedigs of he 27h Aual Ieraioal ACM SIGIR Coferece, SIGIR 04, pages 64 71, [19] Y. Neserov ad I. Neserov. Iroducory Lecures o Covex Opimizaio: A Basic Course. Applied Opimizaio. Spriger, [20] R. L. Placke. The aalysis of permuaios. Applied Saisics, 24: , [21] J. M. Poe ad W. B. Crof. A laguage modelig approach o iformaio rerieval. I Proceedigs of he 21s Aual Ieraioal ACM SIGIR Coferece, SIGIR 98, pages , [22] T. Qi, T.-Y. Liu, J. Xu, ad H. Li. Leor: A bechmark collecio for research o learig o rak for iformaio rerieval. If. Rer., 13(4): , Aug [23] F. Radliski ad S. Dumais. Improvig persoalized web search usig resul diversificaio. I Proceedigs of he 29h Aual Ieraioal ACM SIGIR Coferece, SIGIR 06, pages , [24] S. Roberso ad D. A. Hull. The rec-9 filerig rack fial repor. pages 25 40, [25] K. Sugiyama, K. Haao, ad M. Yoshikawa. Adapive web search based o user profile cosruced wihou ay effor from users. I Proceedigs of he 13h Ieraioal Coferece o World Wide Web, pages , [26] J. Teeva, S. T. Dumais, ad E. Horviz. Persoalizig search via auomaed aalysis of ieress ad aciviies. I Proceedigs of he 28h Aual Ieraioal ACM SIGIR Coferece, SIGIR 05, pages , [27] E. Vee, U. Srivasava, J. Shamugasudaram, P. Bha, ad S. Yahia. Efficie compuaio of diverse query resuls. I IEEE 24h Ieraioal Coferece o Daa Egieerig, pages , [28] Q. Wu, C. J. Burges, K. M. Svore, ad J. Gao. Adapig boosig for iformaio rerieval measures. If. Rer., 13(3): , Jue [29] B. Xiag, D. Jiag, J. Pei, X. Su, E. Che, ad H. Li. Coex-aware rakig i web search. I Proceedigs of he 33rd Ieraioal ACM SIGIR Coferece, SIGIR 10, pages , [30] J. Xu ad H. Li. Adarak: A boosig algorihm for iformaio rerieval. I Proceedigs of he 30h Aual Ieraioal ACM SIGIR Coferece, SIGIR 07, pages , [31] J. Xu, T.-Y. Liu, M. Lu, H. Li, ad W.-Y. Ma. Direcly opimizig evaluaio measures i learig o rak. I Proceedigs of he 31s Aual Ieraioal ACM SIGIR Coferece, SIGIR 08, pages , [32] Y. Yue, T. Filey, F. Radliski, ad T. Joachims. A suppor vecor mehod for opimizig average precisio. I Proceedigs of he 30h Aual Ieraioal ACM SIGIR Coferece, SIGIR 07, pages , Appedix Proof of Theorem 4.1. Proof. The objecive fucio f ca be decomposed as f = f a f b, where f a = (i,j) P a (i,j) log(e s i + e s j ) ad f b = N i=1 b i s i. Here a (i,j) 0 for all (i, j) pairs i P, which is he uio of all preferece pairs. b i 0 for i = 1,, N. e s i is a log-covex fucio w.r. s i. Thus, log(e s i + e s j ) is also covex for ay (i, j) P, because log-covexiy ca be persevered uder addiio (cf., [1]). Hece, f a is covex because a (i,j) 0 holds for all (i, j) P. Fially, we coclude ha f = f a f b is covex because f b is liear. Proof of Theorem 4.2. Proof. Sice s i < log(e s i + e s j ) holds for ay preferece pair (i, j), obviously he objecive fucio f i Equaio (5) has a lower boud of 0. Also, accordig o Theorem 4.1, f is covex. Thus, here exis oe ad oly oe opimal soluio for he problem of Equaio (5), deoed as S. Sice gradie desce ad backrackig are adoped for he opimizaio, he followig iequaliy holds (cf., [19]): f(s () ) f(s ) S(0) S 2, 2η mi where is he umber of ieraios, η mi = mi{1, α/l}, L is Lipschiz cosa for f, ad α is he shrikage rae. Thus, Algorihm 1 ca coverge wihi O( 1 ) o reach he ϵ olerace ϵ.