A Consstent Weghted Rnkng Scheme wth n Applcton to NCAA College Footbll Rnkngs Ity Fnmesser, Chm Fershtmn nd Nel Gndl 1 Mrch 20, 2007 Abstrct The NCAA college footbll rnkng, n whch the so-clled ntonl chmpon s determned, hs been plgued by controverses the lst few yers. The dffculty rses becuse there s need to mke complete rnkng of tems even though ech tem hs dfferent schedule of gmes wth dfferent set of opponents. A smlr problem rses whenever one wnts to estblsh rnkng of ptents or cdemc ournls, etc. Ths pper develops smple consstent weghted rnkng (CWR) scheme n whch the mportnce of (weghts on) every success nd flure re endogenously determned by the rnkng procedure. Ths consstency requrement does not unquely determne the rnkng, s the rnkng lso depends on set of prmeters relevnt for ech problem. For sports rnkngs, the prmeters reflect the mportnce of wnnng vs. losng, the strength of schedule nd the reltve mportnce of home vs. wy gmes. Rther thn ssgn exogenous vlues to these prmeters, we estmte them s prt of the rnkng procedure. The NCAA college footbll hs specl structure tht enbles the evluton of ech rnkng scheme nd hence, the estmton of the prmeters. Ech seson s essentlly dvded nto two prts: the regulr seson nd the post seson bowl gmes. If rnkng scheme s ccurte t should correctly predct reltvely lrge number of the bowl gme outcomes. We use ths structure to estmte the four prmeters of our rnkng functon usng hstorcl dt from the 1999-2003 sesons. Fnlly we use the prmeters tht were estmted usng the hstorcl dt (1999-2003) nd the outcome of the 2004 regulr seson to rnk the tems for 2004. We then clculte the number of bowl gmes whose outcomes were correctly predcted followng the 2004 seson. Our methodology predcted more bowl gmes correctly n 2004 thn ny of the sx computer rnkng schemes used by the BCS. 1 Fnmesser: Hrvrd Unversty, fnmesser@hbs.edu. Fershtmn: Tel Avv Unversty, Ersmus Unversty Rotterdm, nd CEPR, fersht@post.tu.c.l. Gndl: Tel Avv Unversty, nd CEPR, gndl@post.tu.c.l. We re grteful to Drew Fudenberg for helpful suggestons.
1. Introducton At the end of the regulr seson, the two top NCAA college footbll tems n the Bowl Chmponshp Seres (BCS) rnkngs ply for the so-clled ntonl chmponshp. Nevertheless, the 2003 college footbll seson ended n controversy nd two ntonl chmpons: LSU nd USC. At the end of the 2003 regulr seson Oklhom, LSU nd USC ll hd sngle loss. Although both the Assocted Press (AP) poll of wrters nd ESPN/USA Tody poll of footbll coches rnked USC #1, the computer rtngs were such tht USC ended up #3 n the offcl BCS rnkngs; hence LSU nd Oklhom plyed n the BCS chmponshp gme. Although LSU bet Oklhom n the chmponshp gme, USC (whch won ts bowl gme gnst #4 Mchgn) ws stll rnked #1 n the fnl (post bowl) AP poll. 2 The dsgreement between the polls nd the computer rnkngs followng the 2003 college footbll seson led to modfcton of the BCS rnkngs tht reduced the weght of the computer rnkngs. Why s there more controversy n the rnkng of NCAA college footbll tems thn there s n the rnkng of other sports tems? Unlke other sport legues, n whch the chmpon s ether determned by plyoff system or structure n whch ll tems ply ech other (Europen Soccer Legues for exmple), n NCAA college footbll, tems typclly ply only twelve-thrteen gmes nd yet, there re 117 tems n (the premer) Dvson I-A NCAA college footbll. Hence, controverses rse becuse there s need to mke complete rnkng of tems even though there s n ncomplete ntercton ; ech tem hs dfferent schedule of gmes wth dfferent set of opponents. In settng n whch ech tem plys gnst smll subset of the other tems nd when tems potentlly ply dfferent number of gmes, rnkng the whole group s nontrvl. If we ust dd up the wns nd losses, we obtn prtl (nd potentlly dstorted) mesure. Some tems my ply prmrly gnst strong tems whle others my ply prmrly gnst wek opponents. Clerly wns gnst hgh-qulty tems cnnot be counted the sme s wns gnst wek 2 By greement, coches who vote n the ESPN/USATody poll re supposed to rnk the wnner of the BCS chmponshp gme s the #1 tem. Hence LSU ws rnked #1 n the fnl ESPN/USA Tody poll. 1
opponents. Moreover such mesure wll crete n ncentve problem s ech tem would prefer to ply esy opponents. Smlr rnkng ssues rse whenever one wnts to estblsh rnkng of scholrs, cdemc ournls, rtcles, ptents, etc. 3 In these settngs, the rw dt for the complete rnkng re blterl cttons or nterctons between obects, or ndvduls. In the cse of cttons, t would lkely be preferble to employ some weghtng functon tht cptures the mportnce of the ctng rtcles or ptents. For exmple, weghng ech ctton by the mportnce of the ctng rtcle (or ournl) mght produce better rnkng. Such methodology s nlogous to tkng nto ccount the strength of the opponents n sports settng. The weghts n the rnkng functon cn be gven exogenously, for exmple when there s known ournl mpct fctor or prevous (.e., preseson) rnkng of tems. Lke pre-seson sport rnkngs, ournl mpct fctors re wdely vlble. The problem s tht the resultng rnkng functons use exogenous weghts. Idelly, the weght or mportnce of ech gme or ctton should be endogenously determned by the rnkng procedure tself. A consstent rnkng requres tht the outcome of the rnkng be dentcl to the weghts tht were used to form the rnkng. Ths consstency requrement ws frst employed by Lebowt nd Plmer (1984) when they constructed ther cdemc ournl rnkng. See lso Plcos-Huert, I., nd O. Vol (2004) for n xomtc pproch for determnng ntellectul nfluence nd n prtculr cdemc ournl rnkng. Ther nvrnt rnkng lso stsfes the consstency requrement. 4 Fnlly, the consstency requrement s relted to the methodology tht the Google serch 3 Cttons counts, typclly usng the Web of Scence nd/or Google Scholr, re ncresngly used n cdem n tenure nd promoton decsons. Cttons counts, typclly usng the Web of Scence nd/or Google Scholr, re ncresngly used n cdem n tenure nd promoton decsons. The mportnce of cttons n exmnng ptents s dscussed n Hll, Jffe nd Trtenberg (2000) who fnd tht "ctton weghed ptent stocks" re more hghly correlted wth frm mrket vlue thn ptent stocks themselves. The role of udcl cttons n the legl professon s consdered by Posner (2000). 4 The consstent weghted rnkng we develop cn lso be nterpreted s mesure of centrlty n network. Centrlty n networks s n mportnt ssue both n socology nd n economcs. Our mesure s vrnt of n mportnt mesure of centrlty suggested by Boncch (1985). Bllester, Clco-Armengol, nd Zenou (2006) hve shown tht the Boncch centrlty mesure hs sgnfcnt mpct on equlbrum ctons n gmes nvolvng networks. 2
engne uses to rnk WebPges. Google nterprets lnk from pge A to pge B s vote, by pge A, for pge B. But, Google looks t more thn the sheer volume of votes, or lnks pge receves; t lso nlyes the pge tht csts the vote. Votes cst by pges tht re themselves "mportnt" wegh more hevly nd help to mke other pges "mportnt". 5 In the cse of ptents or ournls rtcles, the problem s reltvely smple: ether there s ctton or there s no ctton. The problem s more complex n the cse of sports rnkngs. The outcomes of gme nclude the result - wnnng, losng, not plyng, nd n some cses, the possblty of te. Addtonlly, t s mportnt to tke nto ccount the locton of the gme, snce there s often home feld dvntge. An nlogy for wns nd losses lso exsts for the cse of cdemc ppers. One could n prncple use dt on reectons nd not ust publctons n formultng the rnkng. A reecton would be equvlent to losng nd would be treted dfferently thn not plyng (.e., or not submtted). 6 Ths pper presents smple consstent weghted rnkng (CWR) scheme to rnk gents or obects n such nterctons nd pples t to NCAA dvson 1-A college footbll. The rnkng functon we develop hs four prmeters: the vlue of wns reltve to losses, mesure tht cptures the strength of the schedule, nd mesures for the reltve mportnce of home vs. wy wns nd home vs. wy losses. Rther thn ssgn exogenous vlues to these prmeters, we estmte them s prt of the rnkng procedure. In most rnkng problems, there re not explct crter to evlute the success of proposed rnkngs. NCAA college footbll hs specl structure tht enbles the evluton of ech rnkng scheme. Ech seson s essentlly dvded nto two prts: the regulr seson nd the post seson bowl gmes. We estmte the four prmeters of our rnkng functon usng hstorcl dt from the regulr seson gmes from 1999-2003. The regulr seson rnkngs ssocted wth ech set of prmeter estmtes s then 5 Quote ppers t http://www.google.com/technology/. 6 A pper tht ws ccepted by the RAND Journl of Economcs wthout ever beng reected would be treted dfferently thn pper tht ws reected by severl other ournls before t ws ccepted by the RAND Journl. But ths s, of course, hypothetcl exmple snce such dt re not publcly vlble. 3
evluted by usng the outcomes of the bowl gmes for those fve yers. For ech vector of prmeters, the procedure uses the regulr seson outcomes to form rnkng mong the tems for ech seson. If rnkng s ccurte t should correctly predct reltvely lrge number of bowl gme outcomes. Our methodology s such tht the optml prmeter estmtes gve rse to the best overll score n bowl gmes over the 1999-2003 perod. Our estmted prmeters suggest the loss penlty from losng to very hghly rted tem s ust 1/3 the loss penlty of losng to tem wth very low rtng. Hence, our estmtes suggest tht t ndeed mtters to whom one loses: the strength of the schedule s mportnt n determnng the rnkng. Further, our estmtes re such tht tem s hevly penled for home loss, reltve to rod loss, whle home wn s rewrded only slghtly less thn wn on the rod. The welth of nformton nd rnkngs vlble on the Internet suggests tht the rtng of college footbll tems ttrcts gret del of ttenton. 7 There re, however, ust sx computer rnkng schemes tht re employed by the BCS. Comprng the CWR rnkng to these sx rnkngs ndctes tht over fve yer perod, the CWR rnkng dd pproxmtely 12-15 percent better (n predctng correct outcomes) thn the other rtngs. Ths comprson s, of course, somewht unfr, becuse our optmton methodology chose the prmeters tht led to the hghest number of correctly predcted bowl gmes durng the 1999-2003 perod. Fnlly we use the 2004 seson, whch ws not used n estmtng the prmeters of the rnkng, nd perform smple test. Usng the estmted prmeters, we employ the CWR nd the outcome of the 2004 regulr seson n order to determne the rnkng of the tems for 2004 seson. We then evlute our rnkng scheme by usng t to predct the outcome of the 2004 post seson (bowl) gmes. Our CWR rnkng scheme predcted more bowl gme outcomes correctly thn ny of the computer rnkngs used n the BCS rnkngs for 7 See http://homepges.ce.wsc.edu/~dwlson/rsfc/rte/ndex.shtml for the numerous rnkngs. Fr nd Oster (2002) compres the reltve predctve power of the BCS rnkng schemes. 4
2004 perod. Ths s, of course, only one yer comprson nd clerly not sttstclly sgnfcnt evdence regrdng the qulty of the vrous rnkngs. On the other hnd the forecstng blty of our CWR scheme should mprove s more sesons (dt) re ncluded n the estmton stge. 2. The BCS Controverses Unlke other sports, there s no plyoff system n college footbll. Hence, t ws not lwys esy for the coches nd wrters polls to gree on ntonl chmpon or the overll rnkng. The BCS rtng system whch employs both computer rnkngs nd polls ws frst mplemented n 1998 to ddress ths ssue nd try to cheve consensus ntonl chmpon, s well s help choose the eght tems tht ply n the four premer (BCS) bowl gmes. 8 Nevertheless, the 2003 college footbll seson ended n controversy nd two ntonl chmpons: LSU nd USC. The polls rted USC #1 t the end of the regulr seson, but only one of the computer formuls ncluded n the 2003 BCS rnkngs hd USC mong the top two tems. Whle ll three tems hd one loss, the computer rnkngs ndcted tht Oklhom nd LSU hd plyed stronger schedule thn USC. The dsgreement between the polls nd the computer rnkngs led to modfcton of the method used to clculte the BCS rnkngs followng the 2003 college footbll seson. Up untl tht tme, the computer rnkngs mde up pproxmtely 50 percent of the overll BSC rtngs. The 2004 BCS rnkngs were bsed on the followng three components, ech wth equl weghts: 9 (I) The ESPN/USA Tody poll of coches, (II) The Assocted Press poll of wrters, (III) Sx computer rnkngs. Hence, the weght plced on the computer rnkngs ws demoted. 10 Followng the 2004 seson, the BCS system gn cme under scrutny. The complnt nvolved Clforn (Cl) whch ppered to be on the verge of ts frst Rose Bowl 8 There re now fve BCS bowl gmes. 9 See http://www.bcsfootbll.org/news.cfm?hedlne=40 for detls. 10 If the new system hd been used durng the 2003 seson, LSU nd USC would hve plyed n the 2003 BCS chmponshp gme. 5
ppernce snce 1959. Despte Cl's vctory n ts fnl gme, t fell from 4 th to 5 th n the fnl BCS stndngs nd lost ts plce to Texs, whch clmbed to 4 th, despte beng dle the fnl weekend. Texs thus obtned the BCS' only t-lrge berth nd n ppernce n the Rose Bowl, nd Cl lost ts plce n BCS bowl gme. 11 The controversy ws due to the chnges n the polls over the lst week of the seson. In the BCS rnkng relesed followng the week endng November 27, Cl ws rnked hed of Texs. There were only few gmes the followng weekend. Cl plyed December 4 gnst Southern Msssspp becuse n erler scheduled gme between the tems hd been rned out by hurrcne. Cl bet Southern Msssspp on the rod 26-16, 12 whle Texs dd not ply. Nevertheless, Cl fell nd Texs gned n the AP nd USA Tody/ ESPN polls. The BCS computer rnkng of the two tems ws unchnged between the November 27 nd December 4 perod. If there hd been no chnges n the polls, Cl would hve plyed n the Rose bowl. Gven ts drop to 5 th, Cl ended up plyng n mnor (non BCS) bowl. 13 Tble 1 below summres the chnges tht occurred n the polls nd computer rnkngs between November 27 nd December 4. In prt becuse of the Cl controversy followng the 2004 seson, the AP nnounced tht t would no longer llow ts poll to be used n the BCS rnkngs nd ESPN wthdrew from the coches poll. Although the BCS eventully dded nother poll, better soluton mght hve been to gve more mportnce to computer rnkngs. Despte the crtcsm of computer rnkngs, they re the only ones tht cn be trnsprent nd bsed on mesurble crter, whch s to sy, mprtl. 11 Ths dscusson should not be tken s crtcsm of Texs. If the BCS hd tken the top eght tems for ts four bowl gmes tht yer, both Cl nd Texs would hve plyed n BCS bowl gme, perhps gnst ech other n the Rose Bowl. 12 Southern Msssspp fnshed the regulr seson 6-5 nd lter won ts bowl gme. 13 Ths hd fnncl mplctons beyond the prde of competng n top (BCS) bowl. Plyng n mnor (non BCS) bowl typclly mens much smller pyouts for the schools nvolved. There re lso clms tht dontons to unverstes ncrese nd the demnd for ttendng unversty ncreses n the success of the footbll tem. Frnk (2004) fnds no sttstcl support for ths clm. 6
Gmes through November 27 December 4 Actul Chnge (% chnge) Polls Cl (AP) 1410 1399-11 (-0.8%) Texs (AP) 1325 1337 +12 (+0.9%) Cl (ESPN/USA) 1314 1286-27 (-2.2%) Texs (ESPN/USA) 1266 1281 +15 (+1.2%) BCS Computer Rnkng: No chnge n Clforn s nd Texs rnkngs Gmes: Clforn 26 Southern Msssspp 16; Texs (dle) Tble 1: Chnges n Rtngs between November 27 nd December 4 3. The CWR Rnkng Methodology 3.1 Development of Consstent Rnkng We develop our forml rnkng n three steps. We frst consder smple blterl ntercton lke cttons (cted rtcles or ptent cttons). Ths s reltvely smple cse becuse ether obect ctes obect or t does not cte obect. We then consder sports settng; n ths cse, there s wnner nd loser or no gme. (In some sports settngs, there s the possblty of te. 14 ) In the fnl stge we ncorporte the possblty of two types of gmes; home gmes nd wy gmes. Ths mens tht wnnng (or losng) home gme cn hve dfferent weght thn wnnng (or losng) n wy gme. Consder group N { 1,..., n} of gents (or obects), wth the relton { 0,1} for every, N. For exmple, N s set of ptents or rtcles, = 1 f ptent or rtcle ctes ptent (or rtcle) nd = 0 otherwse. Our dtset s hence unquely defned by the mtrx [ ] A =. We nterpret ech = 1 s postve sgnl regrdng obect. The obectve s to defne rtng functon: R n : A R whch genertes rtng (nd not ust rnkng) for every gent tht summres the nformton n A. 14 In NCAA college footbll, gme ted t the end of regulton goes nto overtme nd the overtme contnues untl there s wnner. 7
There re mny possble wys to defne the functon R; the most trvl (nd commonly used) s the summton ( A) r =, = 1,..., n, whch s ust count; n exmple s the number of cttons tht ech rtcle receves. The dvntge of such rnkng s ts smplcty but t gnores much of the nformton emboded n A. Such rnkng my be pproprte when the nterctons between the obects re not mportnt; for exmple, when rnkng bestsellers, smple count of sles s probbly pproprte. In other stutons the dentty or the "mportnce" of should be tken nto ccount when ggregtng the. For exmple, n formng rnkng bsed on cttons one my wnt to tke nto ccount the "mportnce" of the ctng ptent or rtcle. One possble resoluton s cheved by usng n exogenous weghtng vector, descrbng the gents mportnce. Exmples nclude Journl Impct Fctors or the use of polls (or prevous rnkngs) n college footbll. Lettng we cn normle the count n the followng wy: m be gent's subectve sgnfcnce, ( A m) = r, m, = 1,..., n However, ths rnkng functon s not consstent. The rtng used to determne ech gent's nfluence (m ) dffers from the fnl rtng (r ) of the gents. Ths nconsstency cn be fxed by requrng tht the weght gven to ech s dentcl to the rtng tself (see Lebowt nd Plmer [1984]),.e. the rtng functon (A,) should stsfy the followng consstency requrement: ( A ) =,. To gurntee unqueness, we cn employ smple normlton requrng, for exmple, tht Σ =1 nd mn = 0. Specfclly, = 1,.., n (1) ( A ) + g, =, = 1,..., n, where mn = 0, =1,.., n + g 8
where g s endogenously determned n order to enble soluton to the system (.e., t s determned by the condton mn 0 ). In order to solve system (1) we need to = =1,.., n smultneously determne the rtngs of ll gents (nd g), snce the rtngs themselves re lso the weghts needed n the clcultons. Equton (1) s lso relted to Google s rnkng of web pges (See Brn nd Pge (1998) nd the Wkped entry on PgeRnk). In Google, the pge rnk vlue of webpge s =(1-d)/N + dl N =1, where N s the number of web pges, d s n exogenous constnt, nd l = (1/# of outgong lnks from webpge ) f webpge lnks to webpge, nd 0 otherwse. 15 The Google normlton s tht the sum of the pge rnks equls one, N.e., l = 1. = 1 3.2 Incorportng Wns nd Losses Our dscusson up to ths pont consdered the cse when { 0,1}. But n sports mtch, the outcome cn be wn, lose, or do not ply. Tems lso mght ply more thn one gme gnst ech other. To ccommodte ths we modfy the rnkng n the followng wy: For every, N, + Z ndctes the number of tmes tem won gnst tem nd + Z ndctes the number of tmes tem lost to tem, so the mtrx A = [ ] s dded to the dtset nd dentfes losses whle the mtrx A s defned s bove nd dentfes the wns. 16 Returnng to the nlogy of rnkng rtcles, f t would hve been fesble to use both cceptnce nd reecton dt, the A mtrx would be the "reecton" mtrx. 15 By defnton, l = (1/# of outgong lnks from webpge ). The dmpng fctor, d, s typclly set equl to 0.85. 16 Note tht for every, =, therefore there s no necessty n defnng the new mtrx A. However, t wll mke the presentton of the system of equtons clerer, especlly when we ntroduce further extensons. 9
n As before, our obectve s to defne consstent rnkng functon R : A, A R. Allowng for dfferent coeffcents for wns nd losses, equton (1) now becomes: (2) ( A A, ) b ( γ ) + g, =, = 1,..., n, mn = 0. = 1,.., n ( ) b γ + g There re two new prmeters n ths rnkng functon; b nd γ. These prmeters ccount for the mportnce of losses reltve to wns. As b nd γ ncrese, the rtng gves hgher weght to losses. The prmeterγ hs n ddtonl nterpretton; keepng b γ constnt, lrge γ mens tht our rnkng functon prmrly depends on the number of losses, whle smll γ mples tht the rnkng s senstve to whom one loses. To nsure tht wnnng ncreses tem s rtng nd losng decreses tem's rtng, t must be the cse tht b>0 nd γ > mx. Clerly dfferent vlues of these prmeters yeld dfferent rtngs. 3.3 Home Feld Advntge In ddton to the lrge set of possble outcomes, the locton of the gme my ffect the outcome s well. Wnnng t home s eser thn wnnng on the rod. Snce the locton of the gme s known, we cn ncorporte t n the rnkng functon by gvng dfferent weghts to wns nd losses t home nd wy gmes. Ths mens tht n ddton to provdng weghts for the reltve mportnce of wns vs. losses, weghts must lso be employed for the mportnce of home gmes vs. wy gmes. We splt ech mtrx A ( A), nto home wns (losses) nd wy wns (losses). Thus, for every pr of tems, N, there re four relevnt vlues hom e hom e wy,,, wy + Z whch (respectvely) descrbe the number of tmes tem won t home, won wy, lost t home, 10
nd lost wy, gnst tem. The four dt mtrces re: we modfy the rnkng functon s follows: home wy hom e A, A, A, A wy nd (3) = hom e wy hom e wy ( A, A, A, A, ) wy wy + h w + h w hom e hom = b e b wy wy l hom e ( γ ) + h ( γ ) l hom e ( γ ) + h ( γ ) + g + g Agn, mn 0. = =1,.., n Rod wns nd rod losses re normled to one. Hence the prmeters h w nd ccount for the weght of home wns (losses) reltve to wy wns (losses) n clcultng the rtngs. Agn dfferent vlues of these prmeters yeld dfferent rtngs. We do not ssume ny specfc vlues of these prmeters, but rther employ the unque dt to estmte them. l h 4. Estmton nd Evluton of Rnkng Prmeters Equton (3) s our rnkng functon, but t requres n nput of four exogenous prmeters: w b,γ,h, nd h l. Determnng the vlues of these prmeters mght be consdered tsk for footbll nlysts. We clerly do not clm to possess such expertse. Insted, we propose to estmte these prmeters usng dt from prevous sesons. The NCAA college footbll seson s set up n unque wy tht fclttes the evluton of dfferent rnkng schemes. There re essentlly two rounds n the college footbll seson. In the frst round, there re regulr seson gmes; n the second round, there re the so-clled bowl gmes. Tems tht ply well durng the regulr seson re nvted to bowl gmes. 11
Ths settng provdes us wth nturl experment to test the dfferent rnkng schemes. The regulr seson rnkng determnes the reltve strength of the tems. The performnce of ech rnkng cn be evluted by ts mpled predcton of the bowl gme outcomes. If rnkng s resonbly good, then n bowl gme nvolvng the #3 nd #9 tems, the probblty tht the tem rnked #3 wns the gme should be more thn 50%. We cn thus use the results of the bowl gmes to evlute the qulty of the pre-bowl rnkngs or to estmte the relevnt prmeters. Approxmtely 50% of the tems prtcpte n bowl gmes Snce we use these bowl gmes n estmtng the prmeters, our rnkng my not be tht ccurte for the tems below the medn nd cuton should be used when comprng the rnkngs of the lower rnked tems. But tht does not pose problem, snce the rnkng of the bottom hlf of brrel s much less mportnt. We use the 1999-2003 sesons to estmte the prmeters: b, γ, w h nd l h. 17 For gven set of prmeters, we construct, for every yer, unque pre-bowl consstent rtng. The second step s to exmne the bowl gmes nd determne whch set of prmeters provde the best predcton. There re clerly dfferent wys to evlute the performnce of ech rtng system; we dopt for ths pper smple rule tht selects the prmeters tht predct the hghest number of bowl gme results correctly over the fve yer perod but we lso dscuss some lterntve estmton methodologes. w l For every set of prmeters we ssgn grde G( b, γ, h, h ) whch s defned by the number of bowl gmes (durng the 1999-2003 perod) predcted correctly by the rnkng derved from these prmeters. A correct predcton mens tht the wnner of the bowl gme s the hgher rnked tem t the end of the regulr seson. Fortuntely bowl gmes re plyed t neutrl stes (.e., no home feld dvntge for ether tem) so the predcton of the outcome of the bowl gmes depends only on the tems' reltve rnkng. 17 Some of the bowl gmes of the 2003 seson, for exmple, tke plce n erly Jnury 2004. For ese of presentton we refer to them s gmes of the 2003 seson. 12
Denote tem ( b ) s the tem tht wns (loses) bowl gme. Formlly, our estmton method mnmes the followng functon (over the N bowl gmes) (4) ( (, )) b N { N, bet b} 1 φ b, where for ech bowl gme, φ, ) 2 ( b =1 f ( b, γ, w h, l h ) > b ( b, γ, w h, l h ) nd φ (, b ) = 0 otherwse. Our estmton methodology cn be thought of s nonlner generl method of moments (GMM) estmtor, where the estmtes re such tht the dstnce between the dt (the ctul outcomes of the bowl gmes) nd the model predctons re mnmed. Followng the 1999 seson there were 24 bowl gmes, followng the 2000-2001 sesons there were 25 bowl gmes ech yer, whle followng the 2002-2003 sesons there were 28 bowl gmes ech yer. Thus the mxmum overll score for the 1999-2003 perod s 130, the number of bowl gmes durng tht perod. We then sum up the number of correct predctons for the fve yers of bowl gmes ssocted wth ech set of prmeter w l estmtes. Ths gves us grde, G( b, γ, h, h ), for every set of prmeters. We frst chose reltvely brod ntervls for the prmeters n order to fnd res whch provded the best grde. The vlues chosen for the ntl grd (see Tble 3 below) were s follows: b whch ccounts for the mportnce of losses reltve to wns ws llowed to vry between 0.1 nd 2.8. Ths mens tht the mportnce of losses reltve to wns could vry between 10% nd 280%. γ ws llowed to vry between from 0.02 to 0.32. A γ of 0.32 s roughly 15 tmes the rtng of the most hghly rnked tem; hence the rnge for γ s lso very lrge. w h nd l h were chosen to llow lrge rnge s well. b γ w h l h Lower bound 0.1 0.02 0.1 0.1 Upper bound 2.8 0.32 2.8 2.8 Brod Grd ntervls 0.3 0.05 0.3 0.3 Nrrow Grd ntervls 0.1 0.01 0.1 0.1 Tble 3: Intl Grd nd Intervls 13
Usng the results from the ntl grd, we chnged nd nrrowed prmeter rnge nd ncresed the resoluton round two dstnct res tht yelded hgh grdes. 18 The best predctons were gven by two sets of prmeters n two res of the grd; these two dstnct res yelded 82 correct predctons over the fve yer perod (out of possble 130), or 63%. The two sets of prmeters shown n Tble 4 re t the center of the two regons wth the hghest scores: Prmeters b γ h w h l Estmtes Set 1 1.1 0.03 0.9 1.8 Estmtes Set 2 1.9 0.03 2.6 1.6 Tble 4: Optml Prmeter Estmtes In the frst set of optml prmeter estmtes, h w <1 whle h l >1. The lrge dfference between h w nd h l suggests tht tem s hevly penled for home loss, reltve to rod loss (whch s normled to one) nd tht home wn s rewrded only slghtly less thn rod wn (whch s normled to one). For ths set of prmeters, losng t home s key fctor n ssessng tem s rtng. When b s close to 1, wns nd losses ffect the rtngs symmetrclly. Hence, b=1.1 suggests tht rtngs re ust slghtly more senstve to losses thn wns. In order to nterpret γ, we need to know tht the hghest rtng n 2004 ws pproxmtely 0.02. Ths mens tht other thngs beng equl, the loss penlty from losng to very hghly rted tem s γ -.02 =.01, whch s 1/3 the loss penlty of losng to tem wth very low rtng (γ - 0 =.03). Hence, the reltvely low γ suggests tht t ndeed mtters to whom one loses. (A reltvely hgh γ mples tht the rnkng s more senstve to the number of losses, rther thn to whom one loses.) 18 The serch lgorthm ws wrtten n Mtlb. The lgorthm nd the complete set of results for the whole brod nd nrrow grds re vlble upon request. 14
In the second set of prmeters, b nd h w re both hgher, whle γ nd h l re essentlly unchnged reltve to the frst set of prmeters. Hence n both cses, the estmted prmeters suggest tht tem should be hevly penled for home loss (h l s reltvely lrge) nd t ndeed mtters to whom one loses (γ s reltvely smll). The two dfferent sets of prmeters gve smlr results becuse of the substtutblty mong b nd h w. For exmple, s b rses from 1.1 to 1.9, more weght s gven to losses reltve to wns. Ths effect s offset n lrge prt by hgher vlue of h w (0.9 n the frst set of prmeters nd 2.6 n the second set of prmeters), whch ncreses the mportnce of the home wns, reltve to home losses. In the ppendx, we choose two reltvely hgh nd two reltvely low vlues for ech prmeter n order to provde sense s to the shpe of the obectve functon. The low prmeters employed n Tble A1 n the ppendx re b=0.7, γ=0.03, h w =0.7, h l =0.7, whle the hgh prmeters re b =2.2, γ=0.27, h w =2.8, h l =2.8. Tble A1, whch presents the results descendng by grde, mkes t cler tht low vlues of γ nd hgh vlues of h l re crtcl for mxmng the number of correctly predcted gmes. 5. Alterntve Estmton Methods There re severl possble wys to use the regulr seson rtngs to forecst the bowl gmes results. In secton 4, we employed qute strghtforwrd methodology; the estmted prmeters were those tht predcted the hghest number of bowl gme outcomes correctly. An lterntve method s to use the rtng (rther thn the rnkng) of two tems to predct the probblty tht tem bets tem b n the bowl gme. For exmple, f, {, b} s the rtng of tem, then Pr { bets b, } order to evlute the qulty of predcton of gven rtng schedule for the bowl gmes, b = + b. In 15
one could then use lest squres method. The obectve functon to be mnmed would then be b N { N, bet b} 1. + b 2 On one hnd, ths method uses more dt thn the method we chose snce t explots the whole crdnl rtng rther thn ust the ordnl rnkng tht we used n the prevous secton. On the other hnd, there s fundmentl problem wth ths methodology: when we use the rtng tself to form /( + b ), the estmton method plces more weght on bowl gmes nvolvng lower rnked tems. Ths s becuse gven pont spred n the rnkngs between two tems wll yeld /( + b ) vlue much closer to ½ for the hgher rnked tems thn for tems lower n the rnkng. Ths problem ndeed occurs n prctce becuse closely rnked tems typclly ply ech other n bowl gmes. When we employed the ltertve estmton scheme, we obtned the followng two sets of prmeters estmtes. Prmeters b γ h w h l Estmtes Set 1 0.7 0.03 2.8 2.8 Estmtes Set 2 0.1 0.12 2.8 2.8 Tble 5: Alterntve Methodology: Prmeter Estmtes If we look t the frst set of prmeter estmtes, we see tht the estmte of b s somewht less thn our preferred results. Thus, compred to our preferred estmtes, these estmtes plce more weght on wnnng thn losng. Addtonlly, the estmte of h w s much hgher. Snce h l s hgh s well, home gmes re much more mportnt thn rod gmes. The prmeter estmtes re ntutve, snce ths (lterntve) methodology plces greter weght on the reltvely wek tems, nd these tems typclly lose on the rod. Hence, there s very lttle nformton vlble from rod gmes. 16
In the second set of prmeter estmtes, the estmte of b s very smll nd the estmte of γ s very lrge. Hence the methodology n ths cse bsclly counts the number of wns. Agn, the ntuton s tht weker tems hve fewer wns, so ny wn s very vluble, regrdless of the opponent. An nlogous problem would rse f we would use the rnkng (rther thn the rtng) to form /( + b ). Such method plces more emphss on tems tht fnsh ner the top. For exmple, n bowl gme between the top two rnked tems, the expected probblty tht tem number #1 wll wn n the methodology usng the lterntve rnkng s /( + b )=2/(2+1)=2/3. On the other hnd, n gme between tems rnked #15 nd #16, the expected probblty tht tem number #15 wll wn s 16/(16+15)=0.52. Ths dscusson suggests tht our methodology s more ttrctve thn the lterntve methodology. 6. Evlutng the Performnce of the CWR Rnkng Methodology Fnlly, we now compre our rnkng methodology wth the rnkngs of the experts. The sx computer rnkngs ncluded n the BCS rnkngs re: 19 AH- Anderson & Hester rtngs (http://www.ndersonsports.com/footbll /ACF_SOS.html), RB - Rchrd Bllngsley rtngs (http://www.cfrc.com/), CM - Colley Mtrx rtngs (http://www.colleyrnkngs.com/mtrte.pdf), KM - Kenneth Mssey rtngs (http://www.mrtngs.com/rte/cf-m.htm), JS Jeff Srgn rtngs, (http://www.ustody.com/sports/sgrn.htm), PW - Peter Wolfe rtngs (http://www.bol.ucl.edu/~prwolfe/cfootbll /rtngs.htm). In Tble 6, we report the number of correct predctons for the CWR s well s the sx BCS rnkng schemes for the 1999-2003 bowl gmes. Tble 6 shows tht over fve yer 19 There re mny other computer rnkngs n ddton to the sx used by the BCS. Mssey, for exmple, ncludes 97 rnkngs on hs comprson pge. See, for exmple, the rtngs comprson pge t the end of the regulr seson n 2003, vlble t http://www.msseyrtngs.com/cf/compre2003-15.htm. 17
perod, the CWR rnkngs do pproxmtely 12-15 percent better (n predctng correct outcomes) thn the other rtngs for whch we hve complete dt. Ths comprson s, of course, somewht unfr, becuse our optmton methodology chose the prmeters tht led to the hghest number of correctly predcted bowl gmes durng the 1999-2003 perod. Despte ths cvet, the results suggest tht there my be benefts from usng hstorcl dt to estmte the prmeters of rnkng schemes. Rnkng CWR 1 CWR 2 AH CM KM RB PW JS 1999 15 17 14 12 14 NA 20 NA NA 2000 17 16 15 13 12 16 NA NA 2001 15 16 14 14 15 12 NA NA 2002 17 17 14 15 13 13 14 13 2003 18 16 15 15 19 21 17 19 Totl 1999-2003 82 82 72 71 73 NA NA NA Tble 6: Bowl Gmes Predcted Correctly for the 1999-2003 Sesons 21 Fnlly we use the 2004 seson, whch ws not used n estmtng the prmeters of the rnkng, nd perform smple test. Usng the prmeters tht we estmted n secton 4 nd the outcome of the 2004 regulr seson, we rnk the tems. We then clculte the number of bowl gmes whose outcomes were correctly predcted followng the 2004 seson nd we compre our result wth the number of correct predctons from the sx computer rnkng schemes employed n the BCS rnkng. As Tble 7 ndctes our methodology predcted 15 or 16 out of the 28 bowl gmes n 2004 correctly, whle the sx computer rnkng schemes used by the BCS predcted between 10-14 gmes correctly. We should dd word of cuton here: whle these results re nterestng, they do not necessrly suggest ny sgnfcnt dfference between our rnkng schemes nd those of the computer rnkng schemes used by the BCS snce the comprson s only for sngle seson. Nevertheless, our lgorthm s constructed such tht the prmeters cn be reevluted every yer wth more dt nd the forecstng blty of our CWR scheme should mprove s more sesons (dt) re ncluded n the estmton stge. 20 NA= Dt Not Avlble. 21 CWR 1 refers to the frst set of prmeters dscussed n secton 4, whle CWR 2 refers to the second set of prmeters n tht secton. 18
Rnkng CWR 1 CWR 2 AH CM KM RB # of correct predctons 15 16 12 13 11 14 % of correct predctons 0.54 0.57 0.43 0.46 0.39 0.50 Tble 7: Bowl Gmes Predcted Correctly for the 2004 Seson PW 10 0.36 JS 14 0.50 7. Concludng Remrk: The pper presents consstent weghted rtng scheme nd showed how the results could be ppled n developng useful rnkngs n sports settngs. Our lgorthm s such tht the prmeters cn be reevluted every yer wth more dt. Hence, wth more dt we would expect the estmton to yeld better predctons. Whle the focus of ths pper s sport tournments, smlr lgorthm cn be used for cdemc rnkng of ppers, ournls or ptents nd my provde better nsghts thn the commonly used ctton counts. 19
References Bllester, C., Clvo-Armengol, A., nd Y. Zenou, 2006, "Who's Who n Networks. Wnted: the Key Plyer, Econometrc, 74:1403-1417 Boncch, P., 1987, Power nd Centrlty: A Fmly of Mesures, The Amercn Journl of Socology, 92:1170-1182. Brn, S., nd L. Pge, 1998, The Antomy of Lrge-Scle Hypertextul WSEb Serch Engne, Stndford Unversty, mmeo Fr R., nd J. Oster, 2002 Comprng the Predctve Informton Content of College Footbll Rtngs, mmeo, vlble t http://ppers.ssrn.com/sol3/ppers.cfm?bstrct_d=335801. Frnk, R., 2004 Chllengng the Myth: A Revew of the Lnks Among College Athletc Success, Student Qulty, nd Dontons, Knght Foundton, executve summry vlble t http://www.knghtfdn.org/defult.sp?story=thletcs/reports/2004_frnkreport/summry. html. Hll, B., Jffe, A., nd M. Trtenberg, 2000, Mrket Vlue nd Ptent Cttons: A Frst Look, NBER Workng Pper W7741. Lebowt, S. nd J. Plmer (1984), "Assessng the reltve mpcts of economc ournls" Journl of Economc Lterture, 22, 77-88. Plcos-Huert, I., nd O. Vol, 2004 The Mesure of Intellectul Influence, Econometrc, 72: 963-977. Posner, R. A. (2000) An economc nlyss of the use of ctton n the lw" Amercn Lw nd Economc Revew 2(2), 381-406. 20
Appendx: Shpe of the Obectve Functon In ths ppendx, we choose two reltvely hgh nd two reltvely low vlues for ech prmeter n order to provde sense s to the shpe of the obectve functon. The low prmeters employed n the tble below re b=0.7, γ=0.03, h w =0.7, h l =0.7, whle the hgh prmeters re b =2.2, γ=0.27, h w =2.8, h l =2.8. Grde refers to the number of correct predctons for the 1999-2003 sesons. Prmeters b γ h w h l Grde Hgh Low Hgh Hgh 81 Hgh Low Low Hgh 77 Low Low Hgh Hgh 77 Low Low Low Hgh 77 Hgh Low Hgh Low 73 Low Low Hgh Low 73 Low Low Low Low 72 Hgh Low Low Low 71 Low Hgh Hgh Low 70 Low Hgh Low Low 69 Hgh Hgh Low Low 59 Hgh Hgh Hgh Low 58 Hgh Hgh Low Hgh 52 Hgh Hgh Hgh Hgh 49 Low Hgh Hgh Hgh 49 Low Hgh Low Hgh 49 Tble A1: Shpe of the Obectve Functon 21