Learning Bounded Treewidth Bayesian Networks

Transcription

1 Learning Bounded Treewidh Bayeian Nework Gal Elidan Deparmen of Saiic Hebrew Univeriy Jerualem, 91905, Irael Sephen Gould Deparmen of Elecrical Engineering Sanford Univeriy Sanford, CA 94305, USA Abrac Wih he increaed availabiliy of daa for complex domain, i i deirable o learn Bayeian nework rucure ha are ufficienly expreive for generalizaion while alo allowing for racable inference. While he mehod of hin juncion ree can, in principle, be ued for hi purpoe, i fully greedy naure make i prone o overfiing, paricularly when daa i carce. In hi work we preen a novel mehod for learning Bayeian nework of bounded reewidh ha employ global rucure modificaion and ha i polynomial in he ize of he graph and he reewidh bound. A he hear of our mehod i a riangulaed graph ha we dynamically updae in a way ha faciliae he addiion of chain rucure ha increae he bound on he model reewidh by a mo one. We demonrae he effecivene of our reewidh-friendly mehod on everal real-life daae. Imporanly, we alo how ha by uing global operaor, we are able o achieve beer generalizaion even when learning Bayeian nework of unbounded reewidh. 1 Inroducion Recen year have een a urge of readily available daa for complex and varied domain. Accordingly, increaed aenion ha been direced oward he auomaic learning of complex probabiliic graphical model [22], and in paricular learning he rucure of a Bayeian nework. Wih he goal of making predicion or providing probabiliic explanaion, i i deirable o learn model ha generalize well and a he ame ime have low inference complexiy or a mall reewidh [23]. While learning opimal ree-rucured model i eay [5], learning he opimal rucure of general and even quie imple (e.g., poly-ree, chain) Bayeian nework i compuaionally difficul [8, 10, 19]. Several work aemp o generalize he ree-rucure reul of Chow and Liu [5], eiher by making aumpion abou he rue diribuion (e.g., [1, 21]), by earching for a local maxima over ree mixure [20], or by approximae mehod ha are polynomial in he ize of he graph bu exponenial in he reewidh bound (e.g., [3, 15]). In he conex of general Bayeian nework, he hin juncion ree approach of Bach and Jordan [2] i a local greedy earch procedure ha relie a each ep on ree-decompoiion heuriic echnique for compuing an upper bound he rue reewidh of he model. Like any local earch approach, hi mehod doe no provide performance guaranee bu i appealing in i abiliy o efficienly learn model wih an arbirary reewidh bound. The hin juncion ree mehod, however, uffer from wo imporan limiaion. Fir, while ueful on average, even he be of he ree-decompoiion heuriic exhibi ome variance in he reewidh eimae [16]. A a reul, a ingle edge addiion can lead o a jump in he reewidh eimae depie he fac ha i can increae he rue reewidh by a mo one. More imporanly, rucure learning core (e.g., BIC, BDe) end o learn puriou edge ha reul in overfiing when he number of ample i relaively mall, a phenomenon ha i made wore by a fully greedy approach. Inuiively, o generalize well, we wan o learn bounded reewidh Bayeian nework where rucure modificaion are globally beneficial (i.e., conribue o he core in many region of he nework). In hi work we propoe a novel mehod for efficienly learning Bayeian nework of bounded reewidh ha addree hee concern. A he hear of our mehod i a dynamic updae of he riangulaion of he model in a way ha i ree-widh friendly: he reewidh of he riangulaed graph (upper bound on he model rue reewidh) i guaraneed o increae by a mo one when an 1

2 edge i added o he nework. Building on he ingle edge riangulaion, we characerize e of edge ha are joinly reewidh-friendly. We ue hi characerizaion in a dynamic programming approach for learning he opimal reewidh-friendly chain wih repec o a node ordering. Finally, we learn a bounded reewidh Bayeian nework by ieraively augmening he model wih uch chain. Inead of uing local edge modificaion, our mehod progree by incremenally adding chain rucure ha are globally beneficial, improving our abiliy o generalize. We are alo able o guaranee ha he bound on he model reewidh grow by a mo one a each ieraion. Thu, our mehod reemble he global naure of Chow and Liu [5] more cloely han he hin juncion ree approach of Bach and Jordan [2], while being applicable in pracice o any deired reewidh. We evaluae our mehod on everal challenging real-life daae and how ha our mehod i able o learn richer model ha generalize beer han he hin juncion ree approach a well a an unbounded aggreive earch raegy. Furhermore, we how ha even when learning model wih unbounded reewidh, by uing global rucure modificaion operaor, we are beer able o cope wih he problem of local maxima and learn beer model. 2 Background: Bayeian nework and ree decompoiion A Bayeian nework [22] i a pair (G,Θ) ha encode a join probabiliy diribuion over a finie e X = {X 1,...,X n } of random variable. G i a direced acyclic graph whoe node correpond o he variable in X. The parameer Θ Xi Pa i encode local condiional probabiliy diribuion (CPD) for each node X i given i paren in G. Togeher, hee define a unique join probabiliy diribuion over X given by P(X 1,...,X n ) = n i=1 P(X i Pa i ). Given a rucure G and a complee raining e D, eimaing he (regularized) maximum likelihood (ML) parameer i eay for many choice of CPD (ee [14] for deail). Learning he rucure of a nework, however, i generally NP-hard [4, 10, 19] a he number of poible rucure i uperexponenial in he number of variable. In pracice, rucure learning relie on a greedy earch procedure ha examine eay o evaluae local rucure change (add, delee or revere an edge). Thi earch i uually guided by a decompoable core ha balance he likelihood of he daa and he complexiy of he model (e.g., BIC [24], Bayeian core [14]). Chow and Liu [5] howed ha he ML ree can be learned efficienly. Their reul i eaily generalized o any decompoable core. Given a model, we are inereed in he ak of inference, or evaluaing querie of he form P(Y Z) where Y and Z are arbirary ube of X. Thi ak i, in general, NP-hard [7], excep when G i ree rucured. The acual complexiy of inference in a Bayeian nework i proporional o i reewidh [23] which, roughly peaking, meaure how cloely he nework reemble a ree. The noion of ree-decompoiion and reewidh were inroduced by Roberon and Seymour [23]: 1 Definiion 2.1: A ree-decompoiion of an undireced graph H = (V,E) i a pair ({C i } i T, T ), where T i a ree, {C i } i a ube of V uch ha i T C i = V and where for all edge (v,w) E here exi an i T wih v C i and w C i. for all i,j,k T : if j i on he (unique) pah from i o k in T, hen C i C k C j. The reewidh of a ree-decompoiion i defined o be max i T C i 1. The reewidh TW(H) of an undireced graph H i he minimum reewidh over all poible ree-decompoiion of H. An equivalen noion of reewidh can be phraed in erm of a graph ha i a riangulaion of H. Definiion 2.2: An induced pah P in an undireced graph H i a pah uch ha for every nonadjacen verice p i,p j P here i no edge (p i p j ) H. A riangulaed (chordal) graph i an undireced graph wih no induced cycle. Equivalenly, i i an undireced graph in which every cycle of lengh four or more conain a chord. I can be eaily hown ha he reewidh of a riangulaed graph i he ize of he maximal clique of he graph minu one [23]. The reewidh of an undireced graph H i hen he minimum reewidh of all riangulaion of H. For he underlying direced acyclic graph of a Bayeian nework, he reewidh can be characerized via a riangulaion of he moralized graph. Definiion 2.3: A moralized graph M of a direced acyclic graph G i an undireced graph ha ha an edge (i j) for every (i j) G and an edge (p q) for every pair (p i),(q i) G. 1 The ree-decompoiion properie are equivalen o he correponding family preerving and running inerecion properie of clique ree inroduced by Laurizen and Spiegelhaler [17] a around he ame ime. 2

3 Inpu: daae D, reewidh bound K Oupu: a nework wih reewidh K G be coring ree M + undireced keleon of G k 1 While k < K O node ordering given G and M + C be chain wih repec o O G G C Foreach (i j) C do M + EdgeUpdae(M +, (i j)) k maximal clique ize of M + Greedily add edge while reewidh K Reurn G v 1 p 1 v 2 v 3 p 1 p 2 p 1 p 2 (a) (b) (c) p2 p 1 p 2 (d) (e) (f) Figure 1: (lef) Ouline of our algorihm for learning Bayeian nework of bounded reewidh. (righ) An example of he differen ep of our riangulaion procedure (b)-(e) when ( ) i added o he graph in (a). The block are {, v 1}, {v 1, }, and {, v 2, v 3, p 1, p 2, } wih correponding cu-verice v 1 and. The augmened graph (e) ha a reewidh of hree (maximal clique of ize four). An alernaive riangulaion (f), connecing o, would reul in a maximal clique of ize five. The reewidh of a Bayeian nework graph G i defined a he reewidh of i moralized graph M. I follow ha he maximal clique of any moralized riangulaion of G i an upper bound on he reewidh of he model, and hu i inference complexiy. 3 Learning Bounded Treewidh Bayeian Nework In hi ecion we ouline our approach for learning Bayeian nework given an arbirary reewidh bound ha i polynomial in boh he number of variable and he deired reewidh. We rely on global rucure modificaion ha are opimal wih repec o a node ordering. A he hear of our mehod i he idea of uing a dynamically mainained riangulaed graph o upper bound he reewidh of he curren model. When an edge i added o he Bayeian nework we updae hi riangulaed graph in a way ha i no only guaraneed o produce a valid riangulaion, bu ha i alo reewidh-friendly. Tha i, our updae i guaraneed o increae he ize of he maximal clique of he riangulaed graph, and hence he reewidh bound, by a mo one. An imporan propery of our edge updae i ha we can characerize he par of he nework ha are conaminaed by he new edge. Thi allow u o define e of edge ha are joinly reewidh-friendly. Building on he characerizaion of hee e, we propoe a dynamic programming approach for efficienly learning he opimal reewidh-friendly chain wih repec o a node ordering. Figure 1 how peudo-code for our mehod. Briefly, we learn a Bayeian nework wih bounded reewidh K by aring from a Chow-Liu ree and ieraively augmening he curren rucure wih an opimal reewidh-friendly chain. During each ieraion (below he reewidh bound) we apply our reewidh-friendly edge updae procedure ha mainain a moralized and riangulaed graph for he model a hand. Appealingly, a each global modificaion can increae he reewidh by a mo one, a lea K uch chain will be added before we face he problem of local maxima. In pracice, a ome chain do no increae he reewidh, many more uch chain are added for a given K. Theorem 3.1: Given a reewidh bound K and daae over N variable, he algorihm oulined in Figure 1 run in ime polynomial in N and K. Thi reul relie on he efficiency of each ep of he algorihm and ha here can be a mo N K ieraion ( edge ) before exceeding he reewidh bound. In he nex ecion we develop he edge updae and be coring chain procedure and how ha boh are polynomial in N and K. 4 Treewidh-Friendly Edge Updae The baic building block of our mehod i a procedure for mainaining a valid riangulaion of he Bayeian nework. An appealing feaure of hi procedure i ha he reewidh bound i guaraneed o grow by a mo one afer he updae. We fir conider ingle edge ( ) addiion o he model. For clariy of expoiion, we ar wih a imple varian of our procedure, and laer refine hi o allow for muliple edge addiion while mainaining our guaranee on he reewidh bound. p 1 p 1 p2 p2 3

4 To gain inuiion ino how he dynamic naure of our updae i ueful, we ue he noion of induced pah or pah wih no horcu (ee Secion 2), and make explici he following obviou fac: Obervaion 4.1: Le G be a Bayeian nework rucure and le M + be a moralized riangulaion of G. Le M ( ) be M + augmened wih he edge ( ) and wih he edge ( p) for every paren p of in G. Then, every non-chordal cycle in M ( ) involve and eiher or a paren of and an induced pah beween he wo verice. Saed imply, if he graph wa riangulaed before he addiion of ( ) o he Bayeian nework, hen we only need o riangulae cycle creaed by he addiion of he new edge or hoe forced by moralizaion. Thi obervaion immediaely ugge a raigh-forward ingle-ource riangulaion whereby we imply add an edge ( v) for every node v on an induced pah beween and or i paren before he edge updae. Clearly, hi naive mehod reul in a valid moralized riangulaion of G ( ). Surpriingly, we can alo how ha i i reewidh-friendly. Theorem 4.2: The reewidh of he graph produced by he ingle-ource riangulaion procedure i greaer han he reewidh of he inpu graph M + by a mo one. Proof: (ouline) For he reewidh o increae by more han one, ome maximal C in M + need o connec o wo new node. Since all edge are being added from, hi can only happen in one of wo way: (i) eiher, a paren p of, or a node v on induced pah beween and i alo conneced o C, bu no par of C, or (ii) wo uch (non-adjacen) node exi and i in C. In eiher cae one edge i miing afer he updae procedure prevening he formaion of a larger clique. One problem wih he propoed ingle-ource riangulaion, depie i being reewidh-friendly, i ha many verice are conneced o he ource node, making he riangulaion hallow. Thi can have an undeirable effec on fuure edge addiion and increae he chance of he formaion of large clique. We can alleviae hi problem wih a refinemen of he ingle-ource riangulaion procedure ha make ue of he concep of cu-verice, block, and block ree. Definiion 4.3: A block of an undireced graph H i a e of conneced node ha canno be diconneced by he removal of a ingle verex. By convenion, if he edge (u v) i in H hen u and v are in he ame block. Verice ha eparae (are in he inerecion of) block are called cu-verice. I i eay o ee ha beween every wo node in a block of ize greaer han wo here are a lea wo diinc pah, i.e. a cycle. There are alo no imple cycle involving node in differen block. Definiion 4.4: The (unique) block ree B of an undireced graph H i a graph wih node ha correpond boh o cu-verice and o block of H. The edge in he block ree connec any block node B i wih a cu-verex node v j if and only if v j B i in H. I can be eaily hown ha any pah in H beween wo node in differen block pae hrough all he cu-verice along he pah beween he block in B. An imporan conequence ha follow from Dirac [11] i ha an undireced graph whoe block are riangulaed i overall riangulaed. Our refined reewidh-friendly riangulaion procedure (illuraed via an example in Figure 1) make ue of hi fac a follow. Fir, he riangulaed graph i augmened wih he edge ( ) and any edge needed for moralizaion (Figure 1(b) and (c)). Second, a block level riangulaion i carried ou by zig-zagging acro cu-verice along he unique pah beween he block conaining and and i paren (Figure 1(d)). Nex, wihin each block (no conaining or ) along he pah, a ingle-ource riangulaion i performed wih repec o he enry and exi cu-verice. Thi hor-circui any oher node pah hrough (and wihin) he block. For he block conaining he ingle-ource riangulaion i performed beween and he exi cu-verex. The block conaining and i paren i reaed differenly: we add chord direcly from o any node v wihin he block ha i on an induced pah beween and (or paren of ) (Figure 1(e)). Thi i required o preven moralizaion and riangulaion edge from ineracing in a way ha will increae he reewidh by more han one (e.g., Figure 1(f)). If and happen o be in he ame block, hen we only riangulae he induced pah beween and, i.e., he la ep oulined above. Finally, in he pecial cae ha and are in diconneced componen of G, he only edge added are hoe required for moralizaion. Theorem 4.5: Our revied edge updae procedure reul in a riangulaed graph wih a reewidh a mo one greaer han ha of he inpu graph. Furhermore, i run in polynomial ime. Proof: (ouline) Fir, oberve ha he final ep of adding chord emanaing from i a ingleource riangulaion once he oher ep have been performed. Since each block along he block pah beween and i riangulaed eparaely, we only need o conider he zig-zag riangulaion beween block. A hi creae 3-cycle, he graph mu alo be riangulaed. To ee ha he reewidh 4

5 increae by a mo one, we ue imilar argumen o hoe ued in he proof of Theorem 4.2, and oberve ha he zig-zag riangulaion only ouche cu-verice and any hree of hee verice could no have been in he ame clique. The fac ha he updae procedure run in polynomial ime follow from he fac ha an adapaion (no hown for lack of pace) of maximum cardinaliy earch (ee, for example [16]) can be ued o efficienly idenify all induced node beween and. Muliple Edge Updae. We now conider he addiion of muliple edge o he graph G. To enure ha muliple edge do no inerac in way ha will increae he reewidh bound by more han one, we need o characerize he node conaminaed by each edge addiion a node v i conaminaed by he adding ( ) o G if i i inciden o a new edge added during our reewidh friendly riangulaion. Below are everal example of conaminaed e (olid node) inciden o edge added (dahed) by our edge updae procedure for differen candidae edge addiion ( ) o he Bayeian nework on he lef. In all example excep he la reewidh i increaed by one. Uing he noion of conaminaion, we can characerize e of edge ha are joinly reewidhfriendly. We will ue hi o learn opimal reewidh friendly chain given a ordering in Secion 5. Theorem 4.6: (Treewidh-friendly e). Le G be a graph rucure and M + be i correponding moralized riangulaion. If {( i i )} i a e of candidae edge aifying he following: he conaminaed e of any ( i i ) and ( j j ) are dijoin, or, he conaminaed e overlap a a ingle cu-verex, bu he endpoin of each edge are no in he ame block and he block pah beween he endpoin do no overlap; hen adding all edge o G can increae he reewidh bound by a mo one. Proof: (ouline) The heorem hold rivially for he fir condiion. Under he econd condiion, he only common verex i a cu-verex. However, ince all oher conaminaed node are in in differen block, hey canno inerac o form a large clique. 5 Learning Opimal Treewidh-Friendly Chain In he previou ecion we decribed our edge updae procedure and characerized edge chain ha joinly increae he reewidh bound by a mo one. We now ue hi o earch for opimal chain rucure ha aify Theorem 4.6, and are hu reewidh friendly, given a opological node ordering. On he urface, one migh queion he need for a pecific node ordering alogeher if chain global operaor are o be ued given he reul of Chow and Liu [5], one migh expec ha learning he opimal chain wih repec o any ordering can be carried ou efficienly. However, Meek [19] howed ha learning an opimal chain over a e of random variable i compuaionally difficul and he reul can be generalized o learning a chain condiioned he curren model. Thu, during any ieraion of our algorihm, we canno expec o find he overall opimal chain. Inead, we commi o a ingle node ordering ha i opologically conien (each node appear afer i paren in he nework) and learn he opimal reewidh-friendly chain wih repec o ha order (we briefly dicu he deail of our ordering below). To find uch a chain in polynomial ime, we ue a raighforward dynamic programming approach: he be reewidh-friendly chain ha conain (O O ) i he concaenaion of: he be chain from he fir node O 1 o O F, he fir node conaminaed by (O O ) he edge (O O ) he be chain aring from he la node conaminaed O L opimal chain opimal chain o he la node in he order O N. O 1 O F O O O L O N We noe ha when he end node are no eparaing cu-verice, we mainain a gap o ha he conaminaion e are dijoin and he condiion of Theorem 4.6 are me. 5

6 Te log-lo / inance Our Thin Juncion-ree Aggreive Treewidh bound Lengh of chain Runime in minue Our Thin Juncion ree Treewidh bound Ieraion Treewidh bound Figure 2: Gene expreion reul: (lef) 5-fold mean e log-lo per/inance v. reewidh bound. Our mehod (olid blue quare) i compared o he hin juncion ree mehod (dahed red circle), and an unbounded aggreive earch (doed black). (middle) he reewidh eimae and he number of edge in he chain during he ieraion of a ypical run wih he bound e o 10. (righ) how running ime a a funcion of he bound. Formally, we define C[i,j] a he opimal chain whoe conaminaion i limied o he range [O i,o j ] and our goal i o compue C[1,N]. Uing F o denoe he fir node ordered in he conaminaion e of ( ) (and L for he la), we can compue C[1,N] via he following recurive updae principle { max,:f=i,l=j ( ) no pli C[i,j] = max k=i+1:j 1 C[i,k] C[k,j] pli leave a gap where he maximizaion i wih repec o he rucure core (e.g., BIC). Tha i, he be chain in a ubequence [i,j] in he ordering i he maximum of hree alernaive: edge whoe conaminaion boundarie are exacly i and j (no pli); wo chain ha are joined a ome node i < k < j (pli); a gap beween i and j when here i no poiive edge whoe conaminaion i in [i,j]. Finally, for lack of pace we only provide a brief decripion of our opological node ordering. Inuiively, ince edge conaminae node along he block pah beween he edge endpoin (ee Secion 4), we wan o adop a DFS ordering over he block o a o faciliae a many edge a poible beween differen branche of he block ree. We order node wih a block by he diance from he enry verex a moivaed by he following reul on he diance d M min (u,v) beween node u,v in he riangulaed graph M + (proof no hown for lack of pace): Theorem 5.1: Le r,, be node in a block B in he riangulaed graph M + wih d M min (r,) d M min (r,). Then for any v on an induced pah beween and we have d M min (r,v) d M min (r,). The efficiency of our mehod oulined in Figure 1 in he number of variable and he reewidh bound (Theorem 3.1) now follow from he efficiency of he ordering and chain learning procedure. 6 Experimenal Evaluaion We compare our approach on four real-world daae o everal mehod. The fir i an improved varian of he hin juncion ree mehod [2]. We ar (a in our mehod) wih a Chow-Liu fore and ieraively add he ingle be coring edge a long a he reewidh bound i no exceeded. To make he comparion independen of he choice of riangulaion mehod, a each ieraion we replace he heuriic riangulaion (be of maximum cardinaliy earch or minimum fill-in [16], which in pracice had negligible difference) wih our riangulaion if i reul in a lower reewidh.the econd baeline i an aggreive rucure learning approach ha combine greedy edge modificaion wih a TABU li (e.g., [13]) and random move and ha i no conrained by a reewidh bound. Where relevan we alo compare our reul o he reul of Checheka and Guerin [3]. Gene Expreion. We fir conider a coninuou daae of he expreion of yea gene (variable) in 173 experimen (inance) [12]. We learn igmoid Bayeian nework uing he BIC rucure core [24] uing he fully oberved e of 89 gene ha paricipae in general meabolic procee. Here a learned model indicae poible regulaory or funcional connecion beween gene. Figure 2(a) how e log-lo a a funcion of reewidh bound. The fir obviou phenomenon i ha boh our mehod and he hin juncion ree approach are uperior o he aggreive baeline. A one migh expec, he aggreive baeline achieve a higher BIC core on raining daa (no hown), bu overfi due o he carciy of he daa. The conien uperioriy of our mehod over hin juncion ree demonrae ha a beer choice of edge, i.e., one choen by a global operaor, can lead o increaed robune and beer generalizaion. Indeed, even when he reewidh bound 6

7 Te log-lo / inance Aggreive Checheka+Guerin Our Thin Juncion-ree Te log-lo / inance Aggreive Our Thin Juncion-ree Checheka+Guerin Te log-lo / inance Our [unordered] Thin Juncion-ree Training inance Training inance Figure 3: 5-fold mean e log-lo/inance for a reewidh bound of wo v. raining e ize for he emperaure (lef) and raffic (righ) daae. Compared are our approach (olid blue quare), he hin juncion ree mehod (dahed red circle), an aggreive unbounded earch (doed black), and he mehod of Checheka and Guerin [3] (dah-do magena diamond) Treewidh bound Figure 4: Average log-lo v. reewidh bound for he Hapmap daa. Compared are an unbounded aggreive earch (doed) and unconrained (hin) and conrained by he DNA order (hick) varian of our and he hin juncion ree mehod. i increaed pa he auraion poin, our mehod urpae boh baeline. In hi cae, we are learning unbounded nework and all benefi come from he global naure of our updae. To qualiaively illurae he progreion of our algorihm, in Figure 2(b) we plo he number of edge in he chain and he reewidh eimae a he end of each ieraion for a ypical run. Our algorihm aggreively add muli-edge chain unil he reewidh bound i reached, a which poin (ieraion 24) i become fully greedy. To appreciae he non-rivialiy of ome of he chain learned wih 4 7 edge, we recall ha he chain are added afer a Chow-Liu model wa iniially learned. I i alo worh noing ha depie heir complexiy, ome chain do no increae he reewidh eimae and we ypically have more han K ieraion where chain wih more han one edge are added. The number of uch ieraion i ill polynomially bounded a for a Bayeian nework wih N variable adding more han K N edge will necearily reul in a reewidh ha i greaer han K. To evaluae he efficiency of our mehod we meaured i running ime a a funcion of he reewidh bound. Figure 2(c) how reul for he gene expreion daae. Oberve ha our mehod and he greedy hin juncion ree approach are boh approximaely linear in he reewidh bound. Appealingly, he addiional compuaion our mehod require i no ignifican ( 25%). Thi hould no come a a urprie a he bulk of he ime i pen on he collecion of he daa ufficien aiic. I i alo worh dicuing he range of reewidh we conidered in he above experimen a well a he Haploype experimen below. While reewidh greaer han 25 eem exceive for exac inference, ae-of-he-ar echnique (e.g., [9, 18]) can reaonably handle inference in nework of hi complexiy. Furhermore, a our reul how, i i beneficial in pracice o learn uch model. Thu, combining our mehod wih ae-of-he-ar inference echnique can allow praciioner o puh he envelope of he complexiy of model learned for real applicaion ha rely on exac inference. The Traffic and Temperaure Daae. We now compare our mehod o he muual-informaion baed LPACJT approach of Checheka and Guerin [3] (we compare o he beer varian). A heir mehod i exponenial in he reewidh and canno be ued in he gene expreion eing, we compare o i on he wo dicree real-life daae Checheka and Guerin [3] conidered: he emperaure daa i from a deploymen of 54 enor node; he raffic daae conain raffic flow informaion meaured every 5 minue in 32 locaion in California. To make he comparion fair, we ued he ame dicreizaion and rain/e pli. Furhermore, a heir mehod can only be applied o a mall reewidh bound, we alo limied our model o a reewidh of wo. Figure 3 compare he differen mehod. Boh our mehod and he hin juncion ree approach ignificanly ouperform he LPACJT on mall ample ize. Thi reul i conien wih he reul repored in Checheka and Guerin [3] and i due o he fac ha he LPACJT mehod doe no faciliae he ue of regularizaion which i crucial in he pare-daa regime. The performance of our mehod i comparable o he greedy hin juncion ree approach wih no obviou uperioriy of eiher mehod. Thi hould no come a a urprie ince he fac ha he unbounded aggreive earch i no ignificanly beer ugge ha he rong ignal in he daa can be capured raher eaily. In fac, Checheka and Guerin [3] how ha even a Chow-Liu ree doe raher well on hee daae (compare hi o he gene expreion daae where he aggreive varian wa uperior even a a reewidh of five). Haploype Sequence. Finally we conider a more difficul dicree daae of a equence of ingle nucleoide polymorphim (SNP) allele from he Human HapMap projec [6]. Our model i defined over 200 SNP (binary variable) from chromoome 22 of a European populaion coniing of 60 individual (we conidered everal differen equence along he chromoome wih imilar reul). 7

8 In hi cae, here i a naural ordering of variable ha correpond o he poiion of he SNP in he DNA equence. Figure 4 how e log-lo reul when hi ordering i enforced (hicker) and when i i no (hinner). The uperioriy of our mehod when he ordering i ued i obviou while he performance of he hin juncion ree mehod degrade. Thi can be expeced a he greedy mehod doe no make ue of a node ordering, while our mehod provide opimaliy guaranee wih repec o a variable ordering a each ieraion. Wheher conrained o he naural variable ordering or no, our mehod ulimaely alo urpae he unbounded aggreive earch. 7 Dicuion and Fuure Work In hi work we preened a novel mehod for learning Bayeian nework of bounded reewidh in ime ha i polynomial in boh he number of variable and he reewidh bound. Our mehod build on an edge updae algorihm ha dynamically mainain a valid moralized riangulaion in a way ha faciliae he addiion of chain ha are guaraneed o increae he reewidh bound by a mo one. We demonraed he effecivene of our reewidh-friendly mehod on real-life daae, and howed ha by uilizing global rucure modificaion operaor, we are able o learn beer model han compeing mehod, even when he reewidh of he model learned i no conrained. Our mehod can be viewed a a generalizaion of he work of Chow and Liu [5] ha i conrained o a chain rucure bu ha provide an opimaliy guaranee (wih repec o a node ordering) a every reewidh. In addiion, unlike he hin juncion ree approach of Bach and Jordan [2], we provide a guaranee ha our eimae of he reewidh bound will no increae by more han one a each ieraion. Furhermore, we add muliple edge a each ieraion, which in urn allow u o beer cope wih he problem of local maxima in he earch. To our knowledge, our i he fir mehod for efficienly learning Bayeian nework wih an arbirary reewidh bound ha i no fully greedy. Our mehod moivae everal exciing fuure direcion. I would be inereing o ee o wha exen we could overcome he need o commi o a pecific node ordering a each ieraion. While we provably canno conider every ordering, i may be poible o polynomially provide a reaonable approximaion. Second, i may be poible o refine our characerizaion of he conaminaion ha reul from an edge updae, which in urn may faciliae he addiion of more complex reewidhfriendly rucure a each ieraion. Finally, we are mo inereed in exploring wheher ool imilar o he one employed in hi work could be ued o dynamically updae he bounded reewidh rucure ha i he approximaing diribuion in a variaional approximae inference eing. Reference [1] P. Abbeel, D. Koller, and A. Ng. Learning facor graph in poly. ime & ample complexiy. JMLR, [2] F. Bach and M. I. Jordan. Thin juncion ree. In NIPS, [3] A. Checheka and C. Guerin. Efficien principled learning of hin juncion ree. In NIPS [4] D. Chickering. Learning Bayeian nework i NP-complee. In Learning from Daa: AI & Sa V [5] C. Chow and C. Liu. Approx. dicree dirib. wih dependence ree. IEEE Tran. on Info. Theory, [6] The Inernaional HapMap Conorium. The inernaional hapmap projec. Naure, [7] G. F. Cooper. The compuaionl complexiy of probabiliic inference uing belief nework. AI, [8] P. Dagum and M. Luby. An opimal approximaion algorihm for bayian inference. AI, [9] A. Darwiche. Recurive condiioning. Arificial Inelligence, [10] S. Dagupa. Learning polyree. In UAI, [11] G. A. Dirac. On rigid circui graph. Abhandlungen au dem Mah. Seminar der Univ. Hamburg 25, [12] A. Gach e al. Genomic expreion program in he repone of yea cell o environmenal change. Molecular Biology of he Cell, [13] F. Glover and M. Laguna. Tabu earch. In Modern Heuriic Tech. for Comb. Problem, [14] D. Heckerman. A uorial on learning Bayeian nework. Technical repor, Microof Reearch, [15] D. Karger and N. Srebro. Learning markov nework: maximum bounded ree-widh graph. In Sympoium on Dicree Algorihm, [16] A. Koer, H. Bodlaender, and S. Van Hoeel. Treewidh: Compuaional experimen. Technical repor, Univeriei Urech, [17] S. Laurizen and D. Spiegelhaler. Local compuaion wih probabiliie on graphical rucure. Journal of he Royal Saiical Sociey, [18] R. Marinecu and R. Decher. And/or branch-and-bound for graphical model. IJCAI, [19] C. Meek. Finding a pah i harder han finding a ree. Journal of Arificial Inelligence Reearch, [20] M. Meila and M. I. Jordan. Learning wih mixure of ree. JMLR, [21] M. Naraimhan and J. Bilme. Pac-learning bounded ree-widh graphical model. In UAI, [22] J. Pearl. Probabiliic Reaoning in Inelligen Syem. Morgan Kaufmann, [23] N. Roberon and P. Seymour. Graph minor II. algorihmic apec of ree-widh. J. of Algorihm, [24] G. Schwarz. Eimaing he dimenion of a model. Annal of Saiic, 6: ,