Readgs: K&F 3., 3.2, 3.3, 3.4. Bayesa Network Represetato Lecture 2 Mar 30, 20 CSE 55, Statstcal Methods, Sprg 20 Istructor: Su-I Lee Uversty of Washgto, Seattle Last tme & today Last tme Probablty theory Codtoal depedece Codtoal parameterzato Today Naïve Bayes model Defto of the Bayesa etwork (BN) Idepedece propertes ecoded BN graphs From dstrbutos to BN graphs CSE 55 Statstcal Methods Sprg 20 2
Codtoal parameterzato S = SAT score, Val(S) = {s 0,s } I = Itellgece, Val(I) = { 0, } G = Grade, Val(G) = {g 0,g,g 2 } Assume that G ad S are depedet gve I S I G I,S,G) = Jot parameterzato I,S) I,S,G) I S I,S) G I,S,G) 0 s 0 0 0.665 g 0 0.425 0 s 0.035 0 s 0 s 0 g 0.06 0.25 ss 0 0.24 g 2 0.0 : : : : 3 parameters parameters Codtoal parameterzato I) S I) G I) I S G 0 I s 0 s I g 0 g g 2 0.7 0.3 0 0.95 0.05 0 0.75 0.05 0.2 0.2 0.8 0.2 0.3 0.5 3 parameters 7 parameters CSE 55 Statstcal Methods Sprg 20 3 Naïve Bayes model Class varable C, Val(C) = {c,,c k } Evdece varables,, Naïve Bayes assumpto: evdece varables are codtoally depedet gve C P ( C,,..., ) = P ( C ) P ( C ) = Applcatos medcal dagoss, text classfcato Used as a classfer: Gve {x,,x } o evdece varables,,, predct the value o C : C = c = = = x,..., x) C c) x C c ) C = c x,..., x ) C = c ) x C c ) 2 = = 2 Problem: Double coutg correlated evdece 2 CSE 55 Statstcal Methods Sprg 20 4 2
Bayesa etwork (formal) Drected acyclc graph (DAG) G Nodes represet radom varables Edges represet drect flueces betwee radom varables Local probablty models (codtoal parameterzato) Codtoal probablty dstrbutos (CPDs) Here are the etworks we have bee dscussg so far I I C S S G 2 Example Example 2 Naïve Bayes CSE 55 Statstcal Methods Sprg 20 5 Bayesa etwork structure Drected acyclc graph (DAG) G Nodes,, represet radom varables G ecodes the followg set of depedece assumptos (called, local depedeces) s depedet of ts o-descedats gve ts parets Formally: ( NoDesc( ) Pa( )) Deoted by I L (G) A B C E {A,C,D,F} B D E F G 6 3
The Studet example Course dffculty (D), Val(D) = {easy, hard} Itellgece (I), Val (I) = {hgh, low} Grade (G), Val (G) = {A, B, C} Qualty of the rec. letter (L), Val(L) = {strog, weak} SAT (S), Val (S) = {hgh, low} Graph G studet Local depedeces I L (G studet ) CSE 55 Statstcal Methods Sprg 20 7 The Studet Bayesa etwork Jot dstrbuto I,D,G,S,L) = I L (G studet ) D I D S G S D, I L I,D,S G S D,G,L I CSE 55 Statstcal Methods Sprg 20 8 4
Idepedecy mappgs (I-Maps) Let P be a dstrbuto over Let I(P) be the depedeces ( ) P A Bayesa etwork structure G s a I-map (depedecy mappg) for P, f I L (G) I(P) I I S I,S) 0 s 0 0.25 I I S I,S) 0 s 0 0.4 0 s 0.25 0 s 0.3 S s 0 0.25 s 0.25 S s 0 0.2 s 0. I L (G)={I S} I(P)={I S} I L (G)= I(P)= CSE 55 Statstcal Methods Sprg 20 9 Factorzato theorem P factorzes over G G s a I-Map of P,..., ) = Pa( )) The codtoal depedeces ecoded G mply factorzato accordg to G. =,..., ) = Pa( )) G s a I-Map of P = Factorzato accordg to G mples the assocated codtoal depedeces. CSE 55 Statstcal Methods Sprg 20 0 5
Factorzato theorem If G s a I-Map of P, the Proof: wlog.,, s a orderg cosstet wth G By cha rule:,..., ) =,..., From assumpto: Pa( {, K, Sce G s a I-Map ( ; NoDesc( ) Pa( )) I(P),..., ) = Pa( )) ) {, K, } Pa( } = ) = ) NoDesc( ) P,..., ) = ( Pa( )) Factorzato mples I-Map = G s a I-Map of P,..., ) Pa( )) = Proof: Need to show ( ; NoDesc( ) Pa( )) I(P) or that NoDesc( )) = Pa( )) wlog.,, s a orderg cosstet wth G NoDesc( )), NoDesc( )) = NoDesc( )) = k = k = = k k Pa( Pa( Pa( )) k k )) )) 2 6
Bayesa etwork defto A Bayesa etwork s a par (G,P) P factorzes over G P s specfed as set of CPDs assocated wth G s odes Parameters Jot dstrbuto: 2 Bayesa etwork (bouded -degree k): 2 k CSE 55 Statstcal Methods Sprg 20 3 Bayesa etwork desg Varable cosderatos Clarty test: ca a omscet beg determe ts value? Hdde varables? Irrelevat varables Structure cosderatos Causal order of varables Whch depedeces (approxmately) hold? Probablty cosderatos ero probabltes Orders of magtude Relatve values CSE 55 Statstcal Methods Sprg 20 4 7
Idepedeces a BN G ecodes local depedeces s depedet of ts o-descedats gve ts parets Formally: ( NoDesc( ) Pa( )) Does G ecode other depedece assumptos that hold every dstrbuto P that factorzes over G? Devse a procedure to fd all depedeces G CSE 55 Statstcal Methods Sprg 20 5 d-separato (drected separato) Goal: procedure that d-sep(;, G) Retur true ff Id(; ) follows from the local depedeces G, I L (G). Strategy: sce fluece must flow alog paths G, cosder reasog patters betwee,, ad, varous structures G Actve path: creates depedeces betwee odes Iactve path: caot create depedeces CSE 55 Statstcal Methods Sprg 20 6 8
Drect coecto ad drectly coected G o exsts for whch Id(; ) Example: determstc fucto CSE 55 Statstcal Methods Sprg 20 7 Idrect coecto ca fluece va ff s ot observed. Actve Case : Idrect causal effect Blocked Actve Case 2: Blocked Actve Commo cause Idrect evdetal effect Case 3: Blocked v-structure Actve Case 4: Commo effect Blocked 8 9
The geeral case Let G be a Bayesa etwork structure Let be a tral G Let E be a subset of evdece odes G The tral s actve gve evdece E f: ALL the three-ode etworks alog the tral s actve. For every V-structure - +, or oe of ts descedats s observed No other odes alog the tral s E CSE 55 Statstcal Methods Sprg 20 9 d-separato ad are d-separated G gve, deoted d-sep G (; ) f there s o actve tral betwee ay ode ad ay ode G I(G) = {( ) : d-sep G (; )} CSE 55 Statstcal Methods Sprg 20 20 0
Examples Are B ad C d-separated? A B C D E A B A B A C D C D C D B E d-sep(b,c)=yes E d-sep(b,c D)=o E d-sep(b,c A,D)=yes d-separato: soudess Theorem: G s a I-map of P d-sep G (; ) = yes P satsfes Id(; ) Defer proof CSE 55 Statstcal Methods Sprg 20 22
d-separato: completeess Theorem: d-sep G (; ) = o There exsts P such that G s a I-map of P P does ot satsfy Id(; ) Proof outle: Costruct dstrbuto P where depedece does ot hold Sce there s o d-sep, there s a actve path For each teracto the path, correlate the varables through the dstrbuto the CPDs Set all other CPDs to uform, esurg that fluece flows oly a sgle path ad caot be cacelled out Detaled dstrbuto costructo qute volved CSE 55 Statstcal Methods Sprg 20 23 Algorthm for d-separato Goal: aswer whether d-sep(;, G) Eumerate all possble trals betwee ad? NO Algorthm: Mark all odes or that have descedats BFS traverse G from Stop traversal at blocked odes: Node that s the mddle of a v-structure ad ot marked set Not such a ode but s If we reach ay ode the there s a actve path ad thus d-sep(;, G) does ot hold Theorem: algorthm returs all odes reachable from va trals that are actve G CSE 55 Statstcal Methods Sprg 20 24 2
I-equvalece betwee graphs I(G) descrbe all codtoal depedeces G Dfferet Bayesa etworks ca have same Id. Id(; ) Id(; ) Id(; ) Id(;) Equvalece class I Equvalece class II Two BN graphs G ad G 2 are I-equvalet f I(G ) = I(G 2 ) CSE 55 Statstcal Methods Sprg 20 25 I-equvalece betwee graphs If P factorzes over a graph a I-equvalece class P factorzes over all other graphs the same class P caot dstgush oe I-equvalet graph from aother Implcatos for structure learg We caot fd the correct structure from wth the same equvalet class. -> wll revst later. Test for I-equvalece: d-separato CSE 55 Statstcal Methods Sprg 20 26 3
Test for I-equvalece Necessary codto: same graph skeleto Otherwse, ca fd actve path oe graph but ot other But, ot suffcet: v-structures Suffcet codto: same skeleto ad v-structures But, ot ecessary: complete graphs (o depedece) Defe as mmoral f, are ot drectly coected Necessary ad Suffcet: same skeleto ad mmoral set of v-structures CSE 55 Statstcal Methods Sprg 20 27 Costructg graphs for P Ca we costruct a graph for a dstrbuto P? Ay graph whch s a I-map for P But, ths s ot so useful: complete graphs A DAG s complete f addg a edge creates cycles Complete graphs mply o depedece assumptos Thus, they are I-maps of ay dstrbuto CSE 55 Statstcal Methods Sprg 20 28 4
Mmal I-Maps A graph G s a mmal I-Map for P f: G s a I-map for P Removg ay edge from G reders t ot a I-map Example: f s a mmal I-map for P, W The:,,, s ot I-maps. W W W W CSE 55 Statstcal Methods Sprg 20 29 BayesNet defto revsted A Bayesa etwork s a par (G,P) P factorzes over G P s specfed as set of CPDs assocated wth G s odes Addtoal requremet: G s a mmal I-map for P CSE 55 Statstcal Methods Sprg 20 30 5
Costructg mmal I-Maps Reverse factorzato theorem,..., ) = Pa( )) G s a I-Map of P = Algorthm for costructg a mmal I-Map Fx a orderg of odes,, Select parets of as mmal subset of,, -, such that Id( ;, - Pa( ) Pa( )) (Outle of) Proof of mmal I-map I-map sce the factorzato above holds by costructo Mmal sce by costructo, removg oe edge destroys the factorzato CSE 55 Statstcal Methods Sprg 20 3 No-uqueess of mmal I-Map Applyg the same I-Map costructo process wth dfferet orders ca lead to dfferet structures Assume: I(G) = I(P) A B A B C D Order: E,C,D,A,B C D E E Dfferet depedece assumptos (dfferet skeletos, e.g., Id(A;B) holds o left) CSE 55 Statstcal Methods Sprg 20 32 6
Choosg order Drastc effects o complexty of mmal I-Map graph Heurstc: use causal order CSE 55 Statstcal Methods Sprg 20 33 Perfect maps G s a perfect map (P-Map) for P f I(P)=I(G) Does every dstrbuto have a P-Map? No: depedeces may be ecoded CPD Id(; =) No: some structures caot be represeted a BN Idepedeces P: Id(A;D B,C), ad Id(B;C A,D) A C B D A C D B Id(B;C A,D) does ot hold Id(A,D) also holds CSE 55 Statstcal Methods Sprg 20 34 7
Fdg a perfect map If P has a P-Map, ca we fd t? Not uquely, sce I-equvalet graphs are dstgushable Thus, represet I-equvalet graphs ad retur t Recall I-Equvalece Necessary ad Suffcet: same skeleto ad mmoral set of v-structures Fdg P-Maps Step I: Fd skeleto Step II: Fd mmoral set of v-structures Step III: Drect costraed edges CSE 55 Statstcal Methods Sprg 20 35 Summary Local depedeces I L (G) basc BN depedeces d-separato all depedeces va graph structure G s a I-Map of P f ad oly f P factorzes over G I-equvalece graphs wth detcal depedeces Mmal I-Map All dstrbutos have I-Maps (sometmes more tha oe) Mmal I-Map does ot capture all depedeces P Perfect map ot every dstrbuto P has oe Readg assgmet: K&F 3., 3.2, 3.3, 3.4 HW wll be haded out ext Moday! 8
Ackowledgemet These lecture otes were geerated based o the sldes from Prof Era Segal. CSE 55 Statstcal Methods Sprg 20 37 9