Bayesian vs. Markov Networks

Transcription

1 Bayesian vs. Markov Networks Le Song Machine Learning II: dvanced Topics CSE 8803ML, Spring 2012

2 Conditional Independence ssumptions Local Markov ssumption Global Markov ssumption 𝑋 𝑁𝑜𝑛𝑑𝑒𝑠𝑐𝑒𝑛𝑑𝑎𝑛𝑡𝑋 𝑃𝑎𝑋 𝐴 𝐵 𝐶, 𝑠𝑒𝑝𝐺 𝐴, 𝐵; 𝐶 𝑁𝑜𝑛𝑑𝑒𝑠𝑐𝑒𝑛𝑑𝑎𝑛𝑡𝑋 𝑃𝑎𝑋 𝐵𝑁 𝑀𝑁 𝑋 𝐻 𝑆 (𝐴 𝐻) 𝐴 𝐻 𝑆 𝐹 𝐴 𝑆 𝐴 𝐹 (𝐴 𝐹 𝑆) 𝑆 𝑁 𝐻 𝐴 𝐵 Derived local and pairwise assumption D-separation, active trail 𝐴 𝐶 𝑁 𝐻 𝑆 (𝑁 𝐻) 𝑋 𝑇ℎ𝑒𝑅𝑒𝑠𝑡 𝑀𝐵𝑋 𝑋 𝑌 𝑇ℎ𝑒𝑅𝑒𝑠𝑡 (no X Y) 𝐴 𝐶 𝑋 𝐵 𝐷 𝑀𝐵𝑋 = {𝐴𝐵𝐶𝐷} 2

3 Distribution Factorization Bayesian Networks (Directed Graphical Models) 𝐼 𝑚𝑎𝑝: 𝐼𝑙 𝐺 𝐼 𝑃 𝑛 𝑃(𝑋1,, 𝑋𝑛 ) = Conditional Probability Tables (CPTs) 𝑃(𝑋𝑖 𝑃𝑎𝑋𝑖 ) 𝑖=1 Markov Networks (Undirected Graphical Models) 𝑠𝑡𝑟𝑖𝑐𝑡𝑙𝑦 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑃, 𝐼 𝑚𝑎𝑝: 𝐼 𝐺 𝐼 𝑃 Clique 𝑚 Potentials 1 𝑃(𝑋1,, 𝑋𝑛 ) = Ψ𝑖 𝐷𝑖 𝑍 Maximal 𝑖=1 Normalization (Partition Function) 𝑚 𝑍 = 𝑥1,𝑥2,,𝑥𝑛 Ψ𝑖 𝐷𝑖 Clique 𝑖=1 3

4 Representation Power? 𝑃 𝑀𝑁 𝐵𝑁 convert? Minimal I-map not unique Do not always have P-map 𝑋1 𝑋1 𝑋3 𝑋2, 𝑋4 𝑋2 𝑋4 𝑋1, 𝑋3 𝑋4 𝑋2 Minimal I-map unique Do not always have P-map 𝐴 𝐹 𝑆 𝐴 𝐹 (𝐴 𝐹 𝑆) 𝑋3 4

5 Is there a BN that is a P-map for a given MN? MN for swing couple of variables does not have a P-map as BN X 1 X 1 X 1 X 2 X 4 X 2 X 4 X 2 X 4 X 3 X 3 X 3 X 1 X 3 X 2, X 4 X 2 X 4 X 1, X 3 X 1 X 3 X 2, X 4 X 2 X 4 X 1, X 3 X 1 X 3 X 2, X 4 X 2 X 4 X 1, X 3 5

6 Is there an MN that is P-map for a given BN? BN for V-structure does not have a P-map as MN F F F S S S F ( F S) F ( F S) F S F ( F S) 6

7 Conversion using Minimal I-map instead Instead of attempting P-maps between BNs and MNs, we can try minimal I-maps for conversion Recall: G is a minimal I-map for P if I G I P Removal of a single edge in G render it not an I-map Note: If G is a minimal I-map of P, G need not necessarily satisfy all conditional independence relation in P 7

8 Conversion from BN to MN MN: Markov blanket MB Xi of X i : the set of immediate neighbors of X i in the graph X i V X i MB Xi MB Xi : i Markov blanket for BN? I J B B X E X F C D H C D G MB X =, B, C, D MB X =? (X DF BCD) 8

9 Markov blanket for BN Strategy: go outward from X, try to block all active trails to X 1 2 I B J I B J 3 E X F 4 E X F H C D G H C D G MB X =, B, C, D, E, F X V X MB X MB X 9

10 Markov Blanket for BN MB X in BN is the set of nodes consisting of X s parents, X s children and other parents of X s children Moral graph M(G) of a BN G is an undirected graph that contains an undirected edge between X and Y if There is a directed edge between them in the either direction X and Y are parents of a common children Moral graph insure that MB X in the set of neighbors in undirected graph M(G) B moralize B M G G C D C D 10

11 Minimal I-map from BNs to MNs Moral graph of M G of any BN G is a minimal I-map for G Moralization turns each X, Pa X into a fully connected component CPTs associated with BN can be used as clique potentials E E B moralize B M G G C D C D The moral graph loses some independence relation eg. BN: E, B, E MN: can not read marginal independence 11

12 Perfect I-maps from BNs to MNs If BN G is already moral, then its moral graph M(G) is a perfect I-map of G Proof Sketch: I M G I G The only independence relations that are potentially lost from G to M(G) are those arising from V-structures Since G has no V-structures (already moral), no independence are lost in M G 12

13 How about d-separation? D-separation, active trail S ( H) H S H N S H N H S (N H) S F F ( F S) Judge D-separation using moral graph Let U = X, Y, Z be three disjoint sets of nodes in a BN G Let G + be the ancestral graph: the induced BN over U ancestors U Then d-sep G (X; Y Z) iff sep M(G + )(X; Y Z) G d-sep G (; B E) d-sep G (; B D, F) C E B F G D B M(G + ) C E sep M(G + )(; B E) B M(G + ) C D F sep M(G + )(; B D, F) 13

14 Why M G + works? Key: Information blocked through common children in G that are not in the conditioning variables, is simulated by ignoring all children X 1 X 1 X 1 X 2 X 4 X 2 X 4 X 2 X 4 X 3 G: X 2 X 4 X 1 X 3 M(G): (X 2 X 4 X 1 ) M(G + ): X 2 X 4 X 1 14

15 Summary: Minimal I-maps from BNs to MNs Moral Graph M(G) is minimal I-map of G If G is already moral, then M(G) is a perfect I-map of G D-sep_G(X; Y Z) sep M(G + ) (X; Y Z) 15

16 Minimal I-maps from MNs to BNs ny BN I-map for an MN must add triangulating edges into the graph Intuition: V-structures in BN introduce immoralities These immoralities were not present in a Markov networks Triangulation eliminates immoralities X 1 X 1 X 2 X 4 triangulate X 2 X 4 X 3 X 3 16

17 Chordal graphs Let X 1 X 2 X k X 1 be a loop in a graph. chord in a loop is an edge connecting non-consecutive X i and X j n undirected graph G is chordal if any loop X 1 X 2 X k X 1 for k 4 has a chord B C D E F directed graph G is chordal if its underlying undirected graph is chordal 17

18 Minimal I-maps from MNs to BNs Let H be an MN, and G be any BN minimal I-map for H. Then G can have no immoralities Intuitive reason: immoralities introduce additional independencies that are not in the original MN F Let G any BN minimal I-map for H. Then G is necessarily chordal! Because any non-triangulated loop of length at least 4 in a Bayesian network necessarily contains an immorality S F ( F S) Process of adding edges are called triangulation B D 18 C

19 Minimal I-maps from MNs to BNs Let H be a non-chordal MN. Then there is no BN G that is a perfect I-map for H Proof Sketch Minimal I-map G for H is chordal It must therefore contain additional edges no present in H Each additional edge eliminate some independence assumptions How about perfect I-maps from MNs to BNs? 19

20 Clique trees Notation Let G be a connected undirected graph. Let D 1,, D k be the set of maximal cliques in G Let T be a tree structured graph whose nodes are D 1,, D k Let D i and D j be two cliques in the graph G connected by an edge, Let S ij = D i D j be the separator set between D i and D j Let W ij = D i S ij, the residue set D 1 ={,B,C} D 1 B C S 12 =D 1 D 2 ={B,C} BC D D 2 = {B, C, D} D 2 20

21 Clique trees (cont.) tree T is a clique tree for G if Each node corresponds to a clique in G and each maximal clique in G is a node in T Each separator set S ij separates W ij and W ji Every undirected chordal graph G has a clique tree T Proof by induction (start from an triangle and add nodes) 21

22 Clique tree example BC BC D BC B C D E L BCD CD DCE CE B E CD CEL DE D L CE F DEF C F DE 22

23 Perfect I-maps from MN to BN Let H be a chordal MN. Then there existis a BN G such that I H = I G BC BC 1 D BC Proof Sketch Since H is a chordal MN, is has a clique tree Number the nodes consistent with the clique ordering -> 1, B -> 2, C-> 3 D -> 4, E-> 5 F -> 6, L -> 7 BCD CD DCE DE DEF 2 B E CD CE CEL 3 D L CE C F DE

24 Perfect I-maps from MNs to BNs (cont.) Let H be a chordal MN. Then there existis a BN G such that I H = I G Proof sketch (cont.) For each node X i, Let D k be the first clique it occurs in 1 BC BC BCD CD DCE DE DEF CE CEL Define Pa Xi = D k X i X 1,, X i 1 MN H and BN G has the same edges ll parents of each X i are in the same clique They are connected No immoralities in G B2 D4 F6 C3 E5 L7 24

25 Summary: Minimal I-maps from MNs to BNs minimal I-map BN of an MN is chordal Obtained by triangulating the MN If the MN is already chordal, there is a pefect BN I-map for the MN Obtained from the corresponding clique tree 25

26 Partially Directed cyclic Graphs lso called chain graphs (superset of MN and BN) Nodes can be disjointly partition into several chain components n edge within the same chain components must be undirected n edge between two nodes in different chain components be directed B C D E H Chain Components:, B, C, D, E, F, G, H, I F G I 26

27 MN: Gaussian Graphical Models Gaussian distribution can be represented by a fully connected graph with pairwise edge potentials over continuous variable nodes The overall exponential form is: P X 1,, X n = exp ( ij E (X i μ i )Σ 1 ij (X j μ j )) = exp ( X μ Σ 1 X μ ) lso know as Gaussian graphical models (GGM) 27

28 Sparse precision vs. sparse covariance in GGM X 1 X 2 X 3 X 4 X 5 Σ 1 = Σ = Σ 1 15 = 0 X 1 X 5 TheRest X 1 X 5 Σ 15 = 0 28

29 Summary undirected trees undirected chordal graph P BN MN Moralize BN Triangulating MN 29