Junction Tree Algorithm

1. Junction Tree Algorithm Brookes Vision Reading Group

2. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

3. Don’t we all know … • P(A) = 1 , if and only if A is certain • P(A or B) = P(A)+P(B) if and only if A and B are mutually exclusive • P(A,B) = P(A|B)P(B) = P(B|A)P(A) • Conditional Independence • A is conditionally independent of C given B • P(A|B,C) = P(A|B)

4. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

5. Graphical Models Compact graphical representation of joint probability. A A ‘causes’ B B P(A,B) = P(A)P(B|A)

6. Graphical Models Compact graphical representation of joint probability. A B ‘causes’ A B P(A,B) = P(B)P(A|B)

7. Graphical Models Compact graphical representation of joint probability. A B P(A,B)

8. A Simple Example P(A,B,C) = P(A)P(B,C | A) = P(A) P(B|A) P(C|B,A) = P(A) P(B|A) P(C|B) C is conditionally independent of A given B Graphical Representation ???

9. A B C Bayesian Network Directed Graphical Model P(U) =  P(Vi | Pa(Vi)) P(A,B,C) = P(A) P(B | A) P(C | B)

10. A B C Markov Random Fields Undirected Graphical Model

11. Markov Random Fields Undirected Graphical Model AB BC B Clique Separator Clique P(U) =  P(Clique) /  P(Separator) P(A,B,C) = P(A,B) P(B,C) / P(B)

12. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

13. Bayesian Networks • A is conditionally independent of B given C • Bayes ball cannot reach A from B

14. Markov Random Fields • A, B, C - (set of) nodes • C is conditionally independent of A given B • All paths from A to C go through B

15. Markov Random Fields

16. Markov Random Fields A node is conditionally independent of all others given its neighbours.

17. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

18. MAP Estimation c*,s*,r*,w* = argmax P(C=c,S=s,R=r,W=w)

19. Computing Marginals P(W=w) = c,s,r P(C=c,S=s,R=r,W=w)

20. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

21. Aim • To perform exact inference efficiently • Transform the graph into an appropriate data structure • Ensure joint probability remains the same • Ensure exact marginals can be computed

22. Junction Tree Algorithm • Converts Bayes Net into an undirected tree • Joint probability remains unchanged • Exact marginals can be computed • Why ??? • Uniform treatment of Bayes Net and MRF • Efficient inference is possible for undirected trees

23. Junction Tree Algorithm • Converts Bayes Net into an undirected tree • Joint probability remains unchanged • Exact marginals can be computed • Why ??? • Uniform treatment of Bayes Net and MRF • Efficient inference is possible for undirected trees

24. Let us recap .. Shall we A B C D P(U) =  P(Vi | Pa(Vi)) =  a(Vi , Pa(Vi)) Potential Lets convert this to an undirected graphical model

25. The cliques of this graph are inconsistent with the original one. Node D just lost a parent. Let us recap .. Shall we A B C D Wait a second …something is wrong here.

26. Marry the parents for a happy family Solution A B C D Ensure that a node and its parents are part of the same clique Now you can make the graph undirected

27. Solution A B C D But we have added extra edges, haven’t we ??? A few conditional independences are lost.

28. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

29. Moralizing a graph • Marry all unconnected parents • Drop the edge directions Ensure joint probability remains the same.

30. Moralizing a graph

31. Clique Potentials a(Ci) Separator Potentials a(Si) Moralizing a graph C S R Initialize a(Ci) = 1 a(Si) = 1 W CSR SR SRW

32. Choose one node Vi Find one clique Ci containing Vi and Pa(Vi) Moralizing a graph C S R W Multiply a(Vi,Pa(Vi)) to a(Ci) CSR Repeat for all Vi SR SRW

33. Moralizing a graph C Choose one node Vi S R Find one clique Ci containing Vi and Pa(Vi) W Multiply a(Vi,Pa(Vi)) to a(Ci) CSR Repeat for all Vi SR SRW

34. Moralizing a graph C Choose one node Vi S R Find one clique Ci containing Vi and Pa(Vi) W Multiply a(Vi,Pa(Vi)) to a(Ci) CSR Repeat for all Vi SR SRW

35. Moralizing a graph C Choose one node Vi S R Find one clique Ci containing Vi and Pa(Vi) W Multiply a(Vi,Pa(Vi)) to a(Ci) CSR Repeat for all Vi SR SRW

36. Moralizing a graph P(U) =  a(Ci) /  a(Si) Now we can form a tree with all the cliques we chose. That was easy. We’re ready to marginalize. OR ARE WE ???

37. A few more examples … A B A B C D C D

38. A few more examples … A B A B C D C D

39. Inconsistency in C A few more examples … AB BD AB BCD Clearly we’re missing something here AC CD

40. Outline • Graphical Models • What are Graphical Models ? • Conditional Independence • Inference • Junction Tree Algorithm • Moralizing a graph • Junction Tree Property • Creating a junction tree • Inference using junction tree algorithm

41. Junction Tree Property In a junction tree, all cliques in the unique path between cliques Ci and Cj must contain CiCj So what we want is a junction tree, right ??? Q. Do all graphs have a junction tree ??? A. NO

42. Decomposable Graphs Decomposition (A,B,C) A C B Undirected graph G = (V,E) • V = A  B  C • All paths between A and B go through C • C is a complete subset of V

43. Decomposable Graphs • A, B and/or C can be empty • A, B are non-empty in a proper decomposition

44. Decomposable Graphs • G is decomposable if and only if • G is complete OR • It possesses a proper decomposition (A,B,C) such that • GAC is decomposable • GBC is decomposable

45. Decomposable Graphs A B A B C D C D Decomposable Not Decomposable

46. Decomposable Graphs A A B C B C D E D E Decomposable Not Decomposable

47. An Important Theorem Theorem: A graph G has a junction tree if and only if it is decomposable. Proof on white board.

48. OK. So how do I convert my graph into a decomposable one.

49. Time for more definitions • Chord of a cycle • An edge between two non-successive nodes • Chordless cycle • A cycle with no chords • Triangulated graph • A graph with no chordless cycles

50. Another Important Theorem Theorem: A graph G is decomposable if and only if it is triangulated. Proof on white board. Alright. So add edges to triangulate the graph.