1 / 51

Lecture 1 Slides March 28 th , 2006

University of Washington Department of Electrical Engineering EE512 Spring, 2006 Graphical Models Jeff A. Bilmes <bilmes@ee.washington.edu>. Lecture 1 Slides March 28 th , 2006. Outline of Today’s Lecture. Class overview What are graphical models Semantics of Bayesian networks.

luke
Download Presentation

Lecture 1 Slides March 28 th , 2006

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. University of WashingtonDepartment of Electrical Engineering EE512 Spring, 2006 Graphical ModelsJeff A. Bilmes <bilmes@ee.washington.edu> Lecture 1 Slides March 28th, 2006 EE512 - Graphical Models - J. Bilmes

  2. Outline of Today’s Lecture • Class overview • What are graphical models • Semantics of Bayesian networks EE512 - Graphical Models - J. Bilmes

  3. Books and Sources for Today • Jordan: Chapters 1 and 2 EE512 - Graphical Models - J. Bilmes

  4. Class Road Map • L1: Tues, 3/28: Overview, GMs, Intro BNs. • L2: Thur, 3/30 • L3: Tues, 4/4 • L4: Thur, 4/6 • L5: Tue, 4/11 • L6: Thur, 4/13 • L7: Tues, 4/18 • L8: Thur, 4/20 • L9: Tue, 4/25 • L10: Thur, 4/27 • L11: Tues, 5/2 • L12: Thur, 5/4 • L13: Tues, 5/9 • L14: Thur, 5/11 • L15: Tue, 5/16 • L16: Thur, 5/18 • L17: Tues, 5/23 • L18: Thur, 5/25 • L19: Tue, 5/30 • L20: Thur, 6/1: final presentations EE512 - Graphical Models - J. Bilmes

  5. Announcements • READING: Chapter 1,2 in Jordan’s book (pick up book from basement of communications copy center). • List handout: name, department, and email • List handout: regular makeup slot, and discussion section • Syllabus • Course web page: http://ssli.ee.washington.edu/ee512 • Goal: powerpoint slides this quarter, they will be on the web page after lecture (but not before as you are getting them hot off the press). EE512 - Graphical Models - J. Bilmes

  6. Graphical Models • A graphical model is a visual, abstract, and mathematically formal description of properties of families of probability distributions (densities, mass functions) • There are many different types of Graphical model, ex: • Bayesian Networks • Markov Random Fields • Factor Graph • Chain Graph EE512 - Graphical Models - J. Bilmes

  7. GMs cover many well-known methods Graphical Models Other Semantics Chain Graphs Causal Models Factor Graphs DGMs UGMs Dependency Networks AR Bayesian Networks ZMs FST Simple Models MRFs Mixture Models DBNs HMM LDA Decision Trees Gibbs/Boltzman Distributions Kalman Factorial HMM/Mixed Memory Markov Models PCA Segment Models BMMs EE512 - Graphical Models - J. Bilmes

  8. Graphical Models Provide: • Structure • Algorithms • Language • Approximations • Data-Bases EE512 - Graphical Models - J. Bilmes

  9. Graphical Models Provide GMs give us: • Structure: A method to explore the structure of “natural” phenomena (causal vs. correlated relations, properties of natural signals and scenes) • Algorithms: A set of algorithms that provide “efficient” probabilistic inference and statistical decision making • Language: A mathematically formal, abstract, visual language with which to efficiently discuss families of probabilistic models and their properties. EE512 - Graphical Models - J. Bilmes

  10. GMs Provide GMs give us (cont): • Approximation: Methods to explore systems of approximation and their implications. E.g., what are the consequences of a (perhaps known to be) wrong assumption? • Data-base: Provide a probabilistic “data-base” and corresponding “search algorithms” for making queries about properties in such model families. EE512 - Graphical Models - J. Bilmes

  11. GMs • There are many different types of GM. • Each GM has its semantics • A GM (under the current semantics) is really a set of constraints. The GM represents all probability distributions that obey these constraints, including those that obey additional constraints (but not including those that obey fewer constraints). • Most often, the constraints are some form of factorization property, e.g., f() factorizes (is composed of a product of factors of subsets of arguments). EE512 - Graphical Models - J. Bilmes

  12. Types of Queries • Several types of queries we may be interested in: • Compute: p(one subset of vars) • Compute: p(one subset of vars| another subset of vars) • Find the N most probable configurations of one subset of variables given assignments of values to some other sets • Q: Is one subset independent of another subset? • Q: Is one subset independent of another given a third? • How efficiently can we do this? Can this question be answered? What if it is too costly, can we approximate, and if so, how well? These are questions we will answer this term. • GMs are like a probabilistic data-base (or data structure), a system that can be queried to provide answers to these sorts of questions. EE512 - Graphical Models - J. Bilmes

  13. Example • Typical goal of pattern recognition: • training (say, EM or gradient descent), need query of form: In this form, we need to compute p(o,h) efficiently. • Bayes decision rule, need to find best class for a given unknown pattern: • but this is yet another query on a probability distribution. • We can train, and perform Bayes decision theory quickly if we can compute with probabilities quickly. Graphical models provide a way to reason about, and understand when this is possible, and if not, how to reasonably approximate. EE512 - Graphical Models - J. Bilmes

  14. Some Notation • Random variables Xi, Yi, X, Y (scalar or vector) • Distributions: • Subsets: EE512 - Graphical Models - J. Bilmes

  15. Main types of Graphical Models • Markov Random Fields • a form of undirected graphical model • relatively simple to understand their semantics • also, log-linear models, Gibbs distributions, Boltzman distributions, many “exponential models”, conditional random fields (CRFs), etc. • Bayesian networks • a form of directed graphical model • originally developed to represent a form of causality, but not ideal for that (they still represent factorization) • Semantics more interesting (but trickier) than MRFs • Factor Graphs • pure, the assembly language models for factorization properties • came out of coding theory community (LDPC, Turbo codes) EE512 - Graphical Models - J. Bilmes

  16. Main types of Graphical Models • Chain graphs: • Hybrid between Bayesian networks and MRFs • A set of clusters of undirected nodes connected as directed links • Not as widely used, but very powerful. • Ancestral graphs • we probably won’t cover these. EE512 - Graphical Models - J. Bilmes

  17. Q1 Q1 Q2 Q2 Q3 Q3 Q4 Q4 Bayesian Network Examples Mixture models Markov Chains C X EE512 - Graphical Models - J. Bilmes

  18. GMs: PCA and Factor Analysis Q PCA: Q = , R  0, A = ortho X  R FA: Q = I, R = diagonal Y Other generalizations possible E.g., Q = gen. diagonal, or capture using general A since EE512 - Graphical Models - J. Bilmes

  19. Independent Component Analysis I1 I2 The data X1:4 is explained by the two (marginally) independent causes. X1 X2 X3 X4 EE512 - Graphical Models - J. Bilmes

  20. C   X Linear Discriminant Analysis • Class conditional data has diff. mean but common covariance matrix. • Fisher’s formulation: project onto space spanned by the means. EE512 - Graphical Models - J. Bilmes

  21. C C C M M       X X X Extensions to LDA MDA HMDA HDA/QDA Heteroschedastic Discriminant Analysis/Quadratic Discriminant Analysis Both Mixture Discriminant Analsysis EE512 - Graphical Models - J. Bilmes

  22. Generalized Decision Trees I • Generalized Probabilistic • Decision Trees • Hierarchical Mixtures of Experts (Jordan) D1 D2 D3 O EE512 - Graphical Models - J. Bilmes

  23. Example: Printer Troubleshooting Dechter EE512 - Graphical Models - J. Bilmes

  24. Classifier Combination Mixture of Experts (Sum rule) Other Combination Schemes Naive Bayes (prod. rule) ... X C X1 X2 XN C1 C2 S C C X1 X2 EE512 - Graphical Models - J. Bilmes

  25. Discriminative and Generative Models Discriminative Model Generative Model C C X X EE512 - Graphical Models - J. Bilmes

  26. HMMs/Kalman Filter HMM Q1 Q2 Q3 Q4 X1 X2 X3 X4 Autoregressive HMM Q1 Q2 Q3 Q4 X1 X2 X3 X4 EE512 - Graphical Models - J. Bilmes

  27. Switching Kalman Filter Q1 Q2 Q3 Q4 S1 S2 S3 S4 X1 X2 X3 X4 EE512 - Graphical Models - J. Bilmes

  28. Factorial HMM Q’1 Q’2 Q’3 Q’4 Q1 Q2 Q3 Q4 X1 X2 X3 X4 EE512 - Graphical Models - J. Bilmes

  29. Standard Language Modeling • Example: standard 4-gram EE512 - Graphical Models - J. Bilmes

  30. Interpolated Uni-,Bi-,Tri-Grams • Nothing gets zero probability EE512 - Graphical Models - J. Bilmes

  31. Conditional mixture tri-grams EE512 - Graphical Models - J. Bilmes

  32. Bayesian Networks • … and so on • We need to be more formal about what BNs mean. • In the rest of this, and in the next, lecture, we start with basic semantics of BNs, move on to undirected models, and then come back to BNs again to clean up … EE512 - Graphical Models - J. Bilmes

  33. Bayesian Networks • Has nothing to do with “Bayesian statistical models” (there are Bayesian and non-Bayesian Bayesian networks). valid: invalid: EE512 - Graphical Models - J. Bilmes

  34. Sub-family specification: Directed acyclic graph (DAG) Nodes - random variables Edges - direct “influence” P(A | E,B) E B e b 0.9 0.1 e b 0.2 0.8 e b 0.9 0.1 0.01 0.99 e b BN: Alarm Network Example • Compact representation: factors of probabilities of children given parents (J. Pearl, 1988) Burglary Earthquake Radio Alarm Call Together (graph and inst.): Defines a unique distribution in a factored form Instantiation: Set of conditional probability distributions EE512 - Graphical Models - J. Bilmes

  35. Bayesian Networks • Reminder: chain rule of probability. For any order  EE512 - Graphical Models - J. Bilmes

  36. Bayesian Networks EE512 - Graphical Models - J. Bilmes

  37. Conditional Independence EE512 - Graphical Models - J. Bilmes

  38. Conditional Independence & BNs • GMs can help answer CI queries: A X B Z C E D No!! Y EE512 - Graphical Models - J. Bilmes

  39. Bayesian Networks & CI EE512 - Graphical Models - J. Bilmes

  40. Bayesian Networks & CI EE512 - Graphical Models - J. Bilmes

  41. Bayesian Networks & CI EE512 - Graphical Models - J. Bilmes

  42. Bayesian Networks & CI EE512 - Graphical Models - J. Bilmes

  43. Pictorial d-separation: blocked/unblocked paths Blocked Paths Unblocked Paths EE512 - Graphical Models - J. Bilmes

  44. V1 V2 V3 Three Canonical Cases • Three 3-node examples of BNs and their conditional independence statements. V3 V2 V1 V1 V2 V3 EE512 - Graphical Models - J. Bilmes

  45. Case 1 • Markov Chain EE512 - Graphical Models - J. Bilmes

  46. Case 2 • Still a Markov Chain EE512 - Graphical Models - J. Bilmes

  47. Case 3 • NOT a Markov Chain EE512 - Graphical Models - J. Bilmes

  48. Examples of the three cases SUVs Greenhouse Gasses Global Warming Lung Cancer Smoking Bad Breath Genetics Cancer Smoking EE512 - Graphical Models - J. Bilmes

  49. Bayes Ball • simple algorithm, ball bouncing along path in graph, to tell if path is blocked or not (more details in text). EE512 - Graphical Models - J. Bilmes

  50. What are implied conditional independences? EE512 - Graphical Models - J. Bilmes

More Related