Uncertainty in Expert Systems

1. Uncertainty in Expert Systems CPS 4801

2. Uncertainty • Uncertainty is the lack of exact knowledge that would enable us to reach a fully reliable solution. • Classical logic assumes perfect knowledge exists: IF A is true THEN B is true • Describing uncertainty: • If A is true, then B is true with probability P

3. Sources of uncertainty • Weak implications: Want to be able to capture associations and correlations, not just cause and effect. • Imprecise language: • How often is “sometimes”? • Can we quantify “often,” “sometimes,” “always?” • Unknown data: In real problems, data is often incomplete or missing. • Differing experts: Experts often disagree, or have different reasons for agreeing. • Solution: attach weight to each expert

5. Two approaches • Bayesian reasoning • Bayesian rule (Bayes’ rule) by Thomas Bayes • Bayesian network (Bayes network) • Certainty factors

6. Probability Theory P(success) + P(failure) = 1 • The probability of an event is the proportion of cases in which the event occurs • Numerically ranges from zero to unity (an absolute certainty) (i.e. 0 to 1)

7. Example • Flip a coin • P(head) = ½ P(tail) = ? • P(head) = ¼ P(tail) = ? • Throw a dice • P(getting a 6) = ? • P(not getting a 6) = ? • P(A) = p  P(¬A) = 1-p

8. Example • P(head) = ½ P(head head head) = ? • Xi = result of i-th coin flip Xi = {head, tail} • P(X1 = X2 = X3 = X4) = ? • Until now, events are independent and mutually exclusive. • P(X,Y) = P(X)P(Y) (P(X,Y) is joint probability.)

9. Example • P( {X1 X2 X3 X4} contains >= 3 head ) = ?

10. Conditional Probability probability of A given B • Suppose events A and B are not mutually exclusive, but occur conditionally on the occurrence of the other • The probability that event A will occur if event B occurs is called the conditional probability

11. Conditional Probability • The probability that bothA and B occur is called the joint probability of A and B, written p(A∩B)

12. Conditional Probability • Similarly, the conditional probability that event B will occur if event A occurs can be written as:

13. Conditional Probability

14. The Bayesian Rule • The Bayesian rule (named after Thomas Bayes, an 18th-century British mathematician):

15. Applying Bayes’ rule • A = disease, B = symptom • P(disease|symptom) = P(symptom|disease) * P(disease) / P(symptom)

16. Applying Bayes’ rule • A doctor knows that the disease meningitis causes the patient to have a stiff neck for 70% of the time. • The probability that a patient has meningitis is 1/50,000. • The probability that any patient has a stiff neck is 1%. • P(s|m) = 0.7 • P(m) = 1/50000 • P(s) = 0.01

17. Applying Bayes’ rule • P(s|m) = 0.7 • P(m) = 1/50000 • P(s) = 0.01 • P(m|s) = P(s|m) * P(m) / P(s) • = 0.7 * 1/50000 / 0.01 • = 0.0014 • = around 1/714 • Conclusion: Less than 1 in 700 patients with a stiff neck have meningitis.

18. Example: Coin Flip • P(X1 = H) = ½ 1) X1 is H: P(X2 = H | X1 = H) = 0.9 2) X1 is T: P(X2 = T | X1 = T ) = 0.8 P(X2 = H) = ?

19. What we learned from the example? • If event A depends on exactly two mutually exclusive events, B and ¬B, we obtain: • P(¬X|Y) = 1 – P(X|Y) • P(X|¬Y) = 1 – P(X|Y)?

20. Conditional probability • If event A depends on exactly two mutually exclusive events, B and ¬B, we obtain: • Similarly, if event B depends on exactly two mutually exclusive events, A and ¬A, we obtain:

21. The Bayesian Rule • Substituting p(B) into the Bayesian rule yields:

22. Bayesian reasoning • Instead of A and B, consider H (a hypothesis) and E (evidence for that hypothesis). • Expert systems use the Bayesian rule to rank potentially true hypotheses based on evidences

23. Bayesian reasoning • If event E occurs, then the probability thatevent H will occur is p(H|E) IF E (evidence) is true THEN H (hypothesis) is true with probability p

24. Bayesian reasoning Example: Cancer and Test • P(C) = 0.01 P(¬C) = 0.99 • P(+|C) = 0.9 P(-|C) = 0.1 • P(+|¬C) = 0.2 P(-|¬C) = 0.8 • P(C|+) = ?

25. Simple Bayes Network from Example

26. Bayesian reasoning • Expert identifies prior probabilities forhypotheses p(H) and p(¬H) • Expert identifies conditional probabilities for: • p(E|H): Observing evidence E if hypothesis H is true • p(E|¬H): Observing evidence E if hypothesis H is false

27. Bayesian reasoning • Experts provide p(H), p(¬H), p(E|H), and p(E|¬H) • Users describe observed evidence E • Expert system calculates p(H|E) using Bayesian rule • p(H|E) is the posterior probability that hypothesis H occurs upon observing evidence E • What about multiple hypotheses and evidences?

28. Bayesian reasoning with multiple hypotheses p(A)

29. Bayesian reasoning with multiple hypotheses • Expand the Bayesian rule to work with multiple hypotheses (H1...Hm)

30. Bayesian reasoning with multiple hypotheses and evidences • Expand the Bayesian rule to work with multiple hypotheses (H1...Hm) and evidences (E1...En)

31. Bayesian reasoning with multiple hypotheses and evidences • Expand the Bayesian rule to work with multiple hypotheses (H1...Hm) and evidences (E1...En) Assuming conditional independence among evidences E1...En

32. Summary

33. Bayesian reasoning Example • Expert is given three conditionally independent evidences E1, E2, and E3 • Expert creates three mutually exclusive and exhaustive hypotheses H1, H2, and H3 • Expert provides prior probabilities p(H1), p(H2), p(H3) • Expert identifies conditional probabilities for observing each evidence Ei for all possible hypotheses Hk

34. Bayesian reasoning Example • Expert data:

35. user observes E3

36. Bayesian reasoning Example user observes E3 expert system computes posterior probabilities

37. user observes E3E1

38. Bayesian reasoning Example user observes E1 expert system computes posterior probabilities

39. user observes E3E1 E2

40. Bayesian reasoning Example expert system computes posterior probabilities user observes E2

41. Bayesian reasoning Example • Initial expert-based ranking: • p(H1) = 0.40; p(H2) = 0.35; p(H3) = 0.25 • Expert system ranking after observing E1, E2, E3: • p(H1) = 0.45; p(H2) = 0.0; p(H3) = 0.55

42. Problems with the Bayesianapproach • Humans are not very good at estimating probability! • In particular, we tend to make different assumptions when calculating prior and conditional probabilities • Reliable and complete statistical information often not available. • Bayesian approach requires evidences to be conditionally independent – often not the case. • One solution: certainty factors