1 / 32

Decision Tree Classification

Decision Tree Classification. Saed Sayad. Decision Tree (Mitchell 97).

lydia
Download Presentation

Decision Tree Classification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Decision Tree Classification Saed Sayad

  2. Decision Tree(Mitchell 97) • Decision tree induction is a simple but powerful learning paradigm. In this method a set of training examples is broken down into smaller and smaller subsets while at the same time an associated decision tree get incrementally developed. At the end of the learning process, a decision tree covering the training set is returned.

  3. Dataset

  4. Outlook Sunny Overcast Rain Humidity Yes Wind High Normal Strong Weak No Yes No Yes Decision Tree Attribute Node Value Node Leaf Node

  5. Frequency Tables No 5 / 14 = 0.36 Sort Yes 9 / 14 = 0.64

  6. Play Frequency Tables … Outlook | No Yes -------------------------------------------- Sunny | 3 2 -------------------------------------------- Overcast | 0 4 -------------------------------------------- Rainy | 2 3 Temp | No Yes -------------------------------------------- Hot | 2 2 -------------------------------------------- Mild | 2 4 -------------------------------------------- Cool | 1 3 Humidity | No Yes -------------------------------------------- High | 4 3 -------------------------------------------- Normal | 1 6 Windy | No Yes -------------------------------------------- False | 2 6 -------------------------------------------- True | 3 3

  7. Entropy Entropy(S) = - p log2 p - q log2 q • Entropy measures the impurity of S • S is a set of examples • p is the proportion of positive examples • q is the proportion of negative examples

  8. Entropy: One Variable Play Yes No 9 / 14 = 0.64 5 / 14 = 0.36 • Entropy(Play) = -p log2 p - q log2 q • = - (0.64 * log2 0.64) - (0.36 * log2 0.36) • = 0.94

  9. Entropy: One Variable • Example: • Entropy(5,3,2) = - (0.5 * log2 0.5) - (0.3 * log2 0.3) - (0.2 * log2 0.2) • = 1.49

  10. Entropy: Two Variables Play Outlook | No Yes -------------------------------------------- Sunny | 3 2 |5 -------------------------------------------- Overcast | 0 4 |4 -------------------------------------------- Rainy | 2 3 |5 -------------------------------------------- | 14 Size of the subset Size of the set E (Play,Outlook) = (5/14)*0.971 + (4/14)*0.0 + (5/14)*0.971 = 0.693

  11. Information Gain Gain(S, A) = E(S) – E(S, A) Example: Gain(Play,Outlook) = 0.940 – 0.693 = 0.247

  12. Selecting The Root Node Play=[9+,5-] E=0.940 Outlook Sunny Overcast Rain [2+, 3-] [4+, 0-] [3+, 2-] E=0.971 E=0.971 E=0.0 Gain(Play,Outlook) = 0.940 – ((5/14)*0.971 + (4/14)*0.0 + (5/14)*0.971) = 0.247

  13. Selecting The Root Node … Play=[9+,5-] E=0.940 Temp Hot Mild Cool [3+, 1-] [2+, 2-] [4+, 2-] E=0.811 E=1.0 E=0.918 Gain(Play,Temp) = 0.940 – ((4/14)*1.0 + (6/14)*0.918 + (4/14)*0.811) = 0.029

  14. Selecting The Root Node … Play=[9+,5-] E=0.940 Humidity High Normal [3+, 4-] [6+, 1-] E=0.592 E=0.985 Gain(Play,Humidity) = 0.940 – ((7/14)*0.985 + (7/14)*0.592) = 0.152

  15. Selecting The Root Node … Play=[9+,5-] E=0.940 Windy Weak Strong [6+, 2-] [3+, 3-] E=0.811 E=1.0 Gain(Play,Wind) = 0.940 – ((8/14)*0.811 + (6/14)*1.0) = 0.048

  16. Selecting The Root Node … Play Outlook Gain=0.247 Temperature Gain=0.029 Humidity Gain=0.152 Windy Gain=0.048

  17. Outlook Sunny Overcast Rain Humidity Yes Wind High Normal Strong Weak No Yes No Yes Decision Tree - Classification Attribute Node Value Node Leaf Node

  18. ID3 Algorithm • A  the “best” decision attribute for next node • Assign A as decision attribute for node • 3. For each value of A create new descendant • Sort training examples to leaf node according to the attribute value of the branch • If all training examples are perfectly classified (same value of target attribute) stop, else iterate over newleaf nodes.

  19. Outlook Sunny Overcast Rain Humidity Yes Wind High Normal Strong Weak No Yes No Yes Converting Tree to Rules R1: IF (Outlook=Sunny) AND (Humidity=High) THEN Play=No R2: IF (Outlook=Sunny) AND (Humidity=Normal) THEN Play=Yes R3: IF (Outlook=Overcast) THEN Play=Yes R4: IF (Outlook=Rain) AND (Wind=Strong) THEN Play=No R5: IF (Outlook=Rain) AND (Wind=Weak) THEN Play=Yes

  20. Decision Tree - Issues • Working with Continuous Attributes – Discretization • Overfitting and Pruning • Super Attributes; attributes with many values • Working with Missing Values • Attributes with Different Costs

  21. Discretization • Equally probable intervals This strategy creates a set of N intervals with the same number of elements. • Equal width intervals The original range of values is divided into N intervals with the same range. • Entropy based For each numeric attribute, instances are sorted and, for each possible threshold, a binary <, >= test is considered and evaluated in exactly the same way that a categorical attribute would be.

  22. Avoid Overfitting Overfitting when our learning algorithm continues develop hypotheses that reduce training set error at the cost of an increased test set error. • Stop growing when data split not statistically significant (Chi2 test) • Grow full tree then post-prune • Minimum description length (MDL): • Minimize: • size(tree) + size(misclassifications(tree))

  23. Post-pruning • First, build full tree • Then, prune it • Fully-grown tree shows all attribute interactions • Problem: some subtrees might be due to chance effects • Two pruning operations: • Subtree replacement • Subtree raising • Possible strategies: • error estimation • significance testing • MDL principle witten & eibe

  24. Subtree Replacement witten & eibe

  25. Error Estimation • Transformed value for f :(i.e. subtract the mean and divide by the standard deviation) • Resulting equation: • Solving for p: witten & eibe

  26. Error Estimation • Error estimate for subtree is weighted sum of error estimates for all its leaves • Error estimate for a node (upper bound): • If c = 25% then z = 0.69 (from normal distribution) • f is the error on the training data • N is the number of instances covered by the leaf witten & eibe

  27. f = 5/14 e = 0.46e < 0.51so prune! f=0.33 e=0.47 f=0.5 e=0.72 f=0.33 e=0.47 Combined using ratios 6:2:6 gives 0.51 witten & eibe

  28. Super Attributes • The information gain equation, G(S,A) is biased toward attributes that have a large number of values over attributes that have a smaller number of values. • Theses ‘Super Attributes’ will easily be selected as the root, result in a broad tree that classifies perfectly but performs poorly on unseen instances. • We can penalize attributes with large numbers of values by using an alternative method for attribute selection, referred to as GainRatio. GainRatio(S,A) = Gain(S,A) / SplitInformation(S,A)

  29. Super Attributes … Outlook | No Yes -------------------------------------------- Sunny | 3 2 |5 -------------------------------------------- Overcast | 0 4 |4 -------------------------------------------- Rainy | 2 3 |5 -------------------------------------------- | 14 Split (Play,Outlook)= - (5/14*log2(5/14)+4/14*log2(4/15)+5/14*log2(5/14)) = 1.577 Gain (Play,Outlook) = 0.247 Gain Ratio (Play,Outlook) = 0.247/1.577 = 0.156

  30. Missing Values • Most common value • Most common value at node K • Mean or Median • Nearest Neighbor • …

  31. Attributes with Different Costs • Sometimes the best attribute for splitting the training elements is very costly. In order to make the overall decision process more cost effective we may wish to penalize the information gain of an attribute by its cost.

  32. Decision Trees: • are simple, quick and robust • are non-parametric • can handle complex datasets • can use any combination of categorical and continuous variables and missing values • are not incremental and adaptive • sometimes are not easy to be read

More Related