1 / 30

Rotation Forest: A New Classifier Ensemble Method

Rotation Forest: A New Classifier Ensemble Method. Juan J. Rodr íguez and Ludmila I. Kuncheva. 交通大學 電子所 蕭晴駿 2007.3.7. Outline. Introduction Rotation forests Experimental results Conclusions. Outline. Introduction Rotation forests Experimental results Conclusions. Introduction(1).

dezso
Download Presentation

Rotation Forest: A New Classifier Ensemble Method

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Rotation Forest: A New Classifier Ensemble Method Juan J. Rodríguez and Ludmila I. Kuncheva 交通大學 電子所 蕭晴駿 2007.3.7

  2. Outline • Introduction • Rotation forests • Experimental results • Conclusions

  3. Outline • Introduction • Rotation forests • Experimental results • Conclusions

  4. Introduction(1) • Why classifier ensemble? combine the predictions of multiple classifiers instead of single classifier • Motivation - reduce variance: less dependent on peculiarities of a single training set - reduce bias: learn a more expressive concept class than a single classifier

  5. Introduction(2) • Key step: formation of an ensemble of diverse classifiers from a single training set • It’s necessary to modify the data set (Bagging, Boosting) or the learning method (Random Forest) to create different classifiers • Performance evaluation: diversity, accuracy

  6. Bagging(1)

  7. Bagging(2) 1. for m = 1 to M// M... number of iterations a) draw (with replacement) a bootstrap sample Smof the data b) learn a classifier Cmfrom Sm 2. for each test example a) try all classifiers Cm b) predict the class that receives the highest number of votes • Bootstrap sample - the individual classifiers have high classification accuracy - low diversity

  8. Boosting • Basic idea: - later classifiers focus on examples that were misclassified by earlier classifiers - weight the predictions of the classifiers with their error

  9. Bagging vs. Boosting • Making the classifiers diverse will reduce individual accuracy  accuracy-diversity dilemma • AdaBoost creates inaccurate classifiers by forcing them to concentrate on difficult objects and ignore the rest of the data  large diversity that boost the ensemble performance

  10. Outline • Introduction • Rotation forests • Experimental results • Conclusions

  11. Rotation Forest(1) • Rotation forest transforms the data set while preserving all information • PCA is used to transform the data - subset of the instances - subset of the classes - subset of the features: low computation, low storage

  12. Rotation Forest(2) • Base classifiers: decision tree  Forest PCA is a simple rotation of the coordinate axes  Rotation Forest

  13. Method(1) X: the objects in the training data set x = [x1, x2, …, xn]T a data point with n features N×n matrix Y = [y1, y2, …, yN]T : class label with c classes

  14. Method(2) • Given: • L : the number of classifiers in the ensemble (D1, D2, …, DL) • F : the feature set • X, Y • All classifiers can be trained in parallel

  15. Fi,1 Fi,2 … Fi,K Fi,3 Method(3) • For i = 1 … L (to construct the training set for classifier Di) F : feature set K subsets (Fi,j j=1…K) each has M = n/K features

  16. F1,1 F1,2 … F1,K F1,3 Method(3) X1,1: data set X for the features in F1,1 • For j = 1 … K Eliminate a random subset of classes Select a bootstrap sample from X1,1 to obtain X’1,1 Run PCA on X’1,1 using only M features Principal components a(1)1,1,…,a(M1)1,1

  17. Method(4) • Arrange the principal components for all j to obtain rotation matrix • Rearrange the rows of R1 so as to match the order of features in F  obtain R1a • Build classifier D1 using XR1a as a training set

  18. How It Works ? • Diversity - Each decision tree uses different set of axes. - Trees are sensitive to rotation of the axes • Accuracy - No principal components are discarded - The whole data set is used to train each classifier (with different extracted features)

  19. Outline • Introduction • Rotation forests • Experimental results • Conclusions

  20. Experimental Results(1) • Experimental settings: 1. Bagging, AdaBoost, and Random Forest were kept at their default values in WEKA 2. for Rotation Forest, M is fixed to be 3 3. all ensemble methods have the same L 4. base classifier: tree classifier J48 (WEKA) 5. database: UCI Machine Learning Repository Waikato environment for knowledge analysis

  21. Database

  22. TABLE 2 Classification Accuracy and Standard Deviation of J48 and Ensemble Methods without Pruning Experimental Results(2) 15 10-fold cross validation

  23. Experimental Results(3) 69.70% 3.03% 24.24 % 3.03% Fig. 1. Percentage diagram for the four studied ensemble methods with unpruned J48 trees.

  24. Experimental Results (4) Fig. 2. Comparison of accuracy of Rotation Forest ensemble (RF) and the best accuracy from any of a single tree, Bagging, Boosting, and Random Forest ensembles.

  25. Ei,j κ Diversity-Error Diagram • Pairwise diversity measures were chosen • Kappa(κ) evaluates the level of agreement between two classifier outputs • Diversity-error diagram - x-axis: κ for the pair - y-axis: averaged individual error of Di and Dj Ei,j=(Ei+Ej)/2 - small values of κ indicate the better diversity and small values of Ei,j indicate better accuracy

  26. Experimental Results (5) • Rotation Forest has the potential to improve on diversity significantly without compromising the individual accuracy Fig. 3. Kappa-error diagrams for the vowel-n data set.

  27. Experimental Results (6) • Rotation Forest is not as diverse as the other ensembles but clearly has the most accurate classifiers • Rotation Forest is similar to Bagging, but more accurate and diverse Fig. 4. Kappa-error diagrams for the waveform data set.

  28. Conclusions • Rotation Forest transforms the data with different axes while preserve the information completely  achieve diversity and accuracy • Rotation Forest gives a scope for ensemble methods “on the side of Bagging”

  29. References • J.J. Rodriguez, L.I Kuncheva, and C.J. Alonso, “Rotation Forest: A New Classifier Ensemble Method,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 10, pp. 1619-1630, Oct. 2006 • J.J. Rodriguez, C. J. Alonso, “Rotation-based ensembles,” Proc. Current Topics in Artificial Intelligence: 10th Conference of the Spanish Association for Artificial Intelligence, LNAI 3040, Springer, 2004, 498-506. • J. Furnkranz, “Ensemble Classifiers” (class notes)

More Related