1 / 26

Model-based Image Interpretation with Application to Facial Expression Recognition

Matthias Wimmer matthias.wimmer@cs.tum.edu. Model-based Image Interpretation with Application to Facial Expression Recognition. Communication Schemes. Natural human-computer interaction. Example 1: Nissan Pivo 2. Example 2: Sony ’s Smile Shutter. Outline of this Presentation.

matty
Download Presentation

Model-based Image Interpretation with Application to Facial Expression Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Matthias Wimmer matthias.wimmer@cs.tum.edu Model-based Image Interpretation with Application toFacial Expression Recognition

  2. Communication Schemes Natural human-computer interaction

  3. Example 1: Nissan Pivo2

  4. Example 2: Sony’s Smile Shutter

  5. Outline of this Presentation facial expression Facial Expression Recognition Model-based image interpretation Adaptive skin color extraction image

  6. Facial Expression Recognition

  7. What are Facial Expressions? • Six universal facial expressions (Ekman et al.) • Laughing, surprised, afraid, disgusted, sad, angry • Cohn-Kanade-Facial-Expression database • Performed • Exaggerated • Determined by • Shape • Muscle motion

  8. Why are they difficult to estimate? • Faces look differently • Hair, beard, skin-color, … • Different facial poses • Only slight muscle activity

  9. Our Approach motion features and structural features

  10. Model Fitting with Learned Objective Functions

  11. Model-based image interpretation • The model The model contains a parameter vector that represents the model’s configuration. • The objective functionCalculates a value that indicates how accurately a parameterized model matches an image. • The fitting algorithmSearches for the model parameters that describe the image best, i.e. it minimizes the objective function.

  12. Local Objective Functions

  13. Ideal Objective Functions P1: Correctness property:Global minimum corresponds to the best fit. P2: Uni-modality property:The objective function has no local extrema. ¬ P1 P1 ¬P2 P2 • Don’t exist for real-world images • Only for annotated images: fn( I , x ) = | cn – x |

  14. Learning the Objective Function • Ideal objective function generates training data • Machine Learning technique generates calculation rules x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

  15. Benefits of the Machine Learning Approach • Accurate and robust calculation rules • Locally customized calculation rules • Generalization from many images • Simple job for the designer • Critical decisions are automated • No domain-dependent knowledge required • No loops

  16. Evaluation: Fitting Accuracy

  17. Adaptive Skin Color Classification

  18. Basics about Skin Color Classification • Skin color depends on several image conditions • Skin color occupies a large cluster • Skin color varies greatly within a set of images. • Skin color varies slightly within one image. image 1 image 2 green green image 2 red red

  19. Our Approach • Learn image-specific skin color characteristics • Parameterize a skin color classifier accordingly Offline: • Learn the skin color mask • Specific for the face detector Online: • Detect the image specific skin color model • Using the face detector • Using the skin color mask • Adapt skin color classifier

  20. Results • Robustness: • Detection of facial parts:eyes, lips, brows,… • Exact shape outline • Ethnic groups original image fixed classifier adapted classifier Correctly detected pixels: • fixed classifier: 90.4% 74.8% 40.2% • adapted classifier: 97.5% 87.5% 97.0% • improvement: 1.08 1.17 2.41

  21. Additional Work lip classifier eye brow classifier iris classifier tooth classifier

  22. Conclusion and Outlook

  23. Conclusion • Possible to derive information from face images • Model-based image interpretation is beneficial • Learn crucial decisions within algorithms • Don’t specify parameters by trial and error • Adaptive skin color classifier • Learned objective functions • Not yet reached goal for natural HCI • Progress is clearly visible. → Goal is achievable!

  24. Outlook • Learn global objective function • Learn discriminative function (direct parameter update) • Rendered AAM provides training images • Many images • Exact ground truth (no manual work required) • Learn with further features • Higher number of features • SIFT, LBP, … • Learn with better classifiers • Relevance Vector Machines • Boosted regressors

  25. Thank you! Online-Demonstration: http://www9.cs.tum.edu/people/wimmerm

  26. Publications 2008 • Tailoring Model-based Techniques for Facial Expression Interpretation. ACHI08 • Face Model Fitting with Generic, Group-specific, and Person-specific Objective Functions. VISAPP • Low-level Fusion of Audio and Video Feature for Multi-modal Emotion Recognition. VISAPP • Facial Expression Recognition for Human-robot Interaction - A Prototype. Robot Vision 2007 • Audiovisual Behavior Modeling by Combined Feature Spaces. ICASSP • Emotionale Aspekte in Produktevaluationen. Multimediatechnik • Application of emotion recognition methods in automotive research. Emotion and Computing • Human Capabilities on Video-based Facial Expression Recognition. Emotion and Computing • SIPBILD - Mimik- und Gestikerkennung in der Mensch-Maschine-Schnittstelle. INFORMATIK • Learning Robust Objective Functions with Application to Face Model Fitting. DAGM • Automatically Learning the Objective Function for Model Fitting. MIRU • Initial Pose Estimation for 3D Models Using Learned Objective Functions. ACCV • Estimating Natural Activity by Fitting 3D Models via Learned Objective Functions. VMV • Learning Local Objective Functions for Robust Face Model Fitting. PAMI (journal paper) • Enabling Users to Guide the Design of Robust Model Fitting Algorithms. ICV 2006 • Learning Robust Objective Functions for Model Fitting in Image Understanding Applications. BMVC • A Person and Context Specific Approach for Skin Color Classification. ICPR • Adaptive Skin Color Classificator. Journal on Graphics, Vision and Image Processing (journal paper) • Bitte recht freundlich. Zukunft im Brennpunkt (journal paper) 2005 • Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments. INFORMATIK • Adaptive Skin Color Classificator. GVIP 2004 • Experiences with an Emotional Sales Agent. Affective Dialogue Systems

More Related