1 / 27

A Robust Method of Detecting Hand Gestures Using Depth Sensors

A Robust Method of Detecting Hand Gestures Using Depth Sensors. Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on. Outline. Introduction Related Works The Proposed Method Experimental Results

ona
Download Presentation

A Robust Method of Detecting Hand Gestures Using Depth Sensors

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Robust Method of Detecting Hand Gestures Using Depth Sensors Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang Haptic Audio Visual Environments and Games (HAVE), 2012 IEEE International Workshop on

  2. Outline • Introduction • Related Works • The Proposed Method • Experimental Results • Conclusion

  3. Introduction

  4. Introduction • In human-computer interaction(HCI) system, recognizing hand and finger gestures are significant. • Medical system, computer games, and human-robot • Depth-sensing camera(Kinect, Xtion) add a dimension to increase accuracy. • Goal: detect hand gestures with color and depth information

  5. Related Works

  6. RelatedWorks • Body[4][5] V.S. Hand • Hand • Superiority: simple • Inferiority: small scale, low resolution • Strict condition: cluttered background, lighting variation

  7. RelatedWorks • Hand gesture recognition • Onlycolor[12] • Data glove[7] • Trainingprocess[9][10] • EarthMover’sDistance(EMD)[11]

  8. References • [7] R. M. Satava, “Virtual reality surgical simulator,” Surgical Endoscopy,vol. 7, pp. 203–205, 1993. • [9] C. Keskin, F. Kirac, Y. Kara, and L. Akarun, “Real time hand poseestimation using depth sensors,” in Computer Vision Workshops (ICCVWorkshops), 2011 IEEE International Conference on, nov.2011. • [10] P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, and V. Athitsos, “Comparing gesture recognition accuracy using color and depth information,” in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments, ser. PETRA ’11. New York, NY, USA: ACM, 2011, pp. 20:1–20:7. • [11] Z. Ren, J. Yuan, and Z. Zhang, “Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera,” in Proceedings of the 19th ACM international conference on Multimedia, ser. MM ’11. New York, NY, USA: ACM, 2011, pp. 1093–1096. • [12] A. Argyros and M. Lourakis, “Real-time tracking of multiple skincoloredobjects with a possibly moving camera,” in Computer Vision -ECCV 2004, ser. Lecture Notes in Computer Science. Springer Berlin/ Heidelberg, 2004, vol. 3023, pp. 368–379.

  9. The Proposed Method

  10. The Proposed Method

  11. Hand Segmentation (I) • Find hands through color • Train skin-color[12], detect face[15], image filtering[16], color threshold • L*a*b color space • b • Operate AND on the two images • Minimum depth (10cm)

  12. Hand Segmentation (I)--Find hands through color RGB image Depth image L*a*b color space where b = 3 Skin color images after AND operation L*a*b color space where b = 2 Binary image of hand segmentation

  13. Hand Segmentation (II) • Separate hands by k-means • k=2 • Assignment: • Update: • Threshold of distance between 2 clusters

  14. Hand Segmentation (II)--Separate Hands By K-means

  15. Hand Segmentation (III) • Find palm center • Inscribed circle • Minimum inner distance • Maximum element of inner distances set

  16. Finger Recognition (I) • Find convex hull • Graham’s scan algorithm • P: the lowest y-coordinate • Sort in increasing order of angle • Point to point is left/right turn • Left-turn: O ; right-turn: X

  17. Finger Recognition (II) • Detect fingertip and direction • Fingers are long and narrow • Find an isosceles triangle with V • V: Every vertex on the convex hull • Set a maximum threshold to the vertex angle • The direction vector is paralleled with the median length of an isoscelestriangle

  18. Blue point : cluster centroid Green point : palm center Red points : fingertips Yellow curves : hand contour Long lines : finger directions Structures around the hand : convex hull Finger Recognition

  19. The Proposed Method

  20. Gesture Representation • All information about hands • Palm center location • Finger number • Fingertips location • Finger direction vectors • Gestures • Rock-paper-scissors game • Drag images • Grasping, releasing

  21. Experimental Results

  22. Experimental Results • Use Kinect as input of depth and color images • The detection successful rate can reach 95%. • No matter the hand is horizontally or vertically placed.

  23. Experimental Results

  24. Experimental Results--Shadow Puppetry

  25. Conclusion

  26. Conclusion • Present a new method to detect hands’ positions and gestures • NO training, NO database • Future works • Set a threshold to the distance between the palm center and the fingers • Add additional sensor devices to overcome no palm detection • Shadow Puppetry project

More Related