1 / 83

Using the Kinect

Using the Kinect. for fun and profit. About /me. Tam HANNA Director, Tamoggemon Holding k,s Runs web sites about mobile computing Writes scientific books. Agenda. Kinect – what is that? Streams Skeletons Facial tracking libfreenect OpenNI. Slide download.

wyoming-snider
Download Presentation

Using the Kinect

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using the Kinect for fun and profit

  2. About /me • Tam HANNA • Director, Tamoggemon Holding k,s • Runs web sites about mobile computing • Writes scientific books

  3. Agenda • Kinect – what is that? • Streams • Skeletons • Facial tracking • libfreenect • OpenNI

  4. Slide download • http://www.tamoggemon.com/test/ Codemotion-Kinect.ppt • URL IS case sensitive

  5. Kinect – what is that?

  6. History - I • Depth: PrimeSense technology • Not from Redmond • First public mention: 2007 • Bill Gates, D3 conference • „Camera for game control“

  7. Contrast detection Where does the shirt end?

  8. Dot matrix

  9. Shadows / dead areas

  10. Shadows / dead areas - II

  11. History - II • 2008: Wii ships • Best-selling console of its generation • 2009: E3 conference • Announcement of „Project Natal“ • 2010: no CPU in sensor • Takes 10% of XBox 360 CPU

  12. History - III • 4. November 2010 • First shipment • “We will sue anyone who reverse engineers“ • June 2011 • Official SDK

  13. System overview

  14. Kinect provides • Video stream • Depth stream • (IR stream) • Accelerometer data • Rest: computed

  15. Kinect for XBOX Normal USB Kinect bundle MS-Fucked USB Needs PSU Kinect for Windows Costs more Legal to deploy Family tree

  16. Cheap from China

  17. Streams

  18. Kinect provides „streams“ • Repeatedly updated bitmaps • Push or Pull processes possible • Attention: processing time!!!

  19. Color stream • Two modes • VGA@30fps • 1280x960@12fps • Simple data format • 8 bits / component • R / G / B / A components

  20. Depth stream • Two modes • Unlimited range • Reduced range, with player indexing

  21. Depth stream - II • 16bit words • Special encoding for limited range:

  22. Depth stream - III

  23. IR stream • Instead of color data • 640x480@30fps • 16bit words • IR data in 10 MSB bits

  24. Finding the Kinect • SDK supports multiple Sensors/PC • Find one • Microsoft.Kinect.Toolkit

  25. XAML part <Window x:Class="KinectWPFD2.MainWindow" xmlns:toolkit="clr-namespace:Microsoft.Kinect.Toolkit;assembly=Microsoft.Kinect.Toolkit" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" Title="MainWindow" Height="759" Width="704"> <Grid> <Image Height="480" HorizontalAlignment="Left" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="640" /> <toolkit:KinectSensorChooserUI x:Name="SensorChooserUI" IsListening="True" HorizontalAlignment="Center" VerticalAlignment="Top" /> <CheckBox Content="Overlay rendern" Height="16" HorizontalAlignment="Left" Margin="267,500,0,0" Name="ChkRender" VerticalAlignment="Top" /> </Grid> </Window>

  26. Code - I public partial class MainWindow : Window { KinectSensor mySensor; KinectSensorChooser myChooser; public MainWindow() { InitializeComponent(); myChooser = new KinectSensorChooser(); myChooser.KinectChanged += new EventHandler<KinectChangedEventArgs>(myChooser_KinectChanged); this.SensorChooserUI.KinectSensorChooser = myChooser; myChooser.Start();

  27. Code - II void myChooser_KinectChanged(object sender, KinectChangedEventArgs e) { if (null != e.OldSensor) { if (mySensor != null) { mySensor.Dispose(); } } if (null != e.NewSensor) { mySensor = e.NewSensor;

  28. Initialize stream mySensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); mySensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30); myArray = new short[this.mySensor.DepthStream.FramePixelDataLength]; myColorArray = new byte[this.mySensor.ColorStream.FramePixelDataLength]; mySensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(mySensor_AllFramesReady); try { this.mySensor.Start(); SensorChooserUI.Visibility = Visibility.Hidden; }

  29. Process stream void mySensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { ColorImageFrame c = e.OpenColorImageFrame(); DepthImageFrame d = e.OpenDepthImageFrame(); if (c == null || d == null) return; c.CopyPixelDataTo(myColorArray); d.CopyPixelDataTo(myArray);

  30. Problem: Calibration • Depth and Color sensors are not aligned • Position of data in array does not match

  31. Solution • CoordinateMapper class • Maps between various frame types • Depth and Color • Skeleton and Color

  32. On Push mode • Kinect can push data to application • Preferred mode of operation • But: sensitive to proc time • If handler takes too long -> App stops

  33. Skeletons

  34. What is tracked? • Data format • Real life coordinates • Color-Mappable

  35. Initialize stream if (null != e.NewSensor) { mySensor = e.NewSensor; mySensor.SkeletonStream.Enable();

  36. Get joints void mySensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { ColorImageFrame c = e.OpenColorImageFrame(); SkeletonFrame s = e.OpenSkeletonFrame(); if (c == null || s == null) return; c.CopyPixelDataTo(myColorArray); s.CopySkeletonDataTo(mySkeletonArray); foreach (Skeleton aSkeleton in mySkeletonArray) { DrawBone(aSkeleton.Joints[JointType.HandLeft], aSkeleton.Joints[JointType.WristLeft], armPen, drawingContext);

  37. Use joints private void DrawBone(Joint jointFrom, Joint jointTo, Pen aPen, DrawingContext aContext) { if (jointFrom.TrackingState == JointTrackingState.NotTracked || jointTo.TrackingState == JointTrackingState.NotTracked) {} if (jointFrom.TrackingState == JointTrackingState.Inferred || jointTo.TrackingState == JointTrackingState.Inferred) { ColorImagePoint p1 = mySensor.CoordinateMapper.MapSkeletonPointToColorPoint(jointFrom.Position, ColorImageFormat.RgbResolution640x480Fps30); } if (jointFrom.TrackingState == JointTrackingState.Tracked || jointTo.TrackingState == JointTrackingState.Tracked)

  38. Facial tracking

  39. What is tracked - I

  40. What is tracked - II

  41. What is tracked - III

  42. AU‘s? • Research by Paul EKMAN • Quantify facial motion

  43. Structure • C++ library with algorithms • Basic .net wrapper provided • Incomplete • Might change!!

  44. Initialize face tracker myFaceTracker = new FaceTracker(mySensor);

  45. Feed face tracker FaceTrackFrame myFrame = null; foreach (Skeleton aSkeleton in mySkeletonArray) { if (aSkeleton.TrackingState == SkeletonTrackingState.Tracked) { myFrame = myFaceTracker.Track(ColorImageFormat.RgbResolution640x480Fps30, myColorArray, DepthImageFormat.Resolution640x480Fps30, myArray, aSkeleton); if (myFrame.TrackSuccessful == true) { break; } } }

  46. Calibration • OUCH! • Not all snouts are equal • Maximums vary

  47. libfreenect

  48. What is it • Result of Kinect hacking competition • Bundled with most Linux distributions • „Basic Kinect data parser“

  49. Set-up • /etc/udev/rules.d/66-kinect.rules #Rules for Kinect ####################################################SYSFS{idVendor}=="045e", SYSFS{idProduct}=="02ae", MODE="0660",GROUP="video"SYSFS{idVendor}=="045e", SYSFS{idProduct}=="02ad", MODE="0660",GROUP="video"SYSFS{idVendor}=="045e", SYSFS{idProduct}=="02b0", MODE="0660",GROUP="video"### END #############################################################

  50. Set-up II • sudo adduser $USER plugdev • sudo usermod -a -G video tamhan • tamhan@tamhan-X360:~$ freenect-glview Kinect camera test Number of devices found: 1 Could not claim interface on camera: -6 Could not open device

More Related