Interactive 3D Modeling of Indoor Environments with a Consumer Depth Camera

1. Interactive 3D Modeling of Indoor Environments with a Consumer Depth Camera Hao Du3, Peter Henry1, Xiaofeng Ren2, Marvin Cheng1, Dan B Goldman4, Steven M. Seitz1, Dieter Fox1,2 1University of Washington: Computer Science & Engineering 2Intel Labs Seattle (now ISTC at UW) 3Google 4Adobe Systems

2. Outline • Introduce RGB-D (Red, Green, Blue + Depth) mapping with a Kinect camera • Show how an interactive mapping system improves usability and results • Demonstrate potential applications to localization, measurement, and interactive visualization

3. The Kinect • Guinness record: “fastest selling consumer device” • $150, more than 10 million sold as of March 2011 • OpenNI and Kinect SDK allow for use by developers

4. RGB-D Camera Red Green Blue Depth [PrimeSense.com]

5. RGB-D Data

6. Interactive Mapping

8. RGB-D Mapping* • RGB-D frames are recorded • The sequence is then processed to obtain the map RGB-D Frames Local Alignment (RANSAC) Update Model Global Alignment Loop Closure Detection *RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments. Henry et al. International Symposium on Experimental Robotics (ISER), 2010.

9. Frame Matching with RANSAC(Random Sample Consensus) • Visual features (from image) in 3D (from depth) • Figure out how the camera moved by matching these features to the previous frame

13. Limitations of PreviousRGB-D Mapping • Limited to offline processing after recording • Even a small number of missing alignments result in a uselessly disconnected map • Expert knowledge of the system is required to get good results • No feedback on coverage of the environment

14. Interactive 3D Mapping

15. Interactive RGB-D Mapping User Acquires Additional Frames Suggest Places to Visit User Controls Viewpoint User Moves and Resumes Mapping Visualization CompletenessAssessment RGB-D Frames RGB-D Frames Local Alignment (RANSAC) Local Alignment (RANSAC) Failure Detection Update Model Update Model Global Alignment Global Alignment Loop Closure Detection Loop Closure Detection User Verifies and Adds Loop Closures

16. User Interface

20. Examining Model Completeness

21. Completeness Hints

22. Interactive System Evaluation • 4 users (2 novice, 2 experienced) were instructed to map three walls of a room both with and without the interactive system • All experienced failures without the interactive system (after a mean count of 40.75 frames) • The proposed interactive system caught 26 potential failures allowing all users to interactively go back and create accurate maps

23. Benefits of Interactive 3D Mapping • Robustness • Real-time feedback of mapping status and failures • Rewind / Recover / Resume • Delete frames (if a person walks in front of you, etc.) • Remove false-positive loop closures • Propose additional loop closures • Completeness • See the map as it is built • Show unexplored areas

24. Large Scale Map

26. Espresso Room Model

27. Applications • Localization • Measuring Dimensions • Interactive Visualization

28. Localization

29. Localization • Centimeter level accuracy within an existing map • Provides full 3D translation and rotation • Requires no instrumentation of the environment • An alternative to RFID / 802.11 / GSM / PowerLine based localization in GPS denied environments

30. Measuring Dimensions

32. Interactive Visualization Interactive gesture controlled visualization Virtual walkthrough for education Online furniture shopping Online real estate sales Remote telepresense Virtual remodeling

33. Conclusion • Incorporating user interaction into real-time 3D mapping allows non-experts to create accurate maps using RGB-D cameras (such as the Kinect) • These maps have applications to localization, measurement, and visualization • Making this system publicly available will allow for a proliferation of 3D mapping and the potential for new context-aware applications peter@cs.washington.edu