Image Based Modeling And Photo Editing

1. Image Based Modeling And Photo Editing Byong Mok Oh Max Chen Julie Dorsey Frédo Durand

2. Photorealistic Images • Hundreds of thousands of polygons! • Painstaking detail for the artist to create. • Something’s not quite convincing.

3. Use a Photo! • Already photorealistic. • Detail exists before the artist begins to • work.

4. How'd They Do That?

5. How'd They Do That? Using two proprietary tools, The Clone Brushing Tool & The Texture-Illuminance Decoupling Filter, a single image can be transformed into a three dimensional scene. • Just follow these simple steps! • Separate the source image into layers. • Assign depth. • Adjust lighting.

6. "That's Deep, Man!"

7. Layers Layering is over 75% of the work!

8. Depth Assignment A reference location is set, and each layer of depth is used to create perspective.

9. Artists can manually adjust • depth by, “chiseling”. • Depth paint is blended for • smooth transitions. Depth Painting

10. Surface Depth • To ensure accuracy, a planar surface is • used as a reference point. • A map is generated to represent the • surface depth. • All objects’ depth will be based on this • surface.

11. Primitive Geometry To create depth in simple objects, spheres, cylinders and cubes are used mask depth.

12. Organic Geometry For more organic looking objects, level sets are used. Level sets break an object into levels from the surface inwards, allowing the object to “bulge” out around the edges, creating fuller, more life-like depth.

13. Example: The Human Face Artists can tweak the depth settings and paint additional depth, or remove added depth to both organic and primitive geometry.

14. Attack of the Clone Brushing Much like Adobe Photoshop’s “Rubber Stamp” tool, Clone Brushing allows an artist to paint one part of an image with the color of another. This does not handle lighting or depth changes very well.

15. Eliminating Distortion Using color primarily from areas of similar depth and orientation, distortion is minimized. A reference image or layer is used as a texture. Brushing becomes real time texture mapping. U & V coordinates from the source are painted onto 3D objects.

16. Keep up, slow poke! 4 pixels are sampled for each pixel painted, allowing the system to interpolate color with some accuracy. Only bilinear interpolation is implemented, so textures can be painted in real time.

17. Optimization Pixels being painted are treated as “active” pixels. Any painting over these pixels is smoothed linearly, giving a natural look, without high overhead.

18. Expansion One pixel ahead of each active pixel is sampled prior to being painted. This allows the system to predict what will be painted next, in any direction, speeding up performance.

19. Freeze, you pixel scum! Painted pixels cannot be sampled. Textured objects are “frozen”.

20. Let There Be Light • The system can identify the major • light sources in a scene. • Light can then be added, removed, • replicated or altered.

21. Feature Separation Once the major light sources in a scene are identified, their effects can be negated, and individual objects can be discerned from a scene.

22. Correcting Depth Understanding the light in a scene greatly enhances the accuracy of the scene’s depth.

23. Soft shadows aren’t always distinguishable. Filtering helps smooth the transition between a soft shadow and an object. Bilateral Filtering

24. Limitations Even with filtering, some shadows will still be visible.

25. Possibilities Originally implemented as an Apple Quick Time Virtual Tour, panoramic stitching was the first implementation of Image Based Modeling. With this system, each object in a scene like this could be modeled.

26. But Does It Work?

27. Are There Other Options? Stereo images have been used, with limited success, to generate 3D models.

28. Image Based Modeling And Photo Editing With Non-Distorted Clone Brushing & Texture-Illuminance Decoupling Filters, a polygon efficient 3D scene can be generated from an image like this within 13 hours.