Step-by-Step model building

1. Step-by-Step model building Jana Kosecka Department of Computer Science George Mason University http://www.cs.gmu.edu/~kosecka

2. Review Feature selection Feature selection Feature correspondence Camera Calibration Landing Augmented Reality Euclidean Reconstruction Vision Based Control Sparse Structure and camera motion SIGGRAPH, August 8, 2004

3. Review Feature selection Feature selection Feature correspondence Camera Calibration Epipolar Rectification Dense Correspondence Texture mapping Euclidean Reconstruction Sparse Structure and motion 3-D Model SIGGRAPH, August 8, 2004

4. Partial Scene Knowledge Partial Motion Knowledge Partial Calibration Knowledge Review Feature selection Feature selection Feature correspondence Projective Reconstruction Epipolar Rectification Camera Self-Calibration Dense Correspondence Texture mapping Euclidean Reconstruction 3-D Model SIGGRAPH, August 8, 2004

5. Examples SIGGRAPH, August 8, 2004

6. Feature Selection • Compute Image Gradient • Compute Feature Quality measure for each pixel • Search for local maxima Feature Quality Function Local maxima of feature quality function SIGGRAPH, August 8, 2004

7. Feature Tracking • Translational motion model • Closed form solution • Build an image pyramid • Start from coarsest level • Estimate the displacement at the coarsest level • Iterate until finest level SIGGRAPH, August 8, 2004

8. Coarse to fine feature tracking 0 1 2 • compute • warp the window in the second image by • update the displacement • go to finer level • At the finest level repeat for several iterations SIGGRAPH, August 8, 2004

9. Optical Flow • Integrate around over image patch • Solve SIGGRAPH, August 8, 2004

10. Affine feature tracking Intensity offset Contrast change SIGGRAPH, August 8, 2004

11. Tracked Features SIGGRAPH, August 8, 2004

12. Wide baseline matching Point features detected by Harris Corner detector SIGGRAPH, August 8, 2004

13. Wide baseline Feature Matching • Select the features in two views • For each feature in the first view • Find the feature in the second view that maximizes • Normalized cross-correlation measure Select the candidate with the similarity above selected threshold SIGGRAPH, August 8, 2004

14. More correspondences and Robust matching • Select set of putative correspondences • Repeat 1. Select at random a set of 8 successful matches 2. Compute fundamental matrix 3. Determine the subset of inliers, compute distance to epipolar line 4. Count the number of points in the consensus set SIGGRAPH, August 8, 2004

15. RANSAC in action Inliers Outliers SIGGRAPH, August 8, 2004

16. Epipolar Geometry • Epipolar geometry in two views • Refined epipolar geometry using nonlinear estimation of F SIGGRAPH, August 8, 2004

17. Two view initialization • Recover epipolar geometry • Compute (Euclidean) projection matrices and 3-D struct. • Compute (Projective) projection matrices and 3-D struct. calibrated uncalibrated unknown SIGGRAPH, August 8, 2004

18. 3-D reconstruction from two views Projective Reconstruction SIGGRAPH, August 8, 2004

19. Repeat until convergence 2. Refine structure estimate given all the motions Multi-view reconstruction • Two view - initialized motion and structure estimates (scales) • Multi-view factorization - recover the remaining camera positions and refine the 3-D structure by iteratively computing 1. Compute i-th motion given the known structure iteration SIGGRAPH, August 8, 2004

20. Projective Ambiguity • calibrated case – projection matrices - and Euclidean structure • Uncalibrated case • for i-th frame • for 1st frame • Transform projection matrices and 3-D structure by H SIGGRAPH, August 8, 2004

21. Upgrade to Euclidean Reconstruction • From Projective to Euclidean Reconstruction • Remove the ambiguity characterized by • How to estimate H ? - Absolute quadric constraint • be recovered by a nonlinear minimization unknowns SIGGRAPH, August 8, 2004

22. Upgrade to Euclidean Reconstruction • Special case unknown focal length • other internal parameters are known (proj. center, aspect ratio) • Q can be parametrized in the following way • Zero entries in lead to linear constraints on Q • Initial estimate of Q can be obtained using linear techniques SIGGRAPH, August 8, 2004

23. Example of multi-view reconstruction Projective Reconstruction Euclidean reconstruction SIGGRAPH, August 8, 2004

24. Nonlinear Refinement • Euclidean Bundle adjustment • Initial estimates of are available • Final refinement, nonlinear minimization with respect to all unknowns SIGGRAPH, August 8, 2004

25. Example - Euclidean multi-view reconstruction SIGGRAPH, August 8, 2004

26. Example Original sequence Tracked Features SIGGRAPH, August 8, 2004

27. Recovered model SIGGRAPH, August 8, 2004

28. Euclidean Reconstruction SIGGRAPH, August 8, 2004

29. 1. Map the epipole to infinity Translate the image center to the origin Rotate around z-axis for the epipole lie on the x-axis Transform the epipole from x-axis to infinity 2. Find a matching transformation is compatible with the epipolar geometry is chosen to minimize overall disparity Epipolar rectification • Make the epipolar lines parallel • Dense correspondences along image scanlines • Computation of warping homographies SIGGRAPH, August 8, 2004

30. Epipolar rectification Rectified Image Pair SIGGRAPH, August 8, 2004

31. Epipolar rectification Rectified Image Pair SIGGRAPH, August 8, 2004

32. Dense Matching • Establish dense correspondences along scan-lines • Standard stereo configuration • Constraints to guide the search 1. ordering constraint 2. disparity constraint – limit on disparity 3. uniqueness constraint – each point has a unique match in the second view SIGGRAPH, August 8, 2004

33. Dense Matching SIGGRAPH, August 8, 2004

34. Dense Reconstruction SIGGRAPH, August 8, 2004

35. Texture mapping, hole filling SIGGRAPH, August 8, 2004

36. Texture mapping SIGGRAPH, August 8, 2004