Multiple View Geometry

1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. Outline • Fundamental matrix estimation • Image rectification • Chapter 10 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. Epipolar geometry: basic equation separate known from unknown (data) (unknowns) (linear)

4. The singularity constraint SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation

6. The minimum case – 7 point correspondences one parameter family of solutions but F1+lF2 not automatically rank 2

7. 3 F7pts F F2 F1 The minimum case – impose rank 2 (obtain 1 or 3 solutions) (cubic equation) Compute possible l as eigenvalues of (only real solutions are potential solutions)

8. ~10000 ~100 ~10000 ~100 ~10000 ~10000 ~100 ~100 1 Orders of magnitude difference between column of data matrix  least-squares yields poor results ! The NOT normalized 8-point algorithm

9. Transform image to ~[-1,1]x[-1,1] (-1,1) (1,1) (0,0) (-1,-1) (1,-1) normalized least squares yields good results(Hartley, PAMI´97) The normalized 8-point algorithm (0,500) (700,500) (0,0) (700,0)

10. Geometric distance Gold standard Sampson error Symmetric epipolar distance

11. Gold standard Maximum Likelihood Estimation (= least-squares for Gaussian noise) Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi Parameterize: (overparametrized F=[t]xM) Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book)

12. Gold standard Alternative, minimal parametrization (with a=1) (note (x,y,1) and (x‘,y‘,1) are epipoles) • problems: • a=0  pick largest of a,b,c,d to fix to 1 • epipole at infinity  pick largest of x,y,w and of x’,y’,w’ 4x3x3=36 parametrizations! reparametrize at every iteration, to be sure

13. First-order geometric error(Sampson error) (one eq./point JJT scalar) (problem if some x is located at epipole) advantage: no subsidiary variables required (coordinates of world points)

14. Symmetric epipolar error

15. Some experiments:

16. Some experiments:

17. Some experiments:

18. Some experiments: Solid: 8-point algorithm Dashed: geometric error Dotted: algebraic error Residual error: (for all matches, not just n)

19. Recommendations: • Do not use unnormalized algorithms • Quick and easy to implement: 8-point normalized • Better: enforce rank-2 constraint during minimization • Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation)

20. Special cases: Enforce constraints for optimal results: Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof) – Essential matrix

21. The envelope of epipolar lines What happens to an epipolar line if there is noise? Monte Carlo n=50 n=25 n=15 n=10 Notice that narrower part of envelope is not the correct match

22. Other entities? Lines give no constraint for two view geometry (but will for three and more views) Curves and surfaces yield some constraints related to tangency

23. Automatic computation of F • Interest points • Putative correspondences • RANSAC • (iv) Non-linear re-estimation of F • Guided matching • (repeat (iv) and (v) until stable)

24. Extract feature points to relate images Required properties: Well-defined (i.e. neigboring points should all be different) Stable across views Feature points (i.e. same 3D point should be extracted as feature for neighboring viewpoints)

25. Feature points (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points homogeneous edge corner Mshould have large eigenvalues Feature = local maxima (subpixel) of F(1,2)

26. Feature points Select strongest features (e.g. 1000/image)

27. Evaluate NCC for all features with similar coordinates ? Feature matching Keep mutual best matches Still many wrong matches!

28. 3 3 2 2 4 4 1 5 1 5 Feature example Gives satisfying results for small image motions

29. Requirement to cope with larger variations between images Translation, rotation, scaling Foreshortening Non-diffuse reflections Illumination Wide-baseline matching geometric transformations photometric changes

30. Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until (#inliers,#samples)<95% (generate hypothesis) (verify hypothesis) RANSAC Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches

31. Finding more matches restrict search range to neighborhood of epipolar line (1.5 pixels) relax disparity restriction (along epipolar line)

32. Degenerate cases Planar scene Pure rotation No unique solution Remaining DOF filled by noise Use simpler model (e.g. homography) Model selection (Torr et al., ICCV´98, Kanatani, Akaike) Compare H and F according to expected residual error (compensate for model complexity) Degenerate cases:

33. Absence of sufficient features (no texture) Repeated structure ambiguity More problems: • Robust matcher also finds • support for wrong hypothesis • solution: detect repetition (Schaffalitzky and Zisserman, BMVC‘98)

34. geometric relation between two views is fully described by recovered 3x3 matrix F Two-view geometry

35. Outline • Fundamental matrix estimation • Image rectification • Chapter 10 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

36. e Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

37. ~ image size (calibrated) Planar rectification (standard approach) Distortion minimization (uncalibrated) Bring two views to standard stereo setup (moves epipole to ) (not possible when in/close to image)

40. Polar rectification (Pollefeys et al. ICCV’99) Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Works for all relative motions Guarantees minimal image size

41. polar rectification: example

42. polar rectification: example

43. Example: Béguinage of Leuven Does not work with standard Homography-based approaches

44. Example: Béguinage of Leuven