Epipolar lines

1. Epipolar lines epipolar plane epipolar lines epipolar lines Baseline O’ O

2. Rectification • Rectification: rotation and scaling of each camera’s coordinate frame to make the epipolar lines horizontal and equi-height,by bringing the two image planes to be parallel to the baseline • Rectification is achieved by applying homography to each of the two images

3. Rectification Baseline O’ O

4. Cyclopean coordinates • In a rectified stereo rig with baseline of length , we place the origin at the midpoint between the camera centers. • a point is projected to: • Left image: , • Right image: , • Cyclopean coordinates:

5. Disparity • Disparity is inverse proportional to depth • Constant disparity constant depth • Larger baseline, more stable reconstruction of depth (but more occlusions, correspondence is harder) (Note that disparity is defined in a rectified rig in a cyclopean coordinate frame)

6. Random dot stereogram • Depth can be perceived from a random dot pair of images (Julesz) • Stereo perception is based solely on local information (low level)

7. Moving random dots

8. Compared elements • Pixel intensities • Pixel color • Small window (e.g. or ), often using normalized correlation to offset gain • Features and edges (less common) • Mini segments

9. Dynamic programming • Each pair of epipolar lines is compared independently • Local cost, sum of unary term and binary term • Unary term: cost of a single match • Binary term: cost of change of disparity (occlusion) • Analogous to string matching (‘diff’ in Unix)

10. String matching • Swing → String S t r i n g Start S w i n g End

11. String matching • Cost: #substitutions + #insertions + #deletions S t r i n g S w i n g

13. Dynamic Programming • Shortest path in a grid • Diagonals: constant disparity • Moving along the diagonal – pay unary cost (cost of pixel match) • Move sideways – pay binary cost, i.e. disparity change (occlusion, right or left) • Cost prefers fronto-parallel planes. Penalty is paid for tilted planes

14. Dynamic Programming Start , Complexity?

15. Probability interpretation: Viterbi algorithm • Markov chain • States: discrete set of disparity • Log probabilities: product sum • Maximum likelihood: minimize sum of negative logs • Viterbi algorithm: equivalent to shortest path

16. Markov Random Field • Graph • In our case: graph isa 4-connected grid • Minimize energy of the form • Interpreted as negative log probabilities

17. Iterated Conditional Modes (ICM) • Initialize states (= disparities) for every pixel • Update repeatedly each pixel by the most likely disparity given the values assigned to its neighbors: • Markov blanket: the state of a pixel only depends on the states of its immediate neighbors • Similar to Gauss-Seidel iterations • Slow convergence to bad local minimum

18. Graph cuts: expansion moves • Assume is non-negative and is metric: • We can apply more semi-global moves using minimal s-t cuts • Converges faster to a better (local) minimum

19. α-Expansion • Expansion move allows in one shot each pixel to either change its state to α or to maintain its current state • We apply expansion moves for all possible disparity values and repeat this to convergence • At convergence energy is within a scale factor from the global optimum: where

20. α-Expansion (1D example)

21. α-Expansion (1D example)

22. α-Expansion (1D example)

23. α-Expansion (1D example)

24. α-Expansion (1D example) • But what about?

25. α-Expansion (1D example)

26. α-Expansion (1D example)

27. α-Expansion (1D example)

28. α-Expansion (1D example) • Such a cut cannot be obtained due to triangle inequality:

29. Common Metrics • Potts model: • Truncated : • Truncated squared difference is not a metric

30. Reconstruction with graph-cuts Original Result Ground truth