Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction

1. Joint Optimisation for Object ClassSegmentation and Dense StereoReconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Oxford Brookes University http://cms.brookes.ac.uk/research/visiongroup/

2. Joint Object Class Segmentationand Dense Stereo Reconstruction Black Box Solver Left Camera Image Object Class Segmentation Right Camera Image Dense Stereo Reconstruction

3. Joint Object Class Segmentationand Dense Stereo Reconstruction Objective : Joint Estimation Black Box Solver Left Camera Image Object Class Segmentation Right Camera Image Dense Stereo Reconstruction

4. Dense Stereo Reconstruction • For each pixel assigns a disparity label : y • Disparities from the discrete set {0, 1, .. D} Left Camera Image Right Camera Image Dense Stereo Result

5. Dense Stereo Reconstruction Unary Potential Disparity = 0 Unary Cost dependent on the similarity of patches, e.g.cross correlation

6. Dense Stereo Reconstruction Unary Potential Disparity = 5 Unary Cost dependent on the similarity of patches, e.g.cross correlation

7. Dense Stereo Reconstruction Unary Potential Disparity = 10 Unary Cost dependent on the similarity of patches, e.g.cross correlation

8. Dense Stereo Reconstruction Unary Potential Disparity = 15 Unary Cost dependent on the similarity of patches, e.g.cross correlation

9. Dense Stereo Reconstruction Pairwise Potential • Encourages label consistency in adjacent pixels • Cost based on the distance of labels Linear Truncated Quadratic Truncated

10. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution

11. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion Final solution

12. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion

13. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion After 3rd expansion

14. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion After 3rd expansion Final solution

15. Dense Stereo Reconstruction Does not work for Road Scenes ! Dense Stereo Reconstruction Original Image

16. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness in cameras Patches can be matched to any other patch for flat surfices

17. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness in cameras Patches can be matched to any other patch for flat surfices Could object recognition for road scenes help? Recognition of road scenes is relatively easy (Sturgess et al., BMVC09)

18. Object Class Segmentation • Aims to assign a class label for each pixel of an image • Classifier trained on the training set • Evaluated on never seen test images

19. Object Class Segmentation Unary Potential • Likelihood of a pixel taking a label (Shotton et al. ECCV06, He et al, CVPR04, Ladický et al. ICCV 09)

20. Object Class Segmentation Pairwise Potential • Contrast sensitive Potts model • Encourages label consistency in adjacent pixels

21. Object Class Segmentation Higher Order Potential • Encouraging consistency in superpixels (Kohli et al. CVPR08) • Merging information at different scales (Ladický et al. ICCV09)

22. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass Original Image Initial solution

23. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion

24. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky building grass Sky expansion

25. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky sky tree building building grass grass Sky expansion Tree expansion

26. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky sky sky tree tree building building building aeroplane grass grass grass Sky expansion Tree expansion Final Solution

27. Object Class Segmentation vs.Dense Stereo Reconstruction  • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) sky

28. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buildings & pedestrians have their typical height sky building car

29. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane sky building car road

30. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides sky building car road

31. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides • Both problems formulated as CRF • Joint approach possible? sky building car road

32. Joint Formulation • Each pixels takes label zi = [ xi yi ] L1  L2 • Dependency of xi and yi encoded as a unary and pairwise potential, e.g. • strong correlation between x = road, y = near ground plane • strong correlation between x= sky, y = 0 • Correlation of edge in object class and disparity domain

33. Joint formulation Unary Potential Object layer Joint unary links Disparity layer • Weighted sum of object class, depth and joint potential • Joint unary potential based on histograms of height

34. Joint Formulation Pairwise Potential Object layer Joint pairwise links Disparity layer • Object class and depth edges correlated • Transitions in depth occur often at the object boundaries

35. Joint Formulation

36. Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], Possible in case of metric pairwise potentials Inference

37. Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], Possible in case of metric pairwise potentials Inference Too many moves! ( |L1| |L2| ) Impractical !

38. Projected move for product label space One / Some of the label components remain(s) constant after the move Set of projected moves α-expansion in the object class projection Range-expansion in the depth projection Inference

39. Leuven Road Scene dataset Contained 3 sequences 643 pairs of images We labelled 50 training + 20 test images Object class (7 labels) Disparity (100 labels) Available on our website http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip Dataset Left camera Right camera Object GT Disparity GT

40. Qualitative results Original Image Object GT Object Result Disparity GT Disparity Alone Disparity Jointly • Large improvement for dense stereo estimation • Minor improvement in object class segmentation

41. Quantitative disparity results Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta

42. Application to monocular sequences Making method (close to) real time Application to multi-view problems Optical flow / motion estimation On-going and Future Work

43. First dataset with both object class and disparity labels Joint estimation improves significantly disparity results Projected moves make inference much faster Questions ? Summary