Learning a Scale-Invariant Model for Curvilinear Continuity

1. Learning a Scale-Invariant Model for Curvilinear Continuity Xiaofeng Ren, Charless Fowlkes, Jitendra Malik 1

2. Relations to Previous Talks • Cf Curve detection w/ hypothesis testing • natural images • practical algorithm • Cf Pb: learning local boundary detectors • input to curvilinear grouping • as a baseline for comparison • Cf Benchmarking boundary detectors • precision-recall curves • improvements over Pb 2

3. The Quest of Boundary Detection • Widely used for mid/high-level vision tasks • Huge literature on edge detection [Canny 86] • Typically measuring local contrast • Approaching human performance? [Martin, Fowlkes & Malik 02] [Fowlkes, Martin & Malik 03] 3

4. Limit of Local Boundary Detection 1 2 3 4 4

5. Curvilinear Continuity • Good Continuation • Visual Completion • Illusory Contours 5

6. Continuity in Human Vision • [Wertheimer 23] • [Kanizsa 55] • [von der Heydt et al 84] • evidence in V2 • [Kellman & Shipley 91] • geometric conditions of completion • [Field, Hayes & Hess 93] • quantitative analysis of factors • [Kapadia, Westheimer & Gilbert 00] • evidence in V1 • [Geisler et al 01] • evidence from ecological statistics … … … … 6

7. Extensive literature on curvilinear continuity [Shashua & Ullman 88], [Parent & Zucker 89], [Heitger & von der Heydt 93], [Mumford 94], [Williams & Jacobs 95], [Elder & Zucker 96], [Williams & Thornber 99], [Jermyn & Ishikawa 99], [Mahamud et al 03], …, … Problems with most of the previous approaches no support from any groundtruth data usually demonstrated on a few simple/synthetic images no quantitative evaluation Continuity in Computer Vision 7

8. Outline • Ecological Statistics of Contours • A Scale-Invariant Representation • Learning Models of Curvilinear Continuity • Quantitative Evaluation • Discussion and Future Work 8

9. Outline • Ecological Statistics of Contours • Groundtruth boundary contours • Power law in contours • A multi-scale Markov model • A Scale-Invariant Representation • Learning Models of Curvilinear Continuity • Quantitative Evaluation • Discussion and Future Work 9

10. Human-Segmented Natural Images [Martin et al, ICCV 2001] 1,000 images, >14,000 segmentations 10

11. t(s+1) s+1 t(s) s Contour Geometry • First-Order Markov Model [Mumford 94, Williams & Jacobs 95] • Curvature: white noise ( independent from position to position ) • Tangent t(s): random walk • Markov assumption: the tangent at the next position, t(s+1), only depends on the current tangent t(s) 11

12. t(s+1) s+1 t(s) s Contours are Smooth P( t(s+1) | t(s) ) marginal distribution of tangent change 12

13. Testing the Markov Assumption Segment the contours at high-curvature positions 13

14. Prediction: Exponential Distribution If the first-order Markov assumption holds… • At every step, there is a constant probability p that a high curvature event will occur • High curvature events are independent from step to step Let L be the length of a segment between high-curvature points • P( L>=k ) = (1-p)k • P( L=k ) = p(1-p)k L has an exponential distribution 14

15. Empirical Distribution: Power Law Probability Contour segment length L 15

16. Power Laws in Nature • Power laws widely exist in nature • Brightness of stars • Magnitude of earthquakes • Population of cities • Word frequency in natural languages • Revenue of commercial corporations • Connectivity in Internet topology … … • Usually characterized by self-similarity and scale-invariant phenomena 16

17. t(1)(s+1) s+1 • Coarse-to-fine contour completion • [Ren & Malik 02] Multi-scale Markov Models t(s+1) • Assume knowledge of contour orientation at coarser scales s+1 2nd Order Markov: P( t(s+1) | t(s) , t(1)(s+1) ) Higher Order Models: P( t(s+1) | t(s) , t(1)(s+1), t(2)(s+1), … ) t(s) s 17

18. Contour Synthesis First-Order Markov: P( t(s+1) | t(s) ) Multi-scale Markov: P( t(s+1) | t(s) , t(1)(s+1), t(2)(s+1), … ) [Ren & Malik 02] 18

19. Outline • Ecological Statistics of Contours • A Scale-Invariant Representation • Piecewise linear approximation • Constrained Delaunay Triangulation • Learning Models of Curvilinear Continuity • Quantitative Evaluation • Discussion and Future Work 19

20. Use Pb (probability of boundary) as input Combining local brightness, texture and color cues Trained from human-marked segmentation boundaries Outperform existing local boundary detectors including Canny Local “Probability of Boundary” • [Martin, Fowlkes & Malik 02] 20

21. Threshold Pb and find connected boundary pixels Recursively split the boundaries until each piece is approximately straight  b b a  c a c Split at C Piecewise Linear Approximation minimize  21

22. Standard in computational geometry Dual of the Voronoi Diagram Unique triangulation that maximizes the minimum angle avoiding long skinny triangles Efficient and simple randomized algorithm Delaunay Triangulation 22

23. A variant of the standard Delaunay Triangulation Keeps a given set of edges in the triangulation [Chew 87] [Shewchuk 96] • Still maximizes the minimum angle • Widely used in geometric modeling and finite elements Constrained Delaunay Triangulation 23

24. A typical scenario of contour completion high contrast high contrast low contrast • CDT picks the “right” edge, completing the gap The “Gap-filling” Property of CDT 24

25. Examples Image Pb CDT 25

26. Black: gradient edges or G-edges Green: completed edges or C-edges Examples 26

27. Outline • Ecological Statistics of Contours • A Scale-Invariant Representation • Learning Models of Curvilinear Continuity • Transferring Groundtruth to CDT • A simple model of local continuity • A global model w/ Conditional Random Fields • Quantitative Evaluation • Discussion and Future Work 27

28. Transferring Groundtruth to CDT • Human-marked boundaries are given on the pixel-grid • Label the CDT edges by bipartite matching d distance threshold d in matching CDT edges Phuman: percentage of pixels matched to groundtruth human-marked boundaries 28

29. pb1, G1  pb0, G0 Model for Continuity • Goal: define a continuity-enhanced Pb on CDT edges • Consider a pair of adjacent edges in CDT: • Each edge has an associated set of features • average Pb over the pixels belonging to this edge • indicator G, gradient edge or completed edge? • Continuity: angle  “bi-gram” 29

30. Binary Classification • Assuming contours are always closed: each vertex in the CDT graph is adjacent to either zero or two true boundary edges • A binary classification problem: (0,0) or (1,1) “bi-gram” 30

31. pb1, G1  pb0, G0 Learning Local Continuity • Binary classification: (0,0) or (1,1) • Transferred Groundtruth labels on CDT edges • Features: • average Pb • (G0*G1): both are gradient edges? • angle  • Logistic regression 31

32. PbL: Pb + Local Continuity Evidence of continuity comes from both ends pb1, G1 pb2, G2 1 2 pb0, G0  take max. over all possible pairs L L = PbL 32

33. Local inference Xi+1 Xi A Global Model of Continuity? X={X1,X2,…,Xm} Global inference incorporating all local continuity information? 33

34. For each edge i, define a set of features {g1,g2,…,gh} Potential function exp(i)at edge i For each junction j, define a set of features {f1,f2,…,fk} Potential function exp(j)at juncion j Conditional Random Fields X={X1,X2,…,Xm} [Pietra, Pietra & Lafferty 97] [Lafferty, McCallum & Pereira 01] 34

35. Conditional Random Fields Potential function on edges {exp(i)} Potential function on junctions {exp(j)} This defines a probability distribution over X: X={X1,X2,…,Xm} where Estimate P(Xi|) 35

36. Buliding a CRF Model • What are the features? • edge features are easy: Pb, G • junction features: type and continuity • How to make inference? • How to learn the parameters? X={X1,X2,…,Xm} Estimate P(Xi|) 36

37.  degg=0,degc=2 degg=0,degc=2 Junction Features in CRF • Junction types (degg,degc): degg=1,degc=0 degg=0,degc=2 degg=1,degc=2 • Continuity term for degree-2 junctions 37

38. Inference w/ Belief Propagation Fr • Belief Propagation • Xi: state of the node (edge) i • Fq: state of the factor (junction) q • potentials on Xi,Xj,Xk, Fq={Xi, Xj, Xk} • want to compute PbG=P(Xi) • mqi: “belief” about Xi from Fq Xj mjq mir mqi Xi Fq mkq Xk • The CDT graph has many loops in it 38

39. Inference w/ Loopy Belief Propagation • Loopy Belief Propagation • just like belief propagation • iterates message passing until convergence • lack of theoretical foundations and known to have convergence issues • however becoming popular in practice • typically applied on pixel-grid • Works well on CDT graphs • converges fast • produces empirically sound results [Berrou 93], [Freeman 98], [Murphy 99], [Weiss 97,99,01] 39

40. Learning the Parameters • Maximum-likelihood estimation in CRF Let denote the groundtruth labeling on the CDT graph • Many possible optimization techniques • gradient descent, iterative scaling, conjugate gradient, … • Gradient descent works well 40

41. there are more non-boundary edges than boundary edges a continuation is better than a line-ending junctions are rare G-edges are better for continuation than C-edges Interpreting the Parameters • The junction parameters (degg,degc) on the horse dataset: (0,0)= 2.8318 (1,0)= 1.1279 (2,0)= 1.3774 (3,0)= 0.0342 (2,0)= 1.3774 (1,1)= -0.6106 (0,2)= -0.9773 41

42. Outline • Ecological Statistics of Contours • A Scale-Invariant Representation • Learning Models of Curvilinear Continuity • Quantitative Evaluation • The precision-recall framework • Experimental results on three datasets • Discussion and Future Work 42

43. Datasets • Baseball player dataset [Mori et al 04] • 30 news photos of baseball players in various poses, 15 training and 15 testing • Horse dataset [Borenstein & Ullman 02] • 350 images of standing horses facing left, 175 training and 175 testing • Berkeley Segmentation Dataset [Martin et al 01] • 300 Corel images of various natural scenes and ~2500 segmentations, 200 training and 100 testing 43

44. Evaluating Boundary Operators • Precision-Recall Curves [Martin, Fowlkes & Malik 02] • threshold the output boundary map • bipartite matching with the groundtruth m pixels on human-marked boundaries k matched pairs n detected pixels above a given threshold Precision = k/n, percentage of true positives Recall = k/m, percentage of groundtruth being detected • Project CDT edges back to the pixel-grid 44

45. Use Phuman the soft groundtruth label defined on CDT graphs: precision close to 100% Pb averaged over CDT edges: no worse than the orignal Pb No Loss of Structure in CDT 45

46. Continuity improves boundary detection in both low-recall and high-recall ranges Global inference helps; mostly in low-recall/high-precision Roughly speaking, CRF>Local>CDT only>Pb 46

47. 47

48. 48

49. Image Pb Local Global 49

50. Image Pb Local Global 50