Recent progress in optical flow

1. Recent progress in optical flow progress Presented by: Darya Frolova and Denis Simakov

2. Optical Flow is not in favor Very popular slide: Often not using Optical Flow is stated as one of the main advantages of a method Optical Flow methods have a reputation of either unreliable or slow Recent works claim: Optical Flow can be computed fastand accurately

3. more methods many methods Optical Flow Research: Timeline Horn&Schunck Lucas&Kanade 1981 1992 1998 now Benchmark:Galvin et.al. Benchmark:Barron et.al. Seminal papers A slow and not very consistent improvement in results, but a lot of useful ingredients were developed

4. In This Lecture We will describe : • Ingredients for an accurate and robust optical flow • How people combine these ingredients • Fast algorithms Papers: • Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005 • High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005 • Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005 • Towards ultimate motion estimation: Combining highest accuracy with real-time performanceA. Bruhn, J. Weickert, 2005 • Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006

5. What is Optical Flow? …

6. Definitions The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] + = frame #2 frame #1 flow field The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck]

7. flow (2) flow (1): true motion Ambiguity of optical flow Frame 1

8. Applications optical flow • video compression • 3D reconstruction • segmentation • object detection • activity detection • key frame extraction • interpolation in time motion field We are usually interested in actual motion

9. Outline • Ingredients for an accurate and robust optical flow • Local image constraints on motion • Robust statistics • Spatial coherence • How people combine these ingredients • Fast algorithms

10. Local image constraints

11. Brightness Constancy u frame t+1 v frame t

12. Complex dependence on Linearized brightness constancy Deviation from brightness constancy (we want it to be zero) Linearize:

13. Linearized brightness constancy Let us square the difference: J – “motion tensor”, or “structure tensor”

14. Averaged linearized constraint J is a function of x, y (a matrix for every point) Combine over small neighborhoods (more robust to noise): = J *

15. Method of Lucas&Kanade • Solve independently for each point [ Lucas&Kanade 1981] linear system Can be solved for every point where matrix is not degenerate

16. Lukas&Kanade - Results Rubik cube Hamburg taxi flow field flow field

17. Brightness is not always constant Rotating cylinder Brightness constancy does not always hold Gradient constancy holds intensity intensity derivative position position

18. Local constraints - Summary We have seen linearized • brightness constancy averaged linearized averaged linearized • gradient constancy

19. Local constraints are not enough!

20. Local constraints work poorly Optical flow direction using only local constraints input video color encodes direction as marked on the boundary

21. Where local constraints fail Uniform regions Motion is not observable in the image (locally)

22. Where local constraints fail “Aperture problem” We can estimate only one flow component (normal)

23. Where local constraints fail Occlusions We have not seen where some points moved Occluded regions are marked in red

24. Obtaining support from neighbors • Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong

25. Robust combination of partially reliable data or: How to hold elections

26. L2: L1: xi → xi + ∆ Influence of xi on E: equal for all xi proportional to Outliers influence the most Majority decides Toy example Find “best” representative for the set of numbers xi

27. Oligarchy Democracy Elections and robust statistics many ordinary people a very rich man wealth Votes proportional to the wealth One vote per person like in L1 norm minimization like in L2 norm minimization

28. usual: L2 robust regularized robust: L1 ε • easy to analyze and minimize • sensitive to outliers • robust in presence of outliers • non-smooth: hard to analyze • smooth: easy to analyze • robust in presence of outliers Combination of two flow constraints [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

29. Spatial Propagation

30. Obtaining support from neighbors • Two main problems with local constraints: • information about motion is missing in some points => need spatial coherency • constraints do not hold everywhere => need methods to combine them robustly good missing wrong

31. - flow in the x direction • flow in the y direction • gradient Homogeneous propagation This constraint is not correct on motion boundaries => over-smoothing of the resulting flow [Horn&Schunck 1981]

32. Robustness to flow discontinuities ε (also known as isotropic flow-driven regularization) [T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004] High accuracy optical flow estimation based on a theory for warping

33. Selective flow filtering • We want to propagate information • without crossing image and flow discontinuities • from “good” points only (not occluded) Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

34. I I I I x x x Bilateral filter Unilateral (usual) Bilateral x Preserves discontinuities! [C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

35. occluded regions Using of bilateral filter - Example cyan rectangle moves to the right and occludes background region marked by red [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

36. Learning of spatial coherence • Come to the next lecture …

37. Spatial coherence: Summary Homogeneous propagation - oversmoothing Robust statistics with homogeneous propagation - preserves flow discontinuities Bilateral filtering - combines information from regions with similar flow and similar intensities Handles occlusions

38. Two more useful ingredients in brief – one slide each

39. 2D vs. 3D Several frames allow more accurate optical flow estimation 2 frames: Several frames:

40. MultiscaleOptical Flow Linearization: valid only for small flow pyramid for frame 1 pyramid for frame 2 frame 1warped ? + upsample + (other names: “warping”, “coarse-to-fine”, “multiresolution”)

41. Methods How to make tasty soup with these ingredients: several recipes

42. Outline • Ingredients for an accurate and robust optical flow • How people combine these ingredients • Lukas & Kanade meet Horn & Schunck • The more ingredients – the better • Bilateral filtering and occlusions • Fast algorithms

43. Combining ingredients • Spatial coherency • Homogeneous • Flow-driven • Bilateral filtering + occlusions • Local constraints • Brightness constancy • Image gradient constancy Energy = ∫ϕ(Data) + ∫ϕ(“Smoothness”) Combined using robust statistics Computed coarse-to-fine Use several frames

44. Combining Local and Global Remember: Lucas&Kanade Horn&Schunk Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002]

45. Sensitivity to noise – quantitative results frame t+1 Error measure: angle between true and computed flow in (x,y,t) space frame t ground truth flow

46. The more ingredients - the better brightness constancy spatial coherence gradient constancy [Bruhn, Weickert, 2005]Towards ultimate motion estimation: Combining highest accuracy with real-time performance

47. Quantitative results Angular error Method Yosemite sequence with clouds Average error decreases, but standard deviation is still high….

48. Influence of each ingredient For Yosemite sequence with clouds

49. Handling occlusions bilateral filtering of flow:preserve intensity and flow discontinuities; model occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection

50. Qualitative results