1 / 49

Background vs. foreground segmentation of video sequences

Background vs. foreground segmentation of video sequences. +. =. The Problem. Separate video into two layers: stationary background moving foreground Sequence is very noisy; reference image (background) is not given. Simple approach (1). background. temporal median. temporal mean.

geordi
Download Presentation

Background vs. foreground segmentation of video sequences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Background vs. foreground segmentation of video sequences + =

  2. The Problem • Separate video into two layers: • stationary background • moving foreground • Sequence is very noisy; reference image (background) is not given

  3. Simple approach (1) background temporal median temporal mean

  4. threshold Simple approach (2)

  5. Simple approach: noise can spoil everything

  6. Variational approach Find the background and foregroundsimultaneously by minimizing energy functional Bonus: remove noise

  7. [0,tmax] Notations given need to find C(x,t) background mask(1 on background, 0 on foreground) N(x,t) original noisy sequence B(x) background image

  8. Energy functional: data term B N B - N C

  9. Energy functional: data term • Degeneracy:can be trivially minimized by • C 0 (everything is foreground) • B N (take original image as background)

  10. Energy functional: data term C 1

  11. Energy functional: data term original images should be close to the restored background imagein the background areas there should be enough of background

  12. Energy functional: smoothness For background image B For background mask C

  13. Energy functional

  14. Edge-preserving smoothnessRegularization term Quadratic regularization [Tikhonov, Arsenin 1977] ELE: Known to produce very strong isotropic smoothing

  15. Edge-preserving smoothnessRegularization term Change regularization ELE:

  16. Edge-preserving smoothnessRegularization term ELE:

  17. n Edge-preserving smoothnessRegularization term Change the coordinate system: ELE: across the edge along the edge Compare:

  18. Edge-preserving smoothnessRegularization term Conditions on  Weak edge (s +0) (s) Isotropic smoothing (s) is quadratic at zero s

  19. Edge-preserving smoothnessRegularization term Conditions on  Strong edge (s +) • no smoothing across the edge: • more smoothing along the edge: (s) Anisotropic smoothing (s) does not grow too fast at infinity s

  20. Edge-preserving smoothnessRegularization term Conclusion Using regularization term of the form: we can achieve both isotropic smoothness in uniform regions and anisotropic smoothness on edges with one function 

  21. Edge-preserving smoothnessRegularization term Example of an edge-preserving function:

  22. Edge-preserving smoothnessSpace of Bounded Variations Even if we have an edge-preserving functional: if the space of solutions{u}contains only smooth functions, we may not achieve the desired minimum:

  23. Edge-preserving smoothnessSpace of Bounded Variations which one is “better”?

  24. Bounded Variation – ND case bounded open subset, function Variation of over φ where

  25. Edge-preserving smoothnessSpace of Bounded Variations integrable (absolute value) and with bounded variation Functions are not required to have an integrable derivative … What is the meaning of u in the regularization term? Intuitively: norm of gradient |u|is replaced with variation |Du|

  26. Total variation Theorem (informally): if uBV() then

  27. Hausdorff measure area = 0 area > 0 How can we measure zero-measure sets?

  28. Hausdorff measure 1) cover with balls of diameter  2) sum up diameters for optimal cover (do not waste balls) 3) refine:  0

  29. Hausdorff measure Formally: For ARNk-dimensional Hausdorff measure of A up to normalization factor; covers are countable • HN is just the Lebesgue measure • curve in image: its length = H1 in R2

  30. Total variation Theorem (more formally): if uBV() then u(x) u+ u- x0 x u+,u- - approximate upper and lower limits Su = {x; u+>u-} the jump set

  31. Energy functional data term regularization for background image regularization for background masks

  32. Total variation: example = perimeter = 4 Divide each side into n parts

  33. Edge-preserving smoothnessSpace of Bounded Variations Small total variation(= sum of perimeters) Large total variation (= sum of perimeters)

  34. Edge-preserving smoothnessSpace of Bounded Variations Small total variation Large total variation

  35. Edge-preserving smoothnessSpace of Bounded Variations BV informally: functions with discontinuities on curves Edges are preserved, texture is not preserved: energy minimization in BV temporal median original sequence

  36. Energy functional Time-discretized problem: Find minimum of E subject to:

  37. Existence of solution Under usual assumptions 1,2: R+R+ strictly convex, nondecreasing, with linear growth at infinity minimum of E exists in BV(B,C1,…,CT)

  38. (non-)Uniqueness is not convex w.r.t. (B,C1,…,CT)! Solution may not be unique.

  39. Uniqueness But if c  3range2(Nt , t=1,…,T, x), then the functional is strictly convex, and solution is unique. Interpretation: if we are allowed to say that everything is foreground, background image is not well-defined

  40. Finding solution BV is a difficult space: you cannot write Euler-Lagrange equations, cannot work numerically with function in BV. • Strategy: • construct approximating functionals admitting solution in a more regular space • solve minimization problem for these functionals • find solution as limit of the approximate solutions

  41. Approximating functionals Recall: 1,2(s) = s2 gives smooth solutions Idea: replace i with i, which are quadratic at s  0 and s 

  42. Approximating functionals

  43. Approximating problems has unique solution in the space • – convergence of functionals: ifE-converge to Ethen approximate solutions ofmin E converge to min E

  44. More results: Sweden

  45. More results: Highway

  46. More results: INRIA_1

  47. More results: INRIA_1Sequence restoration

  48. More results: INRIA_2Sequence restoration

  49. Thank You!

More Related