An Iterative Optimization Approach for Unified Image Segmentation and Matting

1. An Iterative Optimization Approach for Unified Image Segmentation and Matting Jue Wang1 and Michael F. Cohen2 University of Washington Microsoft Research

2. Known Unknown Introduction • Image Matting Observed Image Alpha Matte

3. Introduction • Trimap-based Matting Background Unknown Foreground Original Trimap Matte

4. Introduction • Matting After Segmentation • Trimap generation can be erroneous Automatically Generated Trimap Original + User’s input Graph-cut Segmentation [GrabCut, Rother et al., LazySnapping, Li et al., SIGGRAPH 2004]

5. Introduction • Our Approach: Unified Segmentation and Matting • No explicit trimap is required Iterative Optimization Initial Input Final Matte

6. Related Work • Blue Screen Matting • Mishima et al. 93 • Smith et al. SIGGRAGPH96 • Problem is simplified by photographing foreground objects against a constant-colored background

7. Related Work • Bayesian Matting, Chuang et al. CVPR2001 • Bayesian framework + MAP solver • Extended to video in Chuang et al. SIGGRAPH 2002 Bayesian Image Matting Bayesian Video Matting

8. Related Work • Poisson Matting, Sun et al. SIGGRAPH 2004 • Formulate matting as solving Poisson equations • Assumption: intensity change in the foreground and background is smooth • User can interactively manipulate the matte

9. Limitations of A Trimap • Key requirement for A trimap • should be as accurate as possible • Automatically generated trimap • is not optimal • can be erroneous • User specified trimap • can be very tedious to create

10. Observation: Image Hidden Random Field: Matte Goal: maximize Our Approach • Iterative Matte Optimization • Solving a Conditional Random Field (CRF)

11. Conditional Random Field (CRF) Markov Random Field (MRF) Goal: maximize Assumption: ‘s are dependent Assumption: ‘s are independent Goal: maximize CRF vs. MRF • In our system we solve CRF by iteratively solving MRFs Observations Y Hidden Field X

12. Iterative Solver • Color Sampling : The “conditional” part. Background Samples Foreground Samples

13. + Iterative Solver • Solve MRF

14. + Iterative Solver Step 1: Color Sampling Step 2: Solve MRF by Belief Propagation

15. MRF Set Up • Attributes of a Pixel/Node Foreground samples Background samples Observed color Observations Hidden Node Quantized alpha level Likelihoods Estimated foreground color 1. Limit the size of the MRF Estimated background color 2. Guide foreground/background sampling Uncertainty 3. Setting weights for nodes

16. Iterative Solver • Matte is estimated in a front-propagation fashion Region of Consideration Modeled as MRF Definite Foreground Definite Background

17. MRF Set Up • Attributes of a Pixel/Node Foreground samples Background samples Observed color Observations Hidden Node Quantized alpha level Likelihoods Estimated foreground color Estimated background color Uncertainty

18. MRF Set Up Local Sampling Local samples from low uncertainty areas are given high weights. • Color Sampling • Global Sampling • Train a GMM model on all foreground samples • Assign each sample to a single Gaussian • For a given node, choose the nearest Gaussian in color space, and collect samples belonging to this Gaussian • Global samples are given lower weight.

19. MRF Set Up • Data Costs Foreground samples Background samples Observed color Observations Hidden Node Quantized alpha level Likelihoods Estimated foreground color Estimated background color Uncertainty

20. MRF Set Up • Data Cost Foreground samples Background samples Observed color Observations Hidden Node Quantized alpha level Likelihoods Estimated foreground color Estimated background color Uncertainty

21. MRF Set Up • Neighborhood Cost p q

22. Solving MRF by Belief Propagation • Minimize T(p) Msg(p, T(p)) Msg(T(p),p) Msg(p, R(p)) Msg(L(p), p) Msg(R(p),p) Msg(p,L(p)) L(p) p R(p) Msg(B(p),p) Msg(p,B(p)) B(p)

23. Solving MRF by Belief Propagation • Minimize T(p) Msg(p, R(p)) L(p) p R(p) B(p)

24. Solving MRF by Belief Propagation • Minimize T(p) L(p) p R(p) B(p)

25. i j CRF Update Foreground samples Background samples Observed color Observations Hidden Node Quantized alpha level Estimated foreground color Estimated background color Uncertainty

26. CRF Update • Defining Uncertainties • Each pixel is associated with an uncertainty value • Pixels with low uncertainties will contribute more for solving the MRF

27. Results 0 3 6 14 9

28. Results Original image with user input Extracted Matte

29. Results • Comparison with Bayesian Matting User specified trimap Extracted matte using Bayesian matting

30. Results Original image with user input Extracted matte User specified trimap Estimated matte by Bayesian Matting

31. Results Original image with user input Extracted matte User specified trimap Estimated matte by Bayesian Matting

32. Extension to Video Automatic initialization on frame 2 Frame 1 Matte1 Matte 2 Matte 20 Matte 5 Matte 15 Matte 10

33. Extension to Video Extracted Matte Original Video All User Inputs

34. Failure Mode Original image with user input Extracted matte

35. Solution Extracted matte using our system User specified rough trimap Extracted matte using Bayesian Matting

36. Timings • A brute force implementation is computational expensive • 15-20 min for a 640*480 input image • Speedups • Fast Belief Propagation ideas [Felzenszwalb et al., CVPR 04] • Hierarchical methods • Using gradient information • Current system: 20-30 seconds per image

37. Future Directions • Combining all visual information • Color, texture, shape…

38. Summary • Solve a CDF for unified segmentation and matting • CDF solved iteratively • each iteration solves an MRF using Belief Propagation • Advantages: • No accurate trimap is required • Efficient for large semi-transparent objects • Extends to video • Disadvantages: • Computational expensive • No real-time interaction • Based only on color information