1 / 46

LOCUS (Learning Object Classes with Unsupervised Segmentation)

LOCUS (Learning Object Classes with Unsupervised Segmentation) A variational approach to learning model-based segmentation. John Winn Microsoft Research Cambridge with Nebojsa Jojic, MSR Redmond. 7 th July 2006. Overview. Learning object models The LOCUS model Experiments & results

kemal
Download Presentation

LOCUS (Learning Object Classes with Unsupervised Segmentation)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. LOCUS(Learning Object Classes with Unsupervised Segmentation) A variational approach to learning model-based segmentation. John WinnMicrosoft Research Cambridge with Nebojsa Jojic, MSR Redmond 7th July 2006

  2. Overview • Learning object models • The LOCUS model • Experiments & results • Extensions to LOCUS

  3. Goal Long Term Goal Recognise ~10,000 object classes.

  4. Learning from ‘buckets’ of images Learningalgorithm Horsemodel • Object Segmentation • Object Recognition • Object Detection

  5. + Object segmentation LOCUS Horsemodel

  6. Related work

  7. Constellation models • Weakly supervised • Probabilistic framework • Sparse • No segmentation Object class recognition by unsupervised scale-invariant learning. R. Fergus, P. Perona, and A. Zisserman. CVPR 2003 A Bayesian approach to unsupervised One-Shot learning of Object categories. L. Fei-Fei, R. Fergus, and P. Perona. ICCV 2003

  8. Fragment-based • Dense model • Supervised • Non-probabilistic • No global shape model Learning to segment. E. Borenstein and S. Ullman. ECCV 2004 Combining top-down and bottom-up segmentation. E. Borenstein, E. Sharon, and S. Ullman. CVPR 2004

  9. Codebook-based • Probabilistic • Dense model • Supervised • Ad-hoc inference Combined object categorization and segmentation with an implicit shape model. B. Leibe, A. Leonardis, and B. Schiele. ECCV ‘04

  10. OBJ CUT • Probabilistic • Dense model • Supervised • Requires video

  11. LOCUS overview • Weakly supervised learning Buckets of images - no annotation required. • Probabilistic generative modelof both object and background. • Dense modelAll pixels modelled, not just at interest points. • Combines global and local cuesModels global shape and local appearance + edges. • Iterative inference processSimultaneous localisation, segmentation, pose estimation.

  12. The LOCUS model

  13. LOCUS model Shared between images Class shape π Class edge sprite μo,σo Deformation field D Position & size T Different for each image Mask m Edge image e Background appearance λ0 Object appearance λ1 Image

  14. background Background mixture coefficients Objectmixture coefficients object λ0 λ1 Shared mixture components: LOCUS model: appearance Mask m Image z

  15. favours segmentation along contrast edges LOCUS model: mask background object 8-neighbour Markov Random Field (as used in GrabCut)

  16. Class shapeπ Transformation TN T4 T2 T3 T1 … LOCUS model: shape/position …

  17. TN T1 Iterative inference Class shapeπ Iteration #1 T2 T3 T4 … …

  18. TN T1 Iterative inference Class shapeπ Iteration #2 T2 T3 T4 … …

  19. TN T1 Iterative inference Class shapeπ Iteration #3 T2 T3 T4 … …

  20. TN T1 Iterative inference Class shapeπ Iteration #5 T2 T3 T4 … …

  21. TN T1 Iterative inference Class shapeπ Iteration #8 T2 T3 T4 … …

  22. TN T1 Iterative inference Class shapeπ Iteration #12 T2 T3 T4 … …

  23. Non-rigid objects Class shapeπ Translation and scale is not enough.

  24. Deformation field D 5x5 blocks T Prior ensures smoothness LOCUS model: pose Class shapeπ

  25. TDN TD1 TD2 TD3 … LOCUS model: pose Class shapeπ …

  26. Class edge sprite μo,σo TDN TD1 TD2 TD3 … Edge images e … LOCUS model: edge Original images …

  27. LOCUS model: overview Shared between images Class shape π Class edge sprite μo,σo Deformation field D Position & size T Different for each image Mask m Edge image e Background appearance λ0 Object appearance λ1 Image

  28. Inference • Aim to infer all latent variables, • For each image:background appearance λ0, object appearanceλ1, deformation D, transformation T, mask m, • Class variables: shape π, edge sprite μo, σo. • Bayesian inference is carried out using variational message passing with a fully factorised variational distribution. • Optimisation of grid-structured variational free energy terms (relating to the deformation field D and the mask m) achieved using graph cuts.

  29. Experiments & results

  30. Experiments LOCUS applied to 8 sets of 20 images each containing objects of the same class. • Horses • Faces • Cars (rear) • Cars (side) • Motorbikes • Aeroplanes • Cows • Trees For each class, we ran separate experiments for color and texture appearance models.

  31. Results: horses

  32. Results: horses

  33. Results: cars

  34. Results: cars

  35. Faces Cars (rear) Motorbikes Planes Cows Trees Results: remaining classes

  36. Segmentation accuracy To evaluate segmentation quantitively, we used hand segmentations for horses and cars (side).

  37. Object registration Transformation + deformation field registers object outlines (and some internal edges).

  38. Object registration

  39. Extensions to LOCUS

  40. Recognition + segmentation Object recognition using only global shape: Overall: 88% accuracy.

  41. Probabilistic Index Maps 2 indices 9 indices Each image has a ‘palette’ of appearance models – palette invariance.

  42. Probabilistic Index Maps

  43. Learning objects from video Object shape Object edge sprite

  44. Locumotion Add flow and track constraints to achieve motion segmentation: Tracking/flow estimation by Larry Zitnick

  45. Conclusions • LOCUS gives unsupervised segmentations of accuracy equivalent to state-of-the-art supervised methods. • General-purpose model allows: • Object localisation • Pose estimation • Object segmentation • Motion segmentation/object tracking • Object recognition/detection (in combination with discriminative model)

  46. Questions ?

More Related