1 / 44

The Layout Consistent Random Field for detecting and segmenting occluded objects

The Layout Consistent Random Field for detecting and segmenting occluded objects. John Winn Jamie Shotton. CVPR, June 2006. LayoutCRF contributions. Detection and segmentation Handles occlusion and deformation Multiple objects simultaneously Multiple classes. Roadmap. Related work

rimona
Download Presentation

The Layout Consistent Random Field for detecting and segmenting occluded objects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Layout Consistent Random Fieldfor detecting and segmenting occluded objects John Winn Jamie Shotton CVPR, June 2006

  2. LayoutCRF contributions • Detection and segmentation • Handles occlusion and deformation • Multiple objects simultaneously • Multiple classes

  3. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

  4. Related work: constellation models X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …

  5. Related work: constellation models X X X X [Fergus et al. CVPR 2003] [Leibe et al. ECCV 2004] [Crandall et al. ECCV 2006] [Kumar et al. CVPR 2005] …

  6. Related work: windowed detectors Classifier Car Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

  7. Related work: windowed detectors Classifier Car? Wall? Localised features Sliding window [Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

  8. Related work: multiclass segmentation TextonBoost [Shotton et al. ECCV 2006] building tree car road Doesn’t exploit layout of parts – can’t identify object instances [Tu et al. CVPR 2003] [He et al. CVPR 2004]

  9. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

  10. Dense part labelling Automatic per-pixel labelling based on a grid of parts Part labels (color-coded)

  11. Dense part labelling Background label Part labels (color-coded)

  12. Patch-based part detector • Decision forest classifier • Features are differences of pixel intensities Classifier [Lepetit et al. CVPR 2005]

  13. Decision trees Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities. Improved performance with multiple decision trees (random forest). Performs as well as boosting with shared features, but can process much more data in the same time.

  14. Colors show posterior over part labels – part detectors are noisy! Part color key Patch-based part detector

  15. Layout consistency

  16. Layout consistency

  17. Layout consistency Neighboring pixels (7,2) (8,2) (9,2) (p,q) ? (7,3) (8,3) (9,3) (7,4) (8,4) (9,4)

  18. Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) Layoutconsistent (7,4) (8,4) (9,4) Neighboring pixels (p,q) (p,q) (p+1,q) (p+1,q-1) Allows for deformation/rotation (p+1,q+1)

  19. Layout consistency (7,2) (8,2) (9,2) (7,3) (8,3) (9,3) (7,4) (8,4) (9,4) Neighboring pixels (p,q) ? (p,q) (p-1,q+1) (p,q+1) (p+1,q+1) Layoutconsistent

  20. Object occludes background (object edge) ‘Background’ occludes object One object instance occludes another Occlusions Not layout consistent = occlusion (or invalid)

  21. Effect of layout consistency Input image Part detector output Layout consistent regions With layout consistency

  22. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

  23. Layout Consistent Random Field Part detector Image I Part labels h

  24. Layout consistency Part detector Image I Layout Consistent Random Field Part labels h

  25. Layout Consistent Random Field Layout consistency Part detector Edge weight Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ} (set by hand)

  26. Inference of MAP labelling Proposed labelling Graph cuts with customised alpha-expansion move Part labels h [Boykov and Jolly, ICCV 2001]

  27. Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

  28. Inference of MAP labelling Graph cuts with customised alpha-expansion move Expansion move not accepted Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

  29. Inference of MAP labelling Graph cuts with customised alpha-expansion move Proposed labelling Part labels h [Boykov and Jolly, ICCV 2001]

  30. Example inference

  31. Decision tree re-learning Part-labels are inferred (constrained by known mask) and decision forest re-trained

  32. Limitation of layout consistency • Allows arbitrary stretching/scaling

  33. Global layout Global layout InstanceT2 Image I Part labels h Global layout constraint is (weak) star-shaped constellation model Constrains part locationsrelative to centroid Instance T1 Allows competition between different object instances

  34. T3 T1 T1 T2 T2 Example with global consistency Input image Layout consistent regions Instance labelling

  35. Roadmap • Related work • Layout consistency • Layout Consistent Random Field • Results

  36. UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

  37. UIUC car database Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

  38. UIUC car database: detection Results refer to detection of unoccluded cars only.

  39. Detecting heavily occluded faces • Caltech face database with artificial occlusions • AR face database with real occlusions

  40. Stability of part labelling Part color key

  41. Multi-class detection • Can extend to multiple classes with different numbers of part labels for each class • Example: building has multiple parts, other classes have one

  42. Summary + future directions Summary: LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects Future directions: • Extend to multiple viewpoints and multiple scales • Share parts between classes • Incorporate object context (‘car above road’) • Incorporate geometric cues

  43. Thank you jwinn@microsoft.com http://johnwinn.org/

More Related