Semi-Local Affine Parts for Object Recognition

1. Semi-Local Affine Partsfor Object Recognition Svetlana Lazebnik, Jean PonceUniversity of Illinois at Urbana-Champaign Cordelia Schmid INRIA Rhône-Alpes BMVC 2004

2. Overview • Goal: • Learning models for recognition of 3D object classes • Challenges: • Geometric invariance • Robustness to clutter, occlusion • Weakly supervised learning • Proposed approach: • An object representation using semi-local affine parts

3. Low-Level Features: Local Affine Regions • This work: Laplacian detector (Gårding & Lindeberg, 1996) • Other detectors: Kadir et al. (2004), Matas et al. (2002), Mikolajczyk & Schmid (2002), Tuytelaars & Van Gool (2004), etc.

4. In practice: two-image matching followed by validation initial pair validation set candidate part Learning Parts • Ideal approach: simultaneous correspondence search across entire training set

5. Two-Image Matching • Goal: to find collections of local affine regions that can be mapped onto each other using a single affine transformation • Implementation: greedy search based on geometric and photometric consistency constraints • Returns multiple correspondence hypotheses • Automatically determines number of regions in correspondence • Works on unsegmented, cluttered images (weakly supervised learning) A

6. Matching: Details • Initialization: • Identify triples of neighboring regions (i, j, k) in first image • Find all triples (i', j', k') in the second image such that i' (resp. j', k') is a potential match of i (resp. j, k), and j', k' are neighbors of i' j j' i i' k' k

7. Matching (cont.) • Beginning with each seed triple, iterate: • Estimate the affine transformation between centers of corresponding regions in current group of matches A

8. Matching (cont.) • Beginning with each seed triple, iterate: • Estimate the affine transformation between centers of corresponding regions in current group of matches • Determine geometric consistency of current group of matches • Geometric consistency criteria: • Distance between ellipse centers (residual) • Difference of major and minor axis lengths • Difference of ellipse orientations

9. Matching (cont.) • Beginning with each seed triple, iterate: • Estimate the affine transformation between centers of corresponding regions in current group of matches • Determine geometric consistency of current group of matches • Search for additional matches in the neighborhood of the current group

10. Matching: 3D Objects

11. Matching: 3D Objects closeup closeup

12. Matching: Faces spurious match ???

13. Finding Repeated Patterns and Symmetries

14. Learning Object Models for Recognition • Match multiple pairs of training images to produce a set of candidate parts • Use additional validation images to evaluate repeatability of parts and individual regions • Retain a fixed number of parts having the best repeatability score

15. Recognition Experiment: Butterflies Admiral Swallowtail Machaon Monarch 1 Monarch 2 Peacock Zebra • 26 training images per class • 8 initial pairs • 10 validation images • 437 test images • 619 images total

16. Butterfly Parts

17. Recognition • Top 10 parts per class used for recognition • Relative repeatability score: • Classification results: total number of regions detectedtotal part size Total part size (smallest/largest)

18. Classification Rate vs. Number of Parts

19. Detection Results (ROC Curves) Circles: reference relative repeatability rates. Red square: ROC equal error rate (in parentheses)

20. Successful Detection Examples Training images Test images (blue: occluded regions) All regions found in the test images

21. Unsuccessful Detection Examples Training images Test images (blue: occluded regions) All regions found in the test images

22. Future Work • Goal: • Recognize highly variable, non-rigid object categories • Proposed approach: • Treat semi-local affine parts as “black boxes” • Model spatial relations between parts • Learn these relations from training data in a weakly supervised fashion