Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on

1. Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on Andrea Frome , EECS, UC Berkeley Yoram Singer, Google, Inc FeiSha , EECS, UC Berkeley JitendraMalik, EECS, UC Berkeley

2. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

3. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

4. What we do? • Goal • classify an image to a more appropriate category • Machine learning • Two steps • Training step • Testing step

5. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

6. Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Compute distance dji, dki Input distances to SVM for training , evaluate W

7. Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Compute distance dji, dki Input distances to SVM for training , evaluate W

8. Choosing features • Dataset: Caltech101 • Patch-based Features • SIFT • Old school • Geometric Blur • It’s a notion of blurring • The measure of similarity between image patches • The extension of Gaussian blur

9. Geometric blur

10. Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Compute distance dji, dki Input distances to SVM for training , evaluate W

11. Triplet • dji is the distance from image j to i • It’s not symmetric, ex: dji≠dij • dki > dji dji dki

12. How to compute distance • L2 norm dji, 1 1 Image i 2 Image j 3 m features dji, 1 distance vectordji

13. Example • Given 101 category, 15 images each category 101*15 Featurej distance vector distance vector 101*15 Image j vs training data

14. Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Compute distance dji, dki Input distances to SVM for training , evaluate W

15. Machine learning: SVM • Support Vector Machine • Function: Classify prediction • Supervised learning • Training data are n dimension vector

16. Example • Male investigate • Annual income • Free time • Have girlfriend?

17. Ex: Training data

18. space free vector income

20. Mathematical expression(1/2)

21. Mathematical expression(2/2)

22. Support vector free Model income

23. But the world is not so ideal.

24. Real world data

25. Hyper-dimension

26. Error cut

27. SVM standard mathematical expression Trade-off

28. In this paper • Goal: to get the weight vector W 101*15 wj feature wj, 1 Image weight wj of W

29. Visualization of the weights

30. How to choose Triplets? • Reference Image • Good friend - In the same class • Bad friend - In the different class • Ex: 101category, 15 images per category • 14 good friends & 15*100(1500) bad friends • 15*101(1515) reference images • total of about 31.8 million triplets

31. Mathematical expression(1/2) • Idealistic: • Scaling: • Different: The length of Weight i 0 0 triplet

32. Mathematical expression(2/2) • Empirical loss: • Vector machine:

33. Dual problem

34. Dual variable • Iterate the dual variables:

35. Early stopping • Satisfy KTT condition • In mathematics, a solution in nonlinear programming to be optimal. • Threshold • Dual variable updatefalls below a value

36. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

37. Flow chart: testing Query an image i Calculate Dxi, xis all training data, except itself. Output the most appropriate category

38. Flow chart: testing Query an image i Calculate Dxi, xis all training data, except itself. Output the most appropriate category

39. Query image? • Goal: classify the query image to an appropriate class • Using the remaining images in the dataset as the query image

40. Flow chart: testing Query an image i Calculate Dxi, xis all training data, except itself. Output the most appropriate category

41. Distance function(1/2) • Query image i Image i feature dxi, 1 distance vector distance vector 101*15 Image ivs all training data

42. Distance function(2/2) 101*15 Dji Image I vs all the training data

43. Flow chart: testing Query an image i Calculate Dxi, xis all training data, except itself. Output the most appropriate category

44. How to choose the best image? • Modified 3-NN classifier • no two images agree on the class within the top 10 • Take the class of the top-ranked image of the 10

45. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

46. Experiment & Result • Caltech 101 • Feature • Geometric blur (shape feature) • HSV histograms (color feature) • 5, 10, 15, 20 training images per category

48. Confusion matrix for 15

49. Outline • Introduction • Training step • Testing step • Experiment & Result • Conclusion

50. Conclusion • Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification