Similarity-based matching for face authentication

1. Similarity-based matching for face authentication Christophe Rosenberger Luc Brun ICPR 2008

2. outline • Introduction • Image description • Dissimilarity measure • Pattern modeling • Experimental results

3. Introduction • Propose in this paper a face authentication method based on a similarity measure • Find interest key points • SIFT descriptor is used • A graph is then built where nodes correspond to these keypoints • Model the authentication problem as a graph matching process

4. Image description • SIFT (Scale Invariant Feature Transform) • Applied locally on key-points • Based upon the image gradients in a local neighborhood • SIFT detector • Scale-space extrema detection • Keypoint localization • SIFT descriptor • Orientation assignment • Keypoint descriptor

5. SIFT detector • Detection of scale-space extrema • DoG(Difference-of-Gaussian) filter for scale space • Keypoint localization (local extrema) X is selected if it is larger or smaller than all 26 neighbors

6. SIFT detector & descriptor • Accurate keypoint localization • Reject points with low contrast (flat) • Elimination edge responses • Orientation assignment • To make the feature invariant to rotation • Find local major orientation • Orientation histogram for a window size • Local image descriptor • 16x16 array of locations in scale space • Create array of orientation histograms • 8 orientations x 4x4 histogram array (128D) Gradient magnitude Gradient orientation

7. Dissimilarity measure • Each individual is described by a set of invariant features Y (I) = {ki = (si, xi, yi)}, i = 1 : N(I) • Si is the SIFT invariant descriptor • (xi, yi) its position in the original image I • N(I) the number of detected keypoints in the image I • Database image • {Y (Ii), i = 1 : M} • Dissimilarity between two sets of points such as Y (I1) and Y (I2) • Often modeled as a weighted bipartite graph matching problem • Greedy algorithm • Hungarian method • Hybrid method

8. Greedy algorithm • Weighted bipartite graph G(U,V,E,W) • Given the graph G , scan the set of edges and find an edge (u,v) with the smallest weight • Match u with v and remove the edges incident on u and those incident on v • Repeat the above step until every node in U is matched to some node in V 3 3 10 12 5 5 20 25 6 4 9 4

9. Greedy algorithm • The greedy algorithm does not always produce a matching of minimum weight • Hungarian method • A well-known algorithm for finding a minimum weight matching in a bipartite graph • Node in U by u1, u2,… un • In V by v1, v2,… vn • x ij= 1 if ui is matched with vjand 0 otherwise subject to constraints 1 A A C C 16 16 0 0 B 2 D B D Each node in U is matched to exactly one node in V and vice versa

10. Hungarian method - Example Step 1 First for each row we subtract the row minimum from the rest of the row R1 B1 R2 B2 R3 B3 R4 B4 Step 2 Then for each column we subtract the column minimum from the rest of the column

11. Hungarian method - Example R1 B1 Step 4 1.Find the min value not covered (B3-R1) 2.subtracted from each uncovered entry and added to each entry covered by both a vertical and horizontal line (B1-R2 & B1-R4) 3. Step 3 is then repeated Step 3 Cover the min number of rows or columns required to cover all of the zeros in the matrix R2 B2 R3 B3 R4 B4 If the # of line = the # of rows/columns in the matrix then we are finished Otherwise, we have to go to the next step 3 Result: Blue1 - Red3Blue2 - Red2Blue3 - Red1Blue4 – Red4 4 13 2

12. Hybrid method • We formulated the point matching problem in terms of minimum weight matching in a bipartite graph • In a real world situation, this is not always appropriate • Feature points extracted from two corresponding images • First image are denoted by a and b • Second image are denoted by c and d One might argue that since b and c both have the same feature value of 100,we should match them together and completely ignore a and d

13. Hybrid method • Feature points forming tight clusters in the feature space can be avoided by a simple combination of the greedy method and the Hungarian method • Combined algorithm • We first apply the Hungarian method to get a matching M1 • Apply the greedy algorithm to get a matching M2 • form their intersection M=M1∩M2 Greedy algorithm Combined matching A bipartite graph Hungarian method

14. Dissimilarity measure • Such a weighted bipartite graph matching method • Optimizes a global matching score • May thus be sensible to outliers • We thus rather use the following matching method • Given two points and , we say that x is associated to y iff : • C is an arbitrary threshold • d(., .) denotes the Euclidean distance between SIFT descriptors • y’ denotes any point of Y (I2) whose distance to x is minimal but greater than d(x,y)

15. Dissimilarity measure • The significance of the necessary gap between d(x, y) and d(x, y’) is encoded by the constant C • y is associated to x • x is the closest point from y among Y (I1) • second smallest value of this distance d(y, x’) satisfies d(y, x) < Cd(y, x’) • X is matched to Y • iff x is associated to y and y is associated to x • If Y (I1) and Y (I2) do not define any matching point • Both sets have an infinite distance • Otherwise • The set of matching points M1 of Y1 and M2 of Y2 may be computed sequentially • If we cannot find two couples of points within Y (I1) × Y (I2) with a similar distance, the relationship “is matched to “ induces a bijective mapping Φ from M1 to M2

16. Dissimilarity measure • First step • The above procedure classify both sets Y (I1) and Y (I2) • Dissimilar (case M1 = M2 =ø) • A set of N = |M1| = |M2| couples (x, Φ(x)) of matched points • Second step • Take into account the spatial relationship of the points among M1 & M2 • Use one of the four pattern representations • Using the locations of each matched keypoint • Weighted graph • Edges are weighted by the similarity of the adjacent points • The matching between M1 and M2being encoded by the application Φ • Encode the adjacency matrices of both graphs • A line or a column encoding the adjacency relationships • Point x of the first matrix • Φ(x) in the second matrix Graph Adjacency matrix

17. Pattern modeling • Spatial relationship between keypoints • Using graph representations • Connect keypoints by considering their ”local neighborhoods” • Each keypoint corresponds to a vertex in the neighborhood graph which connects “close” vertices • Epsilon-neighborhood graph (ENG) • Two vertices are connected if the distance of the corresponding keypoints is less than epsilon • A completely connected graph (CCG) • epsilon greater than the maximal distance between keypoints.

18. Pattern modeling • Symmetric k-nearest neighbor graph (SKNG) • Two vertices V1, V2are connected if V1 is among the k-nearest neighbors of V2 or vice versa • Mutual k-nearest neighbor graph (MKNG) • Two vertices V1, V2 are connected if V1 is among the k-nearest neighbors of V2 and vice versa • Given both adjacency matrices • The quality of the match is estimated using the squared Euclidean distance which provides a global measure of the dissimilarity between both graphs

19. Experimental results • Database • AR face database • This face database was created by Aleix Martinez and Robert Benavente in the Computer Vision Center (CVC) at the U.A.B • It contains over 4,000 color images • 126 people's faces (70 men and 56 women) • 26 image per individual • Images feature frontal view faces • Different facial expressions, illumination conditions, and occlusions (sun glasses and scarf).

20. Experimental results • Tested the four graph representations • Two parameters have to be set • C is used in the first step of the keypoints matching • C induce a restrictive selection of matching points • K for the three graph representations • It corresponds to the number of neighbors considered in the graph definition • The dissimilarity measure gives a coefficient that can be interpreted as a confidence criterion on the fact that two faces belong to different individuals • Gives the experimental results on the AR face database on a ROC curve

21. Experimental results • We use the FAR (False Acceptance Rate) and FRR (False Rejection Rate) criteria to analyse the behavior of an algorithm The representation based on a Completely connected graph (CCG) gives the worst result This means that the selection of edges is necessary Mutual k-nearest neighbor graph (MKNG) with k=5 and C=0.8 provides

22. Conclusion • We proposed in this paper a face authentication method based on a similarity measure • A graph representation based on SIFT invariant keypoints is used • The graph definition allows us to take into account their spatial relationships