Image Alignment by Image Averaging

1. Image Alignment byImage Averaging David Hong NCSSM, IE364 2008

2. Example Problem 1

3. Example Problem 2

4. Example Problem 3

5. Problem (Formal Statement)

6. Motivation Many Applications: • Special Effects (Movie) • Video Compression • Pattern Recognition • Image Stabilization (Digital Cameras) • Dead-reckoning (Mobile Robotics)

7. State of the Art Optical Flow Lucas-Kanade (1985) Optical Flow with Smoothness Constraint Horn-Schunck(1980) Phase Correlation

8. Lucas-Kanade U(X,Y) u(x,y,t) Y y x X

9. Lucas-Kanade Let us consider a point on the plane. Then the floor-coordinate is (X0,Y0) and the sensor-coordinate is (x,y) at time t. From there, we can see:

10. Lucas-Kanade Differentiating on time gives us:

11. Lucas-Kanade Expressing (x,y) in terms of (X0,Y0) and the sensor position (Xs,Ys,Θs) gives us:

12. Lucas-Kanade Putting the two together, we get: This is underdetermined!

13. Algorithm u’ u

14. Improvement by Iteration u u’ u’’’ u’’

15. Improvement by Iteration u’(x’,y’) u’ u’’(x’’,y’’) u’’ u

16. Improvement by Iteration

17. Improvement by Iteration u’ (x’0, y’0) (x’1, y’0) (x’, y’) (x’0, y’1) (x’1, y’1) Places u’ is defined Place we need to evaluate u’

18. Improvement by Iteration

19. Improvement by Iteration u’ u’ not defined! u’’ u

20. Improvement by Iteration u’ u’’ was not evaluated here u’’ Valid Region

21. Improvement by Iteration (i0,1’, j0,1’ ) (i-1,0’, j-1,0’ ) (i1,0’, j1,0’ ) Places we need to evaluate u’ (i0,-1’, j0,-1’ )

22. Performance of Algorithm Good Surface: Bad Surface: Algorithm Fails!

23. Performance of Algorithm Surface:

24. Performance of Algorithm Surface:

25. Performance of Algorithm Surface:

26. Assumptions Made • The Error Function is locally quadratic • The floor is linear

27. Weaknesses • Many Iterations • Inherent to Technique • “Fooled” by symmetry (Aliasing problem) • Inherent to Problem

28. Strengths • Accurate • Improvement by Iteration • Finds Error Function Root by Newton’s

29. Phase Correlation u(xu,yu) v(xv,yv) yu yv xv xu

30. Phase Correlation We consider the image to be like a 2-D wave. Then, displacement is simply a “phase shift” Rotation can similarly be found So, we “correlate” the “phases”

31. Phase Correlation

32. Phase Correlation

33. Phase Correlation

34. Phase Correlation

35. Weaknesses • Inaccurate on first iteration • Boundary Problem (Repetion Assumption) • High complexity • FFT is “O(nlogn)”

36. Strengths • Elegant • Makes a big leap • Works well on images with pattern • Separates displacement and rotation (DFT)

37. Image Averaging u(xu,yu) v(xv,yv) yu yv xv xu

38. Image Averaging • Find a “Center-of-Mass” of each image • Track the motion of the center-of-mass

39. Weaknesses • Boundary Problem (Average Point Moves) • Average is affected by small discretization issues

40. Strengths • Elegant • Makes a big leap • Very fast • Complexity of O(n) • Yields itself well to Improvement by Iteration • Using same technique as in Lucas-Kanade

41. Handling the Weaknesses • Here we decide to take an alternative approach • Separate displacements from rotation • Do this using FFT (as in Phase Correlation)

42. Handling the Weaknesses • We handle rotation first • Post-FFT, only rotation remains

43. Handling the Weaknesses

44. Weaknesses • We introduce an FFT ( O(nlogn) operation) • However, only requires 2 • Phase correlation requires up to 3 or 4

45. Future Work • Make Image Alignment Rigorous • Use complex numbers to notate displacement • Smoothness Constraint • Pre-processing the image • Condition for Convergence

46. Thank You!