Forward-Backward Correlation for Template-Based Tracking

1. Forward-Backward Correlation for Template-Based Tracking Xiao Wang ECE Dept. Clemson University

2. Introduction • Object tracking: An important computer vision problem • Security and surveillance • Medical therapy • Retail space instrumentation • Video abstraction • Traffic Management • Video editing • Template-Based Tracking • A classic technique • Idea of template-based tracker

3. Related Work • Jepson et al. Robust Online Appearance Models for Visual Tracking, CVPR 2001 • Ho et al. Visual Tracking Using Learned Subspaces, CVPR 2004 • Davis et al. Tracking Rigid Motion using a Compact-Structure Constraint, ICCV 1999 • Avidan et al. Ensemble Tracking, CVPR 2005

4. Backward Correlation Module Gradient Module Update Template Overview of the Approach Forward Correlation Module Textured Background? Next Frame No Yes

5. Template-Based Tracking • Template Selection: first frame vs. previous frame • Motion Model: • Similarity transformation scaling displacement

6. Template-Based Tracking • Cross Correlation: SSD reference image displacement search image

7. Template-Based Tracking • Similarity measure: s(Δx, Δy) • Correlation Coefficient: c(Δx, Δy) Mean of template Mean of image region

8. Forward Correlation • Forward Correlation: • Reference frame: previous frame • Goal: find transformation vector (dx, dy, α) • Approach: cross-correlation Put into correlation coefficient framework • Template Update:

9. Forward Correlation Out-of-plane rotation • Drifting Problem: • Forward correlation approximates rotation with translation. • Forward correlation does not check the reliability of the template. • We need a mechanism to question the assumption of forward correlation. Previous frame Current Frame

10. Backward Correlation • Consider our problem as motion segmentation • Goal of motion segmentation • Why is motion segmentation of video sequences difficult? • Under-constrained • Occlusion & Disocclusion • Image noise • A two-step procedure: • Determine the motion vectors associate with each pixel or feature point. • Group pixels or feature points that perform common motion.

11. Backward Correlation • Kanade-Lucas-Tomasi (KLT) feature tracker • Idea: minimize the dissimilarity of feature windows in two images • Assumption: mutual correspondence

12. Backward Correlation • Now consider the dissimilarity under the template window. • Decompose the template window into 2 partitions: foreground background • Rewrite dissimilarity as: low high

13. Backward Correlation • Background is moving at a different velocity than the foreground. • Foreground pixels have similar velocity and generate low SSD error. • Correlation between background pixels using foreground velocity generates high SSD error. • Goal: group foreground pixels which are moving at similar velocities Reference frame I(x) Difference image D(x)=[I(x)-J(x+d)]2 Current image J(x)

14. Backward Correlation • Formulations for backward correlation Set of template candidates Correlation coefficient (likelihood)

15. Untextured Backgrounds • Limitation of backward correlation: • Fails if background has little texture. • Why? --- Examine the assumption. • Backward correlation has no reason to prefer the foreground to the background which is untextured. low Also low if untextured

16. Untextured Backgrounds • Likelihood of backward correlation: textured vs. untextured Foreground Template containing background pixels Textured background Untextured background

17. Gradient Module • Motivation: • Seek a module focusing on the boundary of the target being tracked. • An edge-based segmentation problem. • Prior information: an ellipse model. • Gradient Module: Unit vector normal at pixel i Intensity gradient

18. Combining Modules • Gradient module and backward correlation module have orthogonal failure modes. • Textured or Untextured? • Use sum of the gradient magnitude of the neighborhood region. • Combination of forward correlation module and backward correlation module is straightforward. • Combination of forward correlation module and gradient module requires the normalization of the matching scores.

19. Combining Modules • Normalize the matching score (likelihood): • Finial state is decided by:

20. Adaptive Scale • Vary the scale by ± 10 percent during search process. • Filter the result to avoid oversensitive scale adaptation. • Comaniciu et al. Kernel-based object tracking,TPAMI 2003 Size of the best state given by the alg. Size of the object in the previous frame

21. Experimental Results:Cluttered Background • Traditional template-based tracker slides off target:

22. Experimental Results:Cluttered Background • Our algorithm remains locked onto target:

23. Experimental Results:Cluttered Background • Tracking error plot: Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed) Error in x direction Error in y direction

24. Experimental Results:Untextured Background • Tracking results of traditional template-based tracker:

25. Experimental Results:Untextured Background • Tracking results of our algorithm:

26. Experimental Results:Occlusion

27. Experimental Results:Tracking a vehicle

28. Conclusion • Presented an extension to template-based tracking. • Achieved robustness to out-of-plane rotation. • Effective tracking in both textured and untextured environment. • Remaining challenges: • Robustness when scale changes. • Use motion discontinuities to improve performance. • Analysis of parameter sensitivity for untextured backgrounds.

29. Questions?

30. Thank You!