Prakash Chockalingam

1. Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Prakash Chockalingam Committee Members Dr Stan Birchfield (chair) Dr Robert Schalkoff Dr Brian Dean Clemson University

2. Tracking Overview Tracker Tasks Feature Descriptors Object Detection Object Model Tracking Framework Update / Learning Mechanism Color Gradients Texture Shape Motion Manual Segmentation Feature Points Template Contour Active Appearance Probability Densities Mean Shift Pixel-wise Classification Optical Flow Filtering techniques No Update Adaboost Expectation Maximization Re-weighting Strategy

3. Approach • Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. • Contour Extraction: Contour is extracted using a discrete implementation of level sets • Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. • Update Mechanism: The parameters of all the Gaussians are updated based on tracked data • Results

4. Tracking Framework Bayesian Formulation: Contour at time t Previously seen contours Image data of all frames Assuming conditional independence among pixels, Feature vector

5. Object Modeling f2 Gaussian Mixture Model (GMM): y ? Strength Image: f1 >0 for Foreground <0 for Background

6. Strength Image GMM Linear Classifier Single Gaussian

7. Strength Image (contd…) Single Gaussian Linear Classifier … Individual Fragments Final Strength Strength Without Spatial Information

8. Topics • Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. • Contour Extraction: Contour is extracted using a discrete implementation of level sets • Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. • Update Mechanism: The parameters of all the Gaussians are updated based on tracked data • Results

9. Contour Extraction (strength image) (frontier) > 0 Inside < 0 Outside Energy Functional: Implicit representation of growing region Likelihood term (Strength image) Regularization term

10. Contour Extraction (contd…) (Region to be shrunk) (Region already grown) (Region to be grown) (Region that need not be considered)

11. Contour Extraction (contd…) such that Contraction x’ x Dilation x x’ such that

12. Contour Extraction (contd…) Expand Remove interior points Contract Remove exterior points

13. Contour Extraction (contd…) Likelihood Final Region

14. Topics • Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. • Contour Extraction: Contour is extracted using a discrete implementation of level sets • Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. • Update Mechanism: The parameters of all the Gaussians are updated based on tracked data • Results

15. Region Segmentation Mode-seeking region growing algorithm: • do { • Pick a seed point that is not associated to any fragment • Grow the fragment from the seed point based on the similarity of the • pixel and its neighbor’s appearance • Stop growing the fragment if no more similar pixels are present in the neighborhood of the fragment • } until all pixels are assigned Seed point: Eigen values of 3x3 RGB covariance matrix where

16. Region Segmentation (contd…) • Pick the minimum element in S. Create a region to hold the pixel and add the neighbors in a fixed window. • Compute Mean μj and Covariance Σj of the region. • Likelihood: • Grow the region as before with two additional steps: • Update μj, and Σj, as a new pixel is added • Remove the corresponding element in S if a pixel is added • Continue above steps if S is not empty. Mahalanobis distance Configurable parameter Initial region

17. Region Segmentation (contd…) Graph-Based Mean-Shift Region Growing

18. Region Segmentation (contd…) Graph-Based Mean-Shift Region Growing

19. Topics • Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. • Contour Extraction: Contour is extracted using a discrete implementation of level sets • Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. • Update Mechanism: The parameters of all the Gaussians are updated based on tracked data • Results

20. Update Mechanism f2 • Update parameters of existing fragments • Detect fragment occlusion • Find new fragments f1 Initial Model Fragment Association Initial Frame

21. Update Mechanism (contd…) Updating parameters of existing fragments: Weight computed by comparing Mahalanobis distance Initial Model (function of past and current values)

22. Update Mechanism (contd…) Occluded fragments: If a fragment is associated with less than 0.2% of the image pixels, then the fragment is declared as occluded. Finding new fragments: Helps in handling self-occlusion

23. Spatial Alignment The spatial parameters are updated using the motion vectors from Joint Lucas-Kanade approach Joint Lucas-Kanade Lucas-Kanade

24. Algorithm summary • Initial frame: • The user marks the object to be tracked. • The target object and background scene are segmented based on their appearance similarity. • The target object and background scene are modeled using a mixture of Gaussians where each Gaussian correspond to a fragment in the joint feature-spatial space • Subsequent frames: • Update the spatial parameters of GMM using the motion vectors of Joint Lucas-Kanade • Each pixel is classified into either foreground or background by generating a strength map using the Gaussian mixture model (GMM) of the object and background. • The strength map is integrated into a discrete level set formulation to obtain accurate contour of the object. • Using the tracked data, the appearance parameters of the GMM are updated.

25. Topics • Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground. • Contour Extraction: Extract contour using a discrete implementation of level sets • Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image. • Update Mechanism: The parameters of all the Gaussians are updated based on tracked data • Results

26. Experimental Results Elmo Sequence Monkey Sequence

27. Experimental Results (Contd…) Person Sequence Fish Sequence

28. Experimental Results: Self-Occlusion Without Self-Occlusion Module With Self-Occlusion Module

29. Conclusion • A tracking framework based on modeling the object as mixture of Gaussians is proposed • An efficient discrete implementation of level sets is employed to extract contour. • A mode-seeking region growing algorithm is used to segment the image. • A simple re-weighting strategy is proposed to update the parameters of Gaussians. • Future Directions: • Incorporate shape priors. • Utilize the extracted shapes to learn more robust priors. • An offline or online evaluation mechanism during the initialization phase. • Adding global information into the region segmentation process. • Automating the object detection and initialization.

30. Questions ?

31. Thank you !