Multi-Object Detection and Tracking from a Moving Platform

1. Multi-ObjectDetection and Tracking from a Moving Platform

2. 1-Analysis and detection: Registration across video group of frames (VGoF) Detection and segmentationof motion blobs (background models, shadow) 2-Representation and tracking: Video object representation (shape, color descriptors, geometric models) Object tracking (prediction, correspondence, occlusion resolution etc.) 3-Access and event modeling: Efficient data structures for video queries in high-dimensional feature space High-level event representation Tracking from a Moving Platform

3. Multi-Object Tracking 1. Detectmoving objects in stabilized frames. 2. Predictlocations of the current set of objects. 3. Matchpredictions to actual measurements. 4. Updateobject trajectories. 5. Updateimage stabilized ref coord system. Multi-object Detection and Tracking Unit Tracking VGoF Registration Into Common Coordinate System Moving Object Detection & Feature Extraction Object States Data Association (Correspondence) Update Trajectories Update Coord System Prediction Context

4. Dynamic State Estimation for Tracking System state State estimate Measurements Dynamic System Measurement System State Estimator State uncertainties System noise Measurement noise • System Errors • Agile motion • Distraction/clutter • Occlusion • Changes in lighting • Changes in pose • Shadow (Object or background models are often inadequate or inaccurate) • Measurement Errors • Camera noise • Framegrabber noise • Compression artifacts • Perspective projection • State Error • Position • Appearance • Color • Shape • Texture etc. • Support map

5. Motion Detection- 3D Spatiotemporal Volume Spatio-temporalvolume of hall monitor sequence: (a) Left entire volume, (b) Middle: cut taken at vertical position y0, (c) Right: Cut taken at vertical Position y1. Gerald Kuhne, “Motion-based segmentation and classification of Video Objects” Dissertation Univ. of Mannheim, 2002

6. Motion Detection - Structure and Flux Tensor Approach Typical Approach: threshold trace(J) Problem: trace(J) fails to capture the nature of gradient changes and results in ambiguities between stationary versus moving features Alternative Approach: Analyze the eigenvalues and the associated eigenvectors of J Problem: Eigen-decompositions at every pixel is computationally expensive for real time performance Proposed Solution: Flux tensor  time derivative of J

7. Motion Detection Flux Tensor vs Gaussian Mixture

8. Probabilistic Bayesian framework Features Used in Data Association:Proximity and Appearance-based Data Association Strategy:Multi-hypothesis testing Gating Strategies:Absolute and Relative Discontinuity Resolution:Prediction (Kalman filter), or Appearance models Filtering: Temporal consistency check and Spatio-temporal cluster check Multi-object Tracking Stages

9. Association Strategy • Multi-hypothesis testing with delayed decision - Many matches are kept with evidence-based pruning • Support for multiple interactions - one-to-one object matches, many-to-one, one-to-many, many-to-many, one-to-none, or none-to-one matches • Corresponding low-level object tracking events • Segmentation errors • Group interactions (merge/split) • Occlusion • Fragmentation • Entering object • Exiting object ObjectMatchGraph

10. MatchConfidence Computation Match confidence quantifies correspondence goodness-of-fit Confidence value has two components: • Similarity confidence (Confsim) • Separation confidence(Confsep) 1,j* is the closest competitor in terms of distance Conf(i,j) Link Nodei Nodej • bounding box • support map • centroid • area etc. • bounding box • support map • centroid • area etc.

11. Segment Source Split Merge Sink Inner Source-Split Sink-Merge Split-Merge Single Trajectory Segment Generation • Trace links in the ObjectMatchGraph to generate possible trajectory segments • SegmentList - Linked list of inner nodes (objects/cells) • Trajectory labeling - Extracted trajectory segments are labeled using a modified connected components labeling • Trajectory linking - Trajectories are formed by linking unfiltered segments sharing the same label. Trajectory ObjectMatchGraph

12. Data Hierarchy Node (Object-Region) Segment Macro segment Trajectory

13. Need for Local Registration

14. Exp Results: DARPA ET01 Video Frame #50 Registered Frame Motion Detection Results Foreground Mask Tracking Results

15. Exp Results - NGA Crystal View HD Video Frame #787 in Coord. #740 UPS c) Predictions d) After occlusion handling

16. Future Work - Trajectory Matching and Filtering • Establishing trajectory continuity (object ID matching) across moving coordinate systems • Customizing trajectory analysis for airborne video tracking with misregistration error, large platform motion, zooming, etc • Maintaining temporal consistency of trajectories • Removing periodic clustered trajectories • Resolving discontinuous trajectories • Trajectory display and visualization: video vs mosaic

17. Future Work – Performance Optimization and Tuning • Moving object detector filters • Flux tensor fixed optimal threshold learning or continuous adaptive thresholding • Morphological post processing filters • Real-time versus offline MATLAB (approximate): • Flux tensor detection 4sec/frame • Object tracking 2sec/frame (for around 10 objects) • Excluding I/O time

18. Frame-to-frame registration accuracy difficult to maintain across a hundred frames or more (few seconds of video) Reducing false motion trajectories due to registration errors due to scene structure Maintaining a common coordinate system for registering long airborne video sequence Tracking through large platform motion Dealing with large camera field-of-view changes Platform motion jitter Future Work - Near Term Performance Improvements

19. Filtering periodic motions produced by clutter, etc. Shadows(e.g. false detections, shape distortions, merges) Sudden illumination changes(e.g. due to cloud movements) Glare from specular surfaces(e.g. windshields, water surfaces) Perspective distortion(e.g. object size, shape and position) Trajectory gaps and distortion due to occlusion Poor video quality(e.g. low resolution, low color saturation) Future Work - Longer Term Performance Improvements