1 / 35

Parametric Motion and Tracking

CS 636 Computer Vision. Parametric Motion and Tracking. Nathan Jacobs. the plan. Questions and concerns? Recap Parametric motion Layered motion. Slides for today from Robert Pless and Ming - Hsuan Yang. review from last time…. Motion fields Optical flow Lucas- Kanade Point Tracking

fairly
Download Presentation

Parametric Motion and Tracking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CS 636 Computer Vision Parametric Motion and Tracking Nathan Jacobs

  2. the plan • Questions and concerns? • Recap • Parametric motion • Layered motion Slides for today from Robert Pless and Ming-Hsuan Yang

  3. review from last time… • Motion fields • Optical flow • Lucas-Kanade Point Tracking • Iterative Lucas-Kanade

  4. Examples of Motion Fields (a) (b) (a) Motion field of a pilot looking straight ahead while approaching a fixed point on a landing strip. (b) Pilot is looking to the right in level flight. Computer Vision, Robert Pless

  5. Examples of Motion Fields (a) (b) (c) (d) (a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background. Computer Vision, Robert Pless

  6. The brightness constancy constraint • How many equations and unknowns per pixel? • One equation, two unknowns • Intuitively, what does this constraint mean? • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown gradient (u,v) • If (u, v) satisfies the equation, so does (u+u’, v+v’) if (u’,v’) (u+u’,v+v’) edge

  7. Lucas-Kanade flow • How to get more equations for a pixel? • Basic idea: impose additional constraints • most common is to assume that the flow field is smooth locally • one method: pretend the pixel’s neighbors have the same (u,v) • If we use a 5x5 window, that gives us 25 equations per pixel!

  8. Iterative Refinement • Iterative Lucas-Kanade Algorithm • Estimate velocity at each pixel by solving Lucas-Kanade equations • Warp H towards I using the estimated flow field • - use image warping techniques • Repeat until convergence

  9. estimate update Initial guess: Estimate: Optical flow: iterative estimation x x0 (usingd for displacement here instead of u)

  10. estimate update Initial guess: Estimate: x x0 Optical flow: iterative estimation

  11. estimate update Initial guess: Estimate: Initial guess: Estimate: x x0 Optical flow: iterative estimation

  12. Optical flow: iterative estimation x x0

  13. Affine Perspective 3D rotation Translation 2 unknowns 6 unknowns 8 unknowns 3 unknowns Motion models

  14. Example: affine motion • Substituting into the B.C. equation Each pixel provides 1 linear constraint in 6 global unknowns Least square minimization (over all pixels):

  15. Parametric motion i.e., the product of image gradient with Jacobian of the correspondence field

  16. Parametric motion the expressions inside the brackets are the same as the cases for simpler translation motion case

  17. Other 2D Motion Models

  18. Generalizing Lucas-Kanade extends to all differentiable warp functions! Lucas-Kanade 20 Years On: A Unifying Framework, Simon Baker and Iain Matthews

  19. to matlab… to explore linear image approximations % load data im = im2double(rgb2gray(imread('peppers.png'))); [gxgy] = gradient(im); sz = size(im); [X Y] = meshgrid((1:sz(2)) -sz(2)/2, (1:sz(1)) -sz(1)/2); motionModel = 'translation'; motionModel = 'affine'; % make jacobian images % i x j x Np x 2 switchmotionModel case 'translation’ Np = 2; imJacobian = zeros([szNp 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; case 'affine’ Np = 6; imJacobian = zeros([szNp 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; imJacobian(:,:,3,1) = X; imJacobian(:,:,4,1) = Y; imJacobian(:,:,5,2) = X; imJacobian(:,:,6,2) = Y; end imSteepest = sum(bsxfun(@times, imJacobian, cat(4, gx, gy)), 4); tmpJacobian = scale2rgb(repmat(reshape(imJacobian, [sz 1 Np*2]), [1 1 3 1])); tmpSteepest = scale2rgb(repmat(reshape(imSteepest, [sz 1 Np]), [1 1 3 1])); figure(1); clf; montage(tmpJacobian, 'Size', [2 Np]) title({'Jacobian Images: change in the warpfieldw.r.t. parameters'}) figure(2); clf; montage(tmpSteepest, 'Size', [1 Np]) title('SteepestDescent Images: innerproductwith image to update parameters') [Tx Ty] = ndgrid(linspace(-1, 1, 20), linspace(-1,1,20)); p = randn(100, Np); p = bsxfun(@times, p, [1 1 .01 .01 .01 .01]); for ix = 1:size(p,1) % make the approximation using the steepest descent images (the first % order approximation of the image) figure(3); clf; subplot(121); imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; title(p(ix,:)) subplot(122) imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off;xlim([40 100]), ylim([160 210]); colormap gray pause(eps) end

  20. Motion representations How can we describe this scene?

  21. Block-based motion prediction • Break image up into square blocks • Estimate translation for each block • Use this to predict next frame, code difference (MPEG-2)

  22. Layered motion • Break image sequence up into “layers”:  = • Describe each layer’s motion

  23. Layered motion • Advantages: • can represent occlusions / dis-occlusions • each layer’s motion can be smooth • video segmentation for semantic processing • Difficulties: • how do we determine the correct number? • how do we assign pixels? • how do we model the motion?

  24. Layers for video summarization

  25. Background modeling (MPEG-4) • Convert masked images into a background sprite for layered video coding + + + + + + =

  26. What are layers? • Layers or Sprites • Alpha matting • Intensities • Velocities • Wang and Adelson, “Representing moving images with layers”, IEEE Transactions on Image Processing, 1994

  27. How do we form them?

  28. How do we form them?

  29. How do we estimate the layers? • compute coarse-to-fine flow • estimate affine motion in blocks (regression) • cluster with k-means • assign pixels to best fitting affine region • re-estimate affine motions in each region…

  30. Layer synthesis • For each layer: • stabilize the sequence with the affine motion • compute median value at each pixel • Determine occlusion relationships

  31. Results

  32. Why is are layers useful?

  33. summary • parametric motion: generalized Lucas-Kanade • layered motion models

  34. for next time • starting 3D, structure from motion • SzeliskiChap 7

More Related