Active Shape Models: Their Training and Applications Cootes, Taylor, et al.

1. Active Shape Models:Their Training and ApplicationsCootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation Today, I will present and review the paper entitled � � by Cootes, Taylor, et al. In this paper, they propose a deformable model which constrains deformation to the allowable variability present in the training set. This model can be used to identify or classify objects in an image.Today, I will present and review the paper entitled � � by Cootes, Taylor, et al. In this paper, they propose a deformable model which constrains deformation to the allowable variability present in the training set. This model can be used to identify or classify objects in an image.

2. Other Deformable Models �Hand Crafted� Models Articulated Models Active Contour Models � �Snakes� Fourier Series Shape Models Statistical Models of Shape Finite Element Models

3. Motivation � Prior Models Lack of practicality Lack of specificity Lack of generality Nonspecific class deformation Local shape constraints

4. Goals of Active Shape Model (ASM) Automated Searches images for represented structures Classify shapes Specific to ranges of variation Robust (noisy, cluttered, and occluded image) Deform to characteristics of the class represented �Learn� specific patterns of variability from a training set Understanding of variability mechanisms(theoretical model of variability) are insufficientUnderstanding of variability mechanisms(theoretical model of variability) are insufficient

5. Goals of ASM (cont�d.) Utilize iterative refinement algorithm Apply global shape constraints Uncorrelated shape parameters Better test for dependence? If model parameters are correlated (general case) over training set, it does not effectively restrict the shapes which can be generated to ones similar to those found in the original training set Place limits on each parameter constrain the model to generate shapes similar to those in the training set. If model parameters are correlated (general case) over training set, it does not effectively restrict the shapes which can be generated to ones similar to those found in the original training set Place limits on each parameter constrain the model to generate shapes similar to those in the training set.

6. Point Distribution Model (PDM) Captures variability of training set by calculating mean shape and main modes of variation Each mode changes the shape by moving landmarks along straight lines through mean positions New shapes created by modifying mean shape with weighted sums of modes

7. PDM Construction

8. Labeling the Training Set Represent example shapes by points Point correspondence between shapes Boundaries, internal features, external features (center of concavity) Landmark points can be used to describe spatially related objects, or components of same object Type 3 good enough (interpolating splines to generate boundaries)Boundaries, internal features, external features (center of concavity) Landmark points can be used to describe spatially related objects, or components of same object Type 3 good enough (interpolating splines to generate boundaries)

9. Aligning the Training Set xi is a vector of n points describing the the ith shape in the set: xi=(xi0, yi0, xi1, yi1,��, xik, yik,��,xin-1, yin-1)T Minimize: Ej = (xi � M(sj, ?j)[xk] � tj)TW(xi � M(sj, ?j)[xk] � tj) Weight matrix used: Compares equivalent points from different shapes, they must be aligned Aligned by scaling, rotation, and translation Minimizes a weighted sum of squares of distances between equivalent points on different shapes Given two similar shapes xi and xj, ?j and sj, and a translation (txj, tyj) can be chosen to map xi onto M(sj, ?j)[xj]+tjCompares equivalent points from different shapes, they must be aligned Aligned by scaling, rotation, and translation Minimizes a weighted sum of squares of distances between equivalent points on different shapes Given two similar shapes xi and xj, ?j and sj, and a translation (txj, tyj) can be chosen to map xi onto M(sj, ?j)[xj]+tj

10. Alignment Algorithm Align each shape to first shape by rotation, scaling, and translation Repeat Calculate the mean shape Normalize the orientation, scale, and origin of the current mean to suitable defaults Realign every shape with the current mean Until the process converges Convergence test � examine average difference between transformation required to align each shape to the recalculated mean and the identity transformConvergence test � examine average difference between transformation required to align each shape to the recalculated mean and the identity transform

11. Mean Normalization Ensures 4N constraints on 4N variables Equations have unique solutions Guarantees convergence Independent of initial shape aligned to Iterative method vs. direct solution Mean is scaled rotated, and translated so it matches the first shape Or arbitrary default such as choosing origin at center of gravity, orientation such that particular part of shape is at top, scale such that distance between two points is oneMean is scaled rotated, and translated so it matches the first shape Or arbitrary default such as choosing origin at center of gravity, orientation such that particular part of shape is at top, scale such that distance between two points is one

12. Aligned Shape Statistics PDM models �cloud� variation in 2n space Assumptions: Points lie within �Allowable Shape Domain� Cloud is hyper-ellipsoid (2n-D) Figure 5 � the coordinates of some vertices of the aligned resistor shapes are plotted with the mean shape Capture the relationships between the positions of the individual landmark points in cloud Figure 5 � the coordinates of some vertices of the aligned resistor shapes are plotted with the mean shape Capture the relationships between the positions of the individual landmark points in cloud

13. Statistics (cont�d.) Center of hyper-ellipsoid is mean shape Axes are found using PCA Each axis yields a mode of variation Defined as , the eigenvectors of covariance matrix , such that ,where is the kth eigenvalue of S �mode of variation� � how landmarks move together as shape varies Eigenvector corresponding to largest eigenvalue describes longest axis of ellipsoid, thus most significant mode of variation in variables used to describe S Variance of eigenvector equals corresponding eigenvalue�mode of variation� � how landmarks move together as shape varies Eigenvector corresponding to largest eigenvalue describes longest axis of ellipsoid, thus most significant mode of variation in variables used to describe S Variance of eigenvector equals corresponding eigenvalue

14. Approximation of 2n-D Ellipsoid Most variation described by t-modes Choose t such that a small number of modes accounts for most of the total variance

15. Generating New Example Shapes Shapes of training set approximated by: , where is the matrix of the first t eigenvectors and is a vector of weights Vary bk within suitable limits for similar shapes Parameters are linearly independent Limits by examining distribution of parameter values required to generate training set Variance bk over training set is lambda k If each parameter is normally distributed, Dm is Chi-Square -> choose Dmax appropriatelyParameters are linearly independent Limits by examining distribution of parameter values required to generate training set Variance bk over training set is lambda k If each parameter is normally distributed, Dm is Chi-Square -> choose Dmax appropriately

16. Application of PDMs Applied to: Resistors �Heart� Hand �Worm� model

17. Resistor Example 32 points 3 parameters capture variability

18. Resistor Example (cont.�d) Lacks structure Independence of parameters b1 and b2 Will generate �legal� shapes What about b3?What about b3?

19. Resistor Example (cont.�d)

20. Resistor Example (cont.�d)

21. Resistor Example (cont.�d)

22. �Heart� Example 66 examples 96 points Left ventricle Right ventricle Left atrium Traced by cardiologists

23. �Heart� Example (cont.�d) B3 and b4?B3 and b4?

24. �Heart� Example (cont.�d) Varies Width A single model represents several shapes AND spatial relationshipsA single model represents several shapes AND spatial relationships

25. Hand Example 18 shapes 72 points 12 landmarks at fingertips and joints

26. Hand Example (cont.�d) 96% of variability due to first 6 modes First 3 modes Vary finger movements

27. �Worm� Example 84 shapes Fixed width Varying curvature and length Failure of PDM due to bending and relative rotational effectsFailure of PDM due to bending and relative rotational effects

28. �Worm� Example (cont.�d) Represented by 12 point Breakdown of PDM

29. �Worm� Example (cont.�d) Curved cloud Mean shape: Varying width Improper length

30. �Worm� Example (cont.�d) Linearly independent Nonlinear dependence Can not choose parameters independently and get a shape similar to those in training setCan not choose parameters independently and get a shape similar to those in training set

31. �Worm� Example Effects of varying first 3 parameters: 1st mode is linear approximation to curvature 2nd mode is correction to poor linear approximation 3rd approximates 2nd order bending Ideally, first and second order curvature are first 2 modes Fitting straight lines to curved �clouds�Ideally, first and second order curvature are first 2 modes Fitting straight lines to curved �clouds�

32. PDM Improvements Automated labeling 3D PDMs Nonlinear PDM Polynomial Regression PDMs Multi-layered PDMs Hybrid PDMs Chord Length Distribution Model Approximation problem

33. PDMs to Search an Image - ASMs Estimate initial position of model Displace points of model to �better fit� data Adjust model parameters Apply global constraints to keep model �legal�

34. Adjusting Model Points

35. Calculating Changes in Parameters Initial position: Move X as close to new position (X + dX) Calculate dx to move X to dX Update parameters to better fit image Not usually consistent with model constraints Residual adjustments made by deformation

36. Model Parameter Space Transforms dx to parameter space giving allowable changes in parameters, db Recall: Find db such that - yields Update model parameters within limits

37. Applications Medical Industrial Surveillance Biometrics

38. ASM Application to Resistor 64 points (32 type III) Adjustments made finding strongest edge Profile 20 pixels long 5 degrees of freedom 30, 60, 90, 120 iterations

39. ASM Application to �Heart� Echocardiogram 96 points 12 degrees of freedom Adjustments made finding strongest edge Profile 40 pixels long Infers missing data (top of ventricle)

40. ASM Application to Hand 72 points Clutter and occlusions 8 degrees of freedom Adjustments made finding strongest edge Profile 35 pixels long 100, 200, 350 iterations

41. Conclusions Sensitivity to orientation of object in image to model Sensitivity to large changes in scale? Sensitive to outliers (reject or accept) Sensitivity to occlusion Quantitative measures of fit Overtraining Occlusion, cluttering, and noise Dependent on boundary strength Real time Extension to 3rd dimension Gray level PDM

42. MR of Brain2 Improves ASM Tests several model hypotheses Outlier detection adjustment/removal 114 landmark points 8 training images Model structures of brain together Model brain structures

43. MR Brain (cont.�d)

44. MR Brain (cont.�d)

45. MR Brain (cont.�d)

48. References 1 � Cootes, Taylor, et al., �Active Shape Models: Their Training and Application.� Computer Vision and Image Understanding, V16, N1, January, pp. 38-59, 1995 2 - Duta and Sonka, �Segmentation and Interpretation of MR Brain Images: An Improved Active Shape Model.� IEEE Transactions on Medical Imaging, V17, N6, December 1998

49. �Hand Crafted� Models Built from subcomponents (circles, lines, arcs) Some degree of freedom May change scale, orientation, size, and position Lacks generality Detailed knowledge of expected shapes Application specific Minimize energy function. Adjust parameters in parameter space along path of steepest decent.Minimize energy function. Adjust parameters in parameter space along path of steepest decent.

50. Articulated Models Built from rigid components connected by sliding or rotating joints Uses generalized Hough transform Limited to a restricted class of variable shapes

51. Active Contours � �Snakes� Energy minimizing spline curves Attracted toward lines and edges Constraints on stiffness and elastic parameters ensure smoothness and limit degree to which they can bend Fit using image evidence and applying force to the model and minimize energy function Uses only local information Vulnerable to initial position and noise Splines are piecewise polynomial functions of order d Splines are piecewise polynomial functions of order d

52. Spline Curves Splines are piecewise polynomial functions of order d Sum of basis functions with applied weights Spans joined by knots

53. Fourier Series Shape Models Models formed from Fourier series Fit by minimizing energy function (parameters) Contains no prior information Not suitable for describing general shapes: Finite number of terms approximates a square corner Relationship between variations in shape and parameters is not straightforward

54. Statistical Models of Shape Register �landmark� points in N-space to estimate: Mean shape Covariance between coordinates Depends on point sequence

55. Finite Element Models Model variable objects as physical entities with internal stiffness and elasticity Build shapes from different modes of vibration Easy to construct compact parameterized shapes