Part-based representation for the retrieval of 3D graphical models

1. Part-based representation for the retrieval of 3D graphical models Alexander G. Agathos PhD Defense Department of Product and Systems Design Engineering, University of The Aegean Institute of Informatics and Telecommunications, NCSR “Demokritos”

2. Contents • 3D Object Retrieval • Retrieval of 3D Articulated Objects using a graph-basedrepresentation • Segmentation Methodologies1. Mesh segmentation using feature point and core extraction (Katz et. al. 2005) 2. Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007) 3. A new shape decomposition scheme for graph-based representation (Kim et. al. 2005) 4. A polygonal mesh partitioning algorithm based onprotrusion conquest for perceptual 3D shape description(Valette et. al. 2005) • Recognition by Components • Proposed Segmentation algorithm – Results • Graph-based 3D Retrieval – Results • Innovations and Future work

3. Query result Query Database 3D Object Retrieval • 3D object retrieval is the process which retrievesfrom adatabase of 3D objects those that match besta 3D object query using a measure of similarity

4. Graph Segmentation Meaningful parts Representation Query Model Attributed Relational Graph (ARG) Retrieval of 3D Articulated Objects using a graph-basedrepresentation Retrieval : Matching of the query ARG with the ARGs stored in the Database

5. More meaningful results : The critical points utilize the human ability to distinguish the main particulars of an object Segmentation Methodologies • Region Growing : The segmentation regions are generated with the expansion of seed elements • Watersheds : The segmentation regions are generated by the simulation of flood filling of • geographic surfaces • Reeb Graph Method : The segmentation regions are generated with the use of Reeb Graphs • Model based Method : The segmentation regions are generated with the use of a model to simulate • the concavities of the mesh • Skeleton based Method : The segmentation regions are generated by the use of the skeleton of the • model • Clustering Method : The segmentation regions are extracted using usually the k-means algorithm • on the faces of the mesh • Spectral Analysis : The segmentation regions are extracted using spectral analysis on the faces • of the mesh • Explicit Boundary Extraction Methods : The segmentation regions are extracted indirectly by finding • first the segmentation boundaries of the mesh • Volumetric Methods : The segmentation regions are extracted using a volumetric function or • volumetric methods • Critical Points : The segmentation regions are extracted using the salient points of the mesh

6. MDS: Transform a mesh into a pose invariant representation Extraction of Salient Points: Spherical mirroring of the MDS transformed object Part extraction using the minimum cut algorithm Mesh segmentation using feature point and core extraction (Katz et. al. 2005) (Critical Points Method)

7. Voxelization Opening Body Class This morphological operator clears the object from protrusions Complement Branch Class A new shape decomposition scheme for graph-based representation (Kim et. al. 2005) (Volumetric Method) Application of the opening morphological operation using a ball-shaped shape element

8. A new shape decomposition scheme for graph-based representation Initial Decomposition • The radius of the sphere is determined by maximizing the weighted convexity : • A partis further split when its CMD yields at least two parts and each of them has only one adjacent part, it is split. Recursive Decomposition

9. A polygonal mesh partitioning algorithm based onprotrusion conquest for perceptual 3D shape description (Valette et. al. 2005)

10. Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007) (Volumetric Method) • Shape Diameter Function : . It is defined as the distance to the antipodal surface using an inward normal direction. In a discrete surface :

11. Recognition by Components Recognition of a 3D object is achieved by understanding its structure. According to the Recognition by Components (RBC) theory of Biederman human perception understands the structure of the 3D object by breaking it into parts and assigning to them basic volumetric primitives Specifically when an image of an object is painted on the retina, RBC assumes that a representation of the image is segmented-or parsed-into regions of deep concavity, particularly at cusps where there are discontinuities in curvature. Each segmented region is then approximated by one of a possible set of simple components called geons (for “geometricalions”) that can be modeled by generalized cones.

12. Proposed Segmentation algorithm • The proposed segmentation methodology is based on the premise that the 3D object consists of a main body (core) and its protrusible parts

13. Segmentation Flow Chart (Inspired from the general framework of Lin et. al.Visual salience-guided mesh decomposition) Salient point extraction stage Input: A mesh representing a 3D manifold Salient points grouping Partitioning boundary approximation for each salient representative Partitioning boundary refinement Core Approximation Until all salient representatives have been addressed

14. Protrusion function (Hilaga et. al.) : The protrusion function receives low values at the center of the object and high values at its protrusions : Geodesic neighborhood with radius Segmentation-Salient Point Extraction Stage Salient Points : Reside at the extrema of the 3D object. They are extracted by finding the local maxima of the protrusion function

15. Grouping : The salient points that are required to be grouped arethose which are close to each other in terms of geodesicdistance. Representative salient points : Segmentation-Salient Point Grouping The computed salient points often belong to sub-parts of the 3D object

16. Segmentation-Core Approximation Core (main body) approximation : An algorithm which approximates the main body of the object is the one that canacquire all the elements (vertices or faces) of the meshexcept those that belong to the protrusions of the mesh. In Our algorithm : The core approximation is extracted by usingthe minimum cost pathsbetween the representative salientpoints. The core approximation Algorithmexpands a set of vertices in ascending order of protrusionfunction value until the expanded set contains a fixedpercentage of all elements of the minimum cost paths (15%).

17. Segmentation-Core Approximation Basic philosophy of the core approximation algorithm All the points of the approximation are kept in a list named CoreList All the points of the Mesh are inserted in a priority queue A minimum cost path remains active if it contains less than 15% of the points of the core that belongs to it A salient representative remains active if there exist a minimum cost path to all other representatives that is active A point can be added to the CoreList if the minimum cost path from the nearest representative salient point is active Check if a salient representative becomes non active Check if a path becomes non active Extract a point from the priority queue Check if it can be added Until all representative salient points become non active

18. Segmentation-Core Approximation The Minimum Cost Paths span the protrusible parts. The selection of a percentage of them provides a high confidence that the core points will cover areas of the protrusible parts or being very close to the neighboring areas in which the real boundary is situated.

19. The partitioning boundary is the boundary between a protrusion and the main body of the mesh. Partitioning Boundary Detection Construction of closed boundaries which span the area containing the partitioning and are defined by a distance function D

20. Partitioning Boundary Detection • Construction of closed boundaries : Iso-contours generated bysetting a constant value Dc on a distance function D computed with cost:

21. Construction of closed boundaries in the Arithmetic interval : Partitioning Boundary Detection The core approximation has its boundaries near the actual boundaries of the distinct parts of the model. Taking advantage of this an area containing the partitioning boundary can be created. Boundary detection : Partitioning boundary approximation :

22. the point of the mesh of minimal distance to the salient points of the group the point of the core with the minimum distance from the first point p of the minimum cost path from smin to cmin mesh where d(smin,p) > 0.3dmin the isocontour generated by D passing from pthres the point of the constrained mesh with the minimum protrusion function Partitioning Boundary Detection • The selection of the representative of the group may lead to skewed closed boundaries :

23. Region C= Partitioning Boundary Refinement The partitioning boundary approximation is not constrained to the concavities where the true partitioning boundary passes, thus there is a need to refine the partitioning boundary approximation so that it passes also through the concave regions of the 3D object.

24. the faces of the dual graph, the edges of the dual graph the faces of C that share a common edge with A, B Partitioning Boundary Refinement

25. Segmentation-Results

26. Results

27. Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007)

28. Extended Core Mesh segmentation using feature point and core extraction (Katz et. al. 2005) Initial Core

29. A new shape decomposition scheme for graph-based representation

30. A polygonal mesh partitioning algorithm based onprotrusion conquest for perceptual 3D shape description (Valette et. al. 2005)

31. Graph-based 3D Retrieval

32. Attributed Relational Graph (ARG) Matching Unary attributes : Assigned to the nodes of the graph and express the parts geometrical characteristics. Binary attributes : Assigned to the edges of the graph and express the relationship between the parts Matching between two ARGs : Accomplished using the EMD similarity measure

33. EMD = Matching TheEMD measure is used to efficiently express the similarity oftwo signatures belonging to two different distributions in afeature space Signatures : , The set of weights can be considered as the piles of earth that needs to be transferred to the holesthat the other set of weights create in the feature space Each unit ofearth is transferred from pile i to hole j with cost : Total amount of earth transferred from pile i to hole j : The EMD measures the minimum amount of work requiredto transfer the piles of earth to the holes.

34. Matching The two ARGs are considered as the two signatures that need to be matched

35. Matching Let and be the two graphs to be matched Ground distance definition :

36. Unary attribute : Delete node : Binary attribute : Matching-Attribute assignment The following unary and binary attributes will be used : Spherical harmonic descriptor of Papadakis et. al. (2007) Kim et. al. Descriptor (MPEG7) (2004) Unary attributes defined by Papadakis et. al. descriptor. In this case no binary attributes are used. Delete node : Vector with zero entries

37. Matching-Papadakis et. al. Descriptor

38. Retrieval-Experimental Results The evaluation is based on the McGill Database of articulated objects which contains 255 models consisting of the following classes: ‘Spectacles’ ‘Ants’ ‘Ctabs’ ‘Hands’ ‘Humans’ ‘Octopuses’ ‘Pliers’ ‘Spiders’ ‘Teddy bears’ ‘Snakes’

39. Retrieval-Experimental Results • The goal of the experiments: • To demonstrate the superior performanceof the proposed retrieval methodology • against Kim et.al. [2004] and Papadakis et. al. [2008] • Second, the proposed retrieval methodology will be used in order to refine Papadakis • et. al. (2008) retrieval results • To demonstrate the efficiency of the proposed mesh segmentationagainst Kim et. al. • [2004] segmentation interms of retrieval accuracy Abbreviations • EMD-PPPT : The proposed retrieval methodology using the attributes of Papadakis et. al. (2007) • EMD-MPEG7 : The proposed retrieval methodology using the attributes of Kim et. al. (2004) • SMPEG7 : Kim et. al. (2004) retrieval methodology using the proposed segmentation algorithm • Hybrid : Papadakis et. al. (2008) retrieval methodology which uses a global descriptor • H-EMD-KIM-R : The refined retrieval methodology which uses Papadakis et. al. (2008) global descriptor and • the proposed retrieval methodology using Kim et. al. (2004) attributes • H-EMD-PPPT-R : The refined retrieval methodology which uses Papadakis et. al. (2008) global descriptor and • the proposed retrieval methodology using Papadakis et. al. (2008) attributes • MPEG7 : Kim et. al. (2004) retrieval methodology

40. Retrieval-Experimental Results • Evaluation is done using Precision-Recall (PR) Diagrams and the quantitative measures: • Nearest Neighbor (NN) • First Tier (FT) • Second Tier (ST) • Discounted Cumulative Gain (DCG) Nearest Neighbor : The percentage of queries where the closest match belongs to the query’s class First Tier : It measures the proportion of the first |C|-1 retrieved models that Belong to class C Second Tier : It measures the proportion of the first 2(|C|-1) retrieved models that Belong to class C Discounted Cumulative Gain : A statistic that weights correct results near the front of the list more than correct results later in the ranked list

41. Retrieval-Experimental Results

42. Retrieval-Experimental Results

43. Retrieval-Experimental Results

44. Retrieval-Experimental Results 1. The best precision results are those of EMD-PPPT 2. EMD-MPEG7 is the second in precision performance value 3. H-EMD-KIM-R and H-EMD-PPPT-R compared to Hybrid has been improved by 20% and 24% approximately respectively in the precision performance value 4. SMPEG7 retrieval curve is better than the MPEG7 retrieval curve by an average increase in the precision recall level of the order of 33%

45. Retrieval-Experimental Results

46. Retrieval-Experimental Results

47. Retrieval-Experimental Results

48. Retrieval-Experimental Results

49. Retrieval-Experimental Results

50. Innovations and Future Enhancements • A new segmentation algorithm has been proposed New algorithm for grouping of points New algorithm for core extraction New algorithm for partitioning boundary extraction • A new retrieval algorithm has been proposed New metric and its use in the EMD algorithm Future Enhancements The arithmetic interval is dependant of Dcoremin. The segmentation algorithm can become hierarachical and used also for matching purposes