1 / 40

Projective Analysis for 3D Shape Segmentation

Projective Analysis for 3D Shape Segmentation. Yunhai Wang 1 Minglun Gong 1 ,2 Tianhua Wang 1,3 Hao (Richard) Zhang 4 Daniel Cohen-Or 5 Baoquan Chen 1,6. 1 Shenzhen Institutes of Advanced Technology. 3 Jilin University. 2 Memorial University of Newfoundland .

janna
Download Presentation

Projective Analysis for 3D Shape Segmentation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Projective Analysis for 3D Shape Segmentation Yunhai Wang1MinglunGong1,2Tianhua Wang1,3Hao (Richard) Zhang 4 Daniel Cohen-Or 5Baoquan Chen1,6 1Shenzhen Institutes of Advanced Technology 3Jilin University 2 Memorial University of Newfoundland 4 Simon Fraser University 5 Tel-Aviv University 6 Shandong University

  2. Segmentation of 3D shapes • One of the most fundamental tasks in shape analysis • Low-level cues (minimal rule; convexity) alone insufficient

  3. Knowledge-driven approach Unsupervised co-analysis [Sidi et al. 2011] Learning segmentation [Kalograkis et al. 10] Keys to success: amount & quality of labelled or unlabelled 3Ddata Joint segmentation [Huang et al. 2011] Active co-analysis [Wang et al. 2012]

  4. 3D data challenge: amount • How many 3D models of strollers, golf carts, gazebos, …? • Not enough 3D models = insufficient knowledge • Labeling 3D shapes is also a non-trivial task 380labeled meshes over 19object categories

  5. Many many more images About14 million images across almost 22,000object categories Labeling images is quite a bit easier than labeling 3D shapes

  6. 3D data challenge: quality Self-intersecting; non-manifold Incomplete Real-world 3D models (e.g., those from Tremble Warehouse) are often imperfect

  7. Projective shape analysis (PSA) • Treat a 3D shape as a set of projected binary images • Label these images by learning from vast amount of image data • Then propagate the image labels to the 3D shape • Alleviate various data artifacts in 3D, e.g., self-intersections

  8. Contributions • Joint image-shape analysis via projective analysis for semantic 3D segmentation • Utilize vast amount of available image data • Allowing us to analyze imperfect 3D shapes

  9. Contributions • Bi-class Symmetric Hausdorff distance = BiSH • Designed for matching 1D binary images • More sensitive to topology changes (holes) • Caters to our needs: part-aware label transfer

  10. Related works onshape-image hybrid processing Many works on 2D-3D fusion, e.g., for reconstruction [Li et al.11] Image-guided 3D modeling [Xuet al.11]

  11. Related works onprojective shape analysis Image-space simplification error [Lindstrom and Turk 10] Light field descriptor for 3D shape retrieval [Chen et al.03] We deal with the higher-level and more delicate task of semantic 3D segmentation

  12. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

  13. Data preparation: image labeling Labeling involves GrabCut and some user assistance

  14. Projecting input 3D shape • Assume all objects are upright oriented; they mostly are! • Project an input 3D shape from multiple pre-set viewpoints

  15. Retrieve labeled images • For each projection of the input 3D shape, retrieve top matches from the set of labelled images

  16. Select projections for label transfer • Select top (non-adjacent) projectionswith the smallest average matching costs for label transfer

  17. Label transfer Later … • Label transfer is done per corresponding horizontal slabs • Pixel correspondence straightforward

  18. Confidence map • Three terms based on image-level, slab-level, and pixel-level similarity: more similar = higher confidence • Label transfer is weighted by a confidence value per pixel

  19. Backprojection (2D-to-3D) • One primitive projects to multiple pixels in multiple images • Per-pixel confidence gathered over multiple retrieved images • Probabilistic map over input 3D shape: computed by integrating per-pixel confidence values over each shape primitive

  20. Graph cuts optimization • Final labeling of 3D shape: multi-label alpha expansion graph cuts based on the probabilistic map

  21. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

  22. The matching problem … … Projections of input 3D shape Database of (labeled) images • Goal: find shapes most suitable for label transfer and FAST! • Not a global visual similarity based retrieval • Want part-aware label transfer but cannot reliably segment Classical descriptors, e.g., shape context, interior distance shape context (IDSC), GIST, Zenike moments, Fourier descriptors, etc., do not quite fulfill our needs • Possibly complex topology (lots of holes), not just a contour • All upright orientated: to be exploited • Characteristics of the data to be matched

  23. Scan-line view Takes advantage of upright orientation

  24. Scan-lines to slabs Classical choice for distance: symmetric Hausdorff (SH) But not sensitive to topology changes; not part-aware • Cluster scan-lines into smaller number of slabs --- efficiency! • Hierarchical clustering by a distance between adjacent slabs

  25. Extend SH to consider both B and W! C SH(C,B)=2, SH(Cc, Bc)=2 B A SH(A,B)=2, SH(Ac, Bc)=10 B SH for only one class may not be topology-sensitive A bi-class SH distance is!

  26. Bi-class symmetric Hausdorff (BiSH) C SH(C,B)=2, SH(Cc, Bc)=2 BiSH(C,B) = 2 B A SH(A,B)=2, SH(Ac, Bc)=10 BiSH(A,B) = 10 B

  27. BiSH vs. SH BiSH is more part-aware: new slabs near part boundaries SH BiSH

  28. Piecewise linear warping • Slabs are scaled/warped vertically for better alignment • Another measure to encourage part-aware label transfer Warp Recolor Slabs of labeled image warped to better align with slabs in projected image Slabs recolored: many-to-oneslab matching possible

  29. Slab matching and image similarity • Dissimilarity between slabs: BiSH scaled by slab height • Slab matching allows linear warp: optimized by a dynamic time warping (DTW) algorithm • Dissimilarity between images: sum over slab dissimilarity after warped slab matching

  30. Outline • PSA for 3D shape segmentation • Results and conclusion • Region-based binary shape matching

  31. vs. learning mesh segmentation • Same inputs, training data (we project), and experimental setting • Models in [K 2010]: manifold, complete, no self-intersections • PSA allows us to handle any category and imperfect shapes

  32. Data • All input 3D shapes tested have self-intersections as well as other data artifacts • 11 object categories; about 2600 labeled images

  33. Complex topology Pavilion (465 pieces) Bicycle (704 pieces)

  34. Incomplete model: point clouds

  35. Timing • Number of selected projections: 5 – 10 • Number of retrieved images per projection: 2 • Matching two images (512 x 512) takes 0.06 seconds • Label transfer (2D-to-2D then to 3D): about 1 minute for a 20K-triangle mesh

  36. Conclusions • Demonstrated potential in labeling 3D models: imperfect, complex topology, over any category • Projective shape analysis (PSA): semantic 3D segmentation by learning from labeled 2D images

  37. Main advantages • Utilize the rich availability and ease of processing of photos for 3D shape analysis • No strong requirements on quality of 3D model

  38. Limitations • Inherent to data-driven: knowledge has to be in data • Relying on spatial and not feature-space analysis • Assuming upright; not designed for articulated shapes • Inherent limitation of 2D projections: they do not fully capture 3D info

  39. Future work • Additional cues from images and projections, e.g., color, depth, etc. • Apply PSA for other knowledge-driven analyses • Labeling 2D images is still tedious: unsupervised projective analysis

  40. Thank you! More results and data can be found from http://web.siat.ac.cn/~yunhai/psa.html

More Related