1 / 70

Visual 3D Modeling using Cameras and Camera Networks

Visual 3D Modeling using Cameras and Camera Networks. Marc Pollefeys University of North Carolina at Chapel Hill. Talk outline. Introduction Visual 3D modeling with a hand-held camera Acquisition of camera motion Acquisition of scene structure Constructing visual models Camera Networks

brooks
Download Presentation

Visual 3D Modeling using Cameras and Camera Networks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Visual 3D Modeling using Cameras and Camera Networks Marc Pollefeys University of North Carolina at Chapel Hill

  2. Talk outline • Introduction • Visual 3D modeling with a hand-held camera • Acquisition of camera motion • Acquisition of scene structure • Constructing visual models • Camera Networks • Camera Network Calibration • Camera Network Synchronization • Towards Active Camera Networks… • Conclusion Visual 3D Modeling using Cameras and Camera Networks

  3. What can be achieved? • Can we get 3D models from images? • How much do we need to know about the camera? • Can we freely move around? Hand-held? • Do we need to keep parameters fixed? Zoom? • What about auto-exposure? • What about camera networks? • Can we provide more flexible systems? Avoid calibration? • What about using IP-based PTZ cameras? Hand-held camcorders? • Unsynchronized or even asynchronous? Visual 3D Modeling using Cameras and Camera Networks

  4. Talk outline • Introduction • Visual 3D modeling with a hand-held camera • Acquisition of camera motion • Acquisition of scene structure • Constructing visual models • Camera Networks • Camera Network Calibration • Camera Network Synchronization • Towards Active Camera Networks… • Conclusion Visual 3D Modeling using Cameras and Camera Networks

  5. (Pollefeys et al. ’98) Visual 3D Modeling using Cameras and Camera Networks

  6. (Pollefeys et al. ’04) Video Key-frame selection More efficient RANSAC Fully projective Improved self-calibration Deal with dominant planes Bundle adjustment Polar stereo rectification Deal with radial distortion Faster stereo algorithm Deal with specularities Volumetric 3D integration Deal with Auto-Exposure Image-based rendering Visual 3D Modeling using Cameras and Camera Networks

  7. Feature tracking/matching • Shape-from-Photographs: match Harris corners • Shape-from-Video: track KLT features Problem: insufficient motion between consecutive video-frames to compute epipolar geometry accurately and use it effectively as an outlier filter Visual 3D Modeling using Cameras and Camera Networks

  8. Key-frame selection Select key-frame when F yields a better model than H • Use Robust Geometric Information Criterion • Given view i as a key-frame, pick view j as next key-frame for first view where GRIC(Fij)>GRIC(Hij) (or a few views later) (Torr ’98) model complexity bad fit penalty H-GRIC F-GRIC (Pollefeys et al.’02) Visual 3D Modeling using Cameras and Camera Networks

  9. P C1 l1 e1 e2 l2 Fundamental matrix (3x3 rank 2 matrix) C2 Epipolar geometry Underlying structure in set of matches for rigid scenes • Computable from corresponding points • Simplifies matching • Allows to detect wrong matches • Related to calibration Visual 3D Modeling using Cameras and Camera Networks

  10. Epipolar geometry computation:robust estimation (RANSAC) Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 count inliers, if not promising stop until (#inliers,#samples)<95% (generate hypothesis) (verify hypothesis) Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches Visual 3D Modeling using Cameras and Camera Networks

  11. Epipolar geometry computation geometric relations between two views is fully described by recovered 3x3 matrix F Visual 3D Modeling using Cameras and Camera Networks

  12. Initialize Motion (P1,P2 compatibel with F) Sequential Structure and Motion Computation Initialize Structure (minimize reprojection error) Extend motion (compute pose through matches seen in 2 or more previous views) Extend structure (Initialize new structure, refine existing structure) Visual 3D Modeling using Cameras and Camera Networks

  13. Dealing with dominant planar scenes (Pollefeys et al., ECCV‘02) • USaM fails when common features are all in a plane • Solution: part 1 Model selection to detect problem Visual 3D Modeling using Cameras and Camera Networks

  14. Dealing with dominant planar scenes (Pollefeys et al., ECCV‘02) • USaM fails when common features are all in a plane • Solution: part 2 Delay ambiguous computations until after self-calibration (couple self-calibration over all 3D parts) Visual 3D Modeling using Cameras and Camera Networks

  15. Refine Structure and Motion • Use projective bundle adjustment • Sparse bundle allows very efficient computation (2 levels) • Take radial distortion into account (1 or 2 parameters) Visual 3D Modeling using Cameras and Camera Networks

  16. * * projection constraints Self-calibration using absolute conic (Faugeras ECCV’92; Triggs CVPR’97; Pollefeys et al. ICCV’98; etc.) Euclidean projection matrix: Translate constraints on K through projection equation to constraints on * some constraints, e.g. constant, no skew,... Absolute conic projection: Upgrade from projective to metric Transform structure and motion so that*diag(1,1,1,0) Visual 3D Modeling using Cameras and Camera Networks

  17. Practical linear self-calibration (Pollefeys et al., ECCV‘02) Don’t treat all constraints equal after normalization! (relatively accurate for most cameras) (only rough aproximation, but still usefull to avoid degenerate configurations) when fixating point at image-center not only absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs., but also diag(1,1,1,a), i.e. real or imaginary spheres! Visual 3D Modeling using Cameras and Camera Networks

  18. Refine Metric Structure and Motion • Use metric bundle adjustment • Use Euclidean parameterization for projection matrices • Same sparseness advantages, also use radial distortion Visual 3D Modeling using Cameras and Camera Networks

  19. Mixing real and virtual elements in video Virtual reconstruction of ancient fountain Preview fragment of sagalassos TV documentary Similar to 2D3‘s Boujou and RealViz‘ MatchMover Visual 3D Modeling using Cameras and Camera Networks

  20. Intermezzo: Auto-calibration of Multi-Projector System (Raij and Pollefeys, submitted) hard because screens are planar, but still possible Visual 3D Modeling using Cameras and Camera Networks

  21. Visual 3D Modeling using Cameras and Camera Networks

  22. Stereo rectification • Resample image to simplify matching process Visual 3D Modeling using Cameras and Camera Networks

  23. Stereo rectification • Resample image to simplify matching process Also take into account radial distortion! Visual 3D Modeling using Cameras and Camera Networks

  24. Polar stereo rectification (Pollefeys et al. ICCV’99) Polar reparametrization of images around epipoles Does not work with standard Homography-based approaches Visual 3D Modeling using Cameras and Camera Networks

  25. General iso-disparity surfaces (Pollefeys and Sinha, ECCV’04) Example: polar rectification preserves disp. Application: Active vision Also interesting relation to human horopter Visual 3D Modeling using Cameras and Camera Networks

  26. Similarity measure (SSD or NCC) Optimal path (dynamic programming ) Stereo matching • Constraints • epipolar • ordering • uniqueness • disparity limit • disparity gradient limit • Trade-off • Matching cost • Discontinuities (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02) Visual 3D Modeling using Cameras and Camera Networks

  27. Hierarchical stereo matching Allows faster computation Deals with large disparity ranges Downsampling (Gaussian pyramid) Disparity propagation Visual 3D Modeling using Cameras and Camera Networks

  28. Disparity map image I´(x´,y´) image I(x,y) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y) Visual 3D Modeling using Cameras and Camera Networks

  29. Example: reconstruct image from neighbors Visual 3D Modeling using Cameras and Camera Networks

  30. Multi-view depth fusion (Koch, Pollefeys and Van Gool. ECCV‘98) • Compute depth for every pixel of reference image • Triangulation • Use multiple views • Up- and down sequence • Use Kalman filter Also allows to compute robust texture Visual 3D Modeling using Cameras and Camera Networks

  31. Real-time stereo on GPU (Yang and Pollefeys, CVPR2003) • Plane-sweep stereo • Computes Sum-of-Square-Differences (use pixelshader) • Hardware mip-map generation for aggregation over window • Trade-off between small and large support window (Demo GeForce4) 150M disparity hypothesis/sec (Radeon9700pro) e.g. 512x512x20disparities at 30Hz GPU is great for vision too! Visual 3D Modeling using Cameras and Camera Networks

  32. Dealing with specular highlights (Yang, Pollefeys and Welch, ICCV’03) Extend photo-consistency model to include highlights Visual 3D Modeling using Cameras and Camera Networks

  33. Visual 3D Modeling using Cameras and Camera Networks

  34. 3D surface model Depth image Texture image Triangle mesh Textured 3D Wireframe model Visual 3D Modeling using Cameras and Camera Networks

  35. Volumetric 3D integration (Curless and Levoy, Siggraph´96) Multiple depth images Volumetric integration Texture integration patchwork texture map Visual 3D Modeling using Cameras and Camera Networks

  36. Dealing with auto-exposure (Kim and Pollefeys, submitted) • Estimate cameras radiometric response curve, exposure and white balance changes • Extends prior HDR work at Columbia, CMU, etc. to moving camera brightness transfer curve robust estimate using DP auto-exposure fixed-exposure response curve model Visual 3D Modeling using Cameras and Camera Networks

  37. Dealing with auto-exposure (Kim and Pollefeys, submitted) Applications: • Photometric alignment of textures (or HDR textures) • HDR video Visual 3D Modeling using Cameras and Camera Networks

  38. Part of Jain temple Recorded during post-ICCV tourist trip in India (Nikon F50; Scanned) Visual 3D Modeling using Cameras and Camera Networks

  39. Example: DV video  3D model accuracy ~1/500 from DV video (i.e. 140kb jpegs 576x720) Visual 3D Modeling using Cameras and Camera Networks

  40. Unstructured lightfield rendering (Heigl et al.’99) demo Visual 3D Modeling using Cameras and Camera Networks

  41. Talk outline • Introduction • Visual 3D modeling with a hand-held camera • Acquisition of camera motion • Acquisition of scene structure • Constructing visual models • Camera Networks • Camera Network Calibration • Camera Network Synchronization • towards active camera networks… • Conclusion Visual 3D Modeling using Cameras and Camera Networks

  42. Camera Networks • CMU’s Dome, 3D Room, etc. • MIT’s Visual Hull • Maryland’s Keck lab, ETHZ’s BLUE-C and more • Recently, Shape-from-Silhouette/Visual-Hull systems have been very popular Visual 3D Modeling using Cameras and Camera Networks

  43. Camera Networks • Offline Calibration Procedure • Special Calibration Data • Planar Pattern • moving LED • Requires physical access to environment • Active Camera Networks • How do we maintain calibration ? Visual 3D Modeling using Cameras and Camera Networks

  44. An example P. Sand, L. McMillan, and J. Popovic. Continuous Capture of Skin Deformation. ACM Transactions on Graphics 22, 3, 578-586, 2003. • 4 NTSC videos recorded by 4 computers for 4 minutes • Manually synchronized and calibrated using MoCap system Visual 3D Modeling using Cameras and Camera Networks

  45. Can we do without explicit calibration? • Feature-based? • Hard to match features between very different views • Not many features on foreground • Background often doesn’t overlap much between views • Silhouette-based? • Necessary for visual-hull anyway • But approach is not obvious Visual 3D Modeling using Cameras and Camera Networks

  46. x2 x1 x’2 x’1 Multiple View Geometry of Silhouettes • Frontier Points • Epipolar Tangents • Points on Silhouettes in 2 views do not correspond in general except for projected Frontier Points • Always at least 2 extremal frontier points per silhouette • In general, correspondence only over two views Visual 3D Modeling using Cameras and Camera Networks

  47. Calibration from Silhouettes: prior work Epipolar Geometry from Silhouettes • Porril and Pollard, ’91 • Astrom, Cipolla and Giblin, ’96 Structure-and-motion from Silhouettes • Joshi, Ahuja and Ponce’95 (trinocular rig/rigid object) • Vijayakumar, Kriegman and Ponce’96 (orthographic) • Wong and Cipolla’01 (circular motion, at least to start) • Yezzi and Soatto’03 (only refinement) None really applicable to calibrate visual hull system Visual 3D Modeling using Cameras and Camera Networks

  48. Camera Network Calibration from Silhouettes (Sinha, Pollefeys and McMillan, submitted) • 7 or more corresponding frontier points needed to compute epipolar geometry for general motion • Hard to find on single silhouette and possibly occluded • However, Visual Hull systems record many silhouettes! Visual 3D Modeling using Cameras and Camera Networks

  49. Camera Network Calibration from Silhouettes • If we know the epipoles, it is simple • Draw 3 outer epipolar tangents (from two silhouettes) • Compute corresponding line homography H-T (not unique) • Epipolar Geometry F=[e]xH Visual 3D Modeling using Cameras and Camera Networks

  50. Let’s just sample: RANSAC • Repeat • Generate random hypothesis for epipoles • Compute epipolar geometry • Verify hypothesis and count inliers until satisfying hypothesis • Refine hypothesis • minimize symmetric transfer error of frontier points • include more inliers Until error and inliers stable (use conservative threshold, e.g. 5 pixels, but abort early if not promising) (use strict threshold, e.g. 1 pixels) We’ll need an efficient representation as we are likely to have to do many trials! Visual 3D Modeling using Cameras and Camera Networks

More Related