1 / 21

Stereoscopic Video Overlay with Deformable Registration

Stereoscopic Video Overlay with Deformable Registration. Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns Hopkins University. The CASA Project. Today’s Surgical Assistant: A Simple Information Channel. The CASA Project. Preoperative Imagery.

sun
Download Presentation

Stereoscopic Video Overlay with Deformable Registration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Stereoscopic Video Overlay with Deformable Registration Balazs Vagvolgyi Prof. Gregory Hager CISST ERC Dr. David Yuh, M.D. Department of Surgery Johns Hopkins University

  2. The CASA Project Today’s Surgical Assistant: A Simple Information Channel

  3. The CASA Project Preoperative Imagery Virtual fixtures with da Vinci Robot Information Fusion with da Vinci Display Stereo surface tracking Task graph execution system Stereo tool tracking Tissue Classification HMM-based Intent Recognition Ultrasound Capabilities of a Context-Aware Surgical Assistant (CASA)

  4. The CASA Project Preoperative Imagery Information Fusion with da Vinci Display Stereo surface tracking Stereo tool tracking Developing a Context-Aware Surgical Assistant (CASA)

  5. Information Overlay • Problem setting: • Given pre-operative scandata from a suitable imagingmodality • Video sequence from a stereo endoscope • Add value • Overlay underlying anatomy on the stereo video stream (x-ray vision) • Include annotations or other information tied to imagery Key Problem: Nonrigid registration of organ surface to data [[ add kidney picture ]]

  6. Inputs: What Do We Know? • Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure • Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response • Stereo image stream- 3D surface can be reconstructed- texture information • A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]

  7. Outputs: What Do We Generate? • Position of 3D model registered to stereo image • Model deformed to the current shape of anatomy • Rendering a synthetic 3D view on the stereo stream • Everything done real-time Original Image Stereo Data Deformed Mesh

  8. image data 3D data disparity parameters All this in a flow chart 3D model stereo video stream 2D 3D 3D texturetracking Stereo imagepre-processing Building andoptimizingdisparity map DeformableRegistration to3D surface Imageoverlay Recognizingdeformations optical parameters

  9. Classical Stereo Vision: The Problem • Blocks of each image are compared using SAD • Optimization for each block independently on entire depth range • Very fast implementation (GPU) • Lousy results Small Vision Systemfrom Videre Design(w/o structured light):

  10. Solution #1: Lighting and Multi-Scale • Input images downsized to several scale levels (½, ¼, …) • Each scale processed with the same algorithm • Propagate coarse search results to the finer scale • Quality of disparity map is better • Even faster than single scale computation • Requires structured light SVL implementation(using structured light):

  11. Solution #2: Dynamic Programming • Solve a (spatially) global optimization with regularization • O(D) = min SAD(D) + Smooth(D) • GLOBAL optimum found in polynomial time

  12. Solution #2: Dynamic Programming • Defining the recursive cost function • Memoization • Finding lowest cost path, which is the disparity map (DMin red) Error Smoothness

  13. Dynamic Programming on Images • Minor issue: previous approach applies to scanline • Approximate DP applied to entire image- 3D disparity space (D):- Cost function (C):- Memoization (P):

  14. Dynamic Programming: Results

  15. Dynamic Programming: In Vivo Results Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm. Textured 3D Model Raw Disparity Map

  16. Surface to 3D Model Registration • Inputs: • point cloud from the stereo surface modeler • point cloud generated from a model or volume image • Outputs:- transformation to register the 3D model to the 3D surface

  17. Results: Rigid Registration Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer) Complete system (stereoplus registration) operatesat 5 frames/second

  18. From Rigid to Deformable • Calculate residual errors in z direction • Define a spring-mass system • Perform local gradient descent

  19. Deformable Registration Results Final registration error of < 1mm exceptfor the area where the tool enters the image

  20. The Language of Surgery Coming in CASA Tool Tracking Tissue Surface Classification

  21. Thank you!

More Related