1 / 52

Texture Optimization for Example-based Synthesis

Texture Optimization for Example-based Synthesis. Vivek Kwatra 1,2 Irfan Essa 1 Aaron Bobick 1 Nipun Kwatra 1. 1 GVU Center/College of Computing, Georgia Tech 2 Computer Science Dept., UNC Chapel Hill. Motivation. +. =. Input Texture. Input Flow Field. Output Sequence.

rhonda-cook
Download Presentation

Texture Optimization for Example-based Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Texture Optimization for Example-based Synthesis Vivek Kwatra1,2Irfan Essa1Aaron Bobick1Nipun Kwatra1 1GVU Center/College of Computing, Georgia Tech 2Computer Science Dept., UNC Chapel Hill

  2. Motivation + = InputTexture InputFlow Field OutputSequence

  3. Example-based Texture Synthesis

  4. Example-based Texture Synthesis • Dynamic texturing (animation/video) • Greater control: external knobs, e.g., flow, shape, illumination • Explicit texture quality evaluation and refinement

  5. Contributions I • Global Optimization for texture synthesis • Global texture quality metric • Iterative multi-scale optimization

  6. + = InputTexture InputFlow Field OutputSequence Contributions II • Flow-guided texture animation • Flow incorporated into optimization • Applications: fluid rendering, flow visualization

  7. Related Work (a selected sample) TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

  8. Related Work (a selected sample) Better Quality TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

  9. Related Work (a selected sample) Better Quality TextureOptimization[Kwatra’05] MRF Annealing[Paget’98] Greater Controllability Histogram Matching[Heeger’95] Local vs. Global Graphcut Textures[Kwatra’03] Non-parametricsampling[Efros’99,Wei’00] Image Quilting[Efros’01] Pixel vs. Patch

  10. Related Work • Flow-guided Synthesis • Advected Textures [Neyret’03] • Flow-based Video Editing [Bhat’04] • Optimization • Image-based Priors [Fitzgibbon’03] • Space-time Video Completion [Wexler’04]

  11. Flow-guided Texture Animation • Flow consistency • Perceived motion similar to flow • Texture similarity • Shape, size, orientation of texture elements similar to source Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

  12. Source Texture Texture Similarity Naive Approach • Ignore Texture Similarity • Warp each frame via flow field Source Texture Warp TargetFrames X0 X1 Xn Texture Similarity FlowConsistency Flowing Target Target Flow

  13. Naive Approach Texture structure not maintained Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

  14. Warp X0 X1 Xn Instead of… Warp TargetFrames

  15. Warp X0 X1 Xn Instead of… Warp TargetFrames

  16. SynthesizedFrame X– X Warp Correct W WarpedFrame Our Approach Warp + Correct TargetFrames

  17. “Correct” Operation Link to original goals TargetFrames X– X Correct Warp W

  18. “Correct” Operation Link to original goals TargetFrames X– X Correct Flow Consistency Warp W

  19. “Correct” Operation Link to original goals Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

  20. Optimization of Energy / Cost Energy = Flow Energy + Texture Energy Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

  21. Flow Energy TargetFrames X– X Warp Flow Energy W

  22. Texture Energy • Tricky! • Pixel-by-pixel comparison of source and target not possible • Compare texture “elements” • Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood

  23. Texture Energy Z(source) X(target frame)

  24. Energy for Single Neighborhood Z(source) p(pixel) X(target frame)

  25. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

  26. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

  27. Energy for Single Neighborhood (nearest neighbor) Zp Z(source) Xp(neighborhood) X(target frame)

  28. Energy for Single Neighborhood (nearest neighbor) Zp Z Xp(neighborhood) Texture Energy(single neighborhood) X

  29. Texture Energy for Entire Image Z X

  30. Texture Energy for Entire Image Z X =  individual neighborhood energy

  31. Optimization • Optimize Total Energy of target frame: • Initialize: X W(target frame  warped frame) • Iteratively improve target frame

  32. Optimization: Intuition Zp Texture Force Xp Source Target Frame Flow Force Wp Warped Frame

  33. Optimization: Intuition Zp Xp Blend Source Target Frame Wp Warped Frame

  34. Overlapping Neighborhoods Zp Xq Zq Blend Source Xp Target Frame Wp Warped Frame

  35. We are not done yet! Nearest neighbor may change Zp Xp Source Target Frame Wp Warped Frame

  36. Iterative Algorithm • Step 1 • Find Nearest Source Neighborhoods • Step 2 • Blend Source & Warped Neighborhoods • Repeat until convergence

  37. Iterative Algorithm • Step 1 • Find Nearest Source Neighborhoods • Minimize E( X ) w.r.t {Zp} • Approximate nearest-neighbors • Hierarchical Clustering [Johnson’67] • Tree-Structured Vector Quantization [Wei’00]

  38. Iterative Algorithm • Step 2 • Blend Source & Warped Neighborhoods • Minimize E( X ) w.r.t X • Set  Solve Linear System

  39. Robust Optimization • Quadratic Texture Energy Etsensitive to outlier neighborhoods • Non-quadratic Etmore robust • Et( X ) = pkXp – Zpkr, 0 < r < 2

  40. Robust Optimization • Quadratic Texture Energy Etsensitive to outlier neighborhoods • Non-quadratic Etmore robust • Et( X ) = pkXp – Zpkr, 0 < r < 2 • Solve weighted quadratic at each iteration Et( X ) = pkXp – Zpkr-2. kXp – Zpk2 = p p . kXp – Zpk2

  41. Minimize Texture Energy Alone • Novel Static Texture Synthesis Algorithm Source Texture Similarity X– X Correct Flow Consistency W

  42. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

  43. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

  44. Results: Texture Synthesis

  45. Results: Texture Synthesis Input Wei-Levoy[Wei’00] Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

  46. Results: Texture Synthesis Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

  47. Results: Flow-guided Synthesis

  48. Results: Timing • Static texture synthesis • 5-10 iterations per resolution level • Flowing texture synthesis • 20-60 seconds per frame

  49. + = InputTexture InputFlow Field OutputSequence Summary • Global Optimization for controllable texture synthesis • Explicit evaluation of texture quality • Iterative progressive refinement of texture • Flow-guided texture animation within optimization framework

More Related