Texture Optimization for Example-based Synthesis

1. Texture Optimization for Example-based Synthesis Vivek KwatraIrfan EssaAaron BobickNipun Kwatra

2. Image vs. Texture

3. Example-based Texture Synthesis InputExample

4. Related Work • Local region-growing method -Pixel-based -Patch-based • Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field ….. • Physical Simulation et al

5. Example-based Texture Synthesis • Dynamic texturing (animation/video) • Greater control: external knobs, e.g., flow, shape, illumination • Texture quality evaluation and refinement

6. OutputSequence Motivation + InputTexture InputFlow Field

7. Limitations of previous work • Pixel-based can not maintain global structure of the texture inconsistencies • Patch-based have no control over individual pixel values • Global methods: pixel-to-pixel interactions insufficient to capture large scale structures complex formulations difficult to optimize

8. Texture Energy • Pixel-by-pixel comparison of source and target not possible • Compare texture “elements” • Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood

9. Texture Energy Z(source) X(target frame)

10. Energy for Single Neighborhood Z(source) p(pixel) X(target frame)

11. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

12. Energy for Single Neighborhood Z(source) Xp(neighborhood) X(target frame)

13. Energy for Single Neighborhood (nearest neighbor) Zp Z(source) Xp(neighborhood) X(target frame)

14. Energy for Single Neighborhood (nearest neighbor) Zp Z Xp(neighborhood) Texture Energy(single neighborhood) X

15. Texture Energy for Entire Image Z X

16. Texture Energy for Entire Image Z X =  individual neighborhood energy

17. Texture Synthesis 1.Initialize texture X 2.Find closest input neighborhood ZP corresponding to each output neighborhood XP 3.Update X to be texture that minimizes the energy 4.Repeat iteratively until set {ZP} stops changing

18. Texture Synthesis

19. Gradient-based Energy D is the differentiation operator and µ is a relative weighting coefficient

20. Muti-level Synthesis • Multi-resolution • Multi-scale/multiple neighborhood sizes Reasons • a good initialization of the texture • avoid undesirable local minima • avoid inconsistencies • capture large scale structures

21. Texture Energy Graph • Random Initialization • Multiple • Resolution Levels • Neighborhood Sizes • Progressively refined output

22. Controllable Synthesis

23. Flow-guided Texture Animation • Flow consistency • Perceived motion similar to flow • Texture similarity • Shape, size, orientation of texture elements similar to source Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

24. Source Texture Texture Similarity Naive Approach • Ignore Texture Similarity • Warp each frame via flow field Source Texture Warp TargetFrames X0 X1 Xn Texture Similarity FlowConsistency Flowing Target Target Flow

25. Naive Approach Texture structure not maintained Source Texture Texture Similarity FlowConsistency Flowing Target Target Flow

26. Warp X0 X1 Xn Instead of… Warp TargetFrames

27. Warp X0 X1 Xn Instead of… Warp TargetFrames

28. SynthesizedFrame X– X Warp Correct W WarpedFrame “Correct” Operation Warp + Correct TargetFrames

29. “Correct” Operation Link to original goals TargetFrames X– X Correct Warp W

30. “Correct” Operation Link to original goals TargetFrames X– X Correct Flow Consistency Warp W

31. “Correct” Operation Link to original goals Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

32. Optimization of Energy Energy = Flow Energy + Texture Energy Source Texture Similarity TargetFrames X– X Correct Flow Consistency Warp W

33. Flow Energy 2D flow field frame i frame i+1 pixel (p,i) pixel (q,i+1)

34. Results: Texture Synthesis

36. Results: Texture Synthesis Input Wei-Levoy[Wei’00] Image Quilting[Efros’01] Graph-cuts[Kwatra’03] Texture Optimization[Kwatra’05]

37. Time Cost • Static texture synthesis • 5-10 iterations per resolution level • Flowing texture synthesis • 20-60 seconds per frame

38. Limitations • Optimization finds local minima • Blurry / misaligned texture elements • Texture scale should be compatible with flow field

39. Thanks a lot!

40. Pixel-based Methods • Compare local causal neighbourhoods Efros and Leung (ICCV ’99) Wei and Levoy (Siggraph 2000) Ashikhmin (I3D 2001) Input Output

41. Patch-based Methods • Copy patches of pixels rather than single pixels Chaos Mosaic, Xu et al, 1997 Patch-Based Sampling, Liang et al(ACM 2001) Image Quilting, Efros and Williams(Siggraph 2001)

42. Back