4 .5 Image and Video Retargeting

1. It is not easy to shrink photos for the presentation on a smaller device (e. g., a mobile phone). Options are • scaling down the photo proportionally • cropping(cutting off the photo on the edges) • content retargeting by cutting out horizontal or vertical lines with minimal energy somewhere in the image Goal • Keep the important visual content of the images • We will look at seam carving techniques for images and for videos. 4.5 Image and Video Retargeting

2. Scaling • Scaling (zooming) reducesthesizeof an imagelinearly, withoutlookingatthecon-tent. • „Letterboxing“ isusediftheaspectratioofthescaledimagedoes not fit on thetargetscreen. • Example:

3. Cropping • Croppingmeansthecuttingof an imagealongitsborders, againwithoutlooking at thecontent. Example:

4. Content-based Image Reduction (1) Reducethewidthoftheimageby 40 %. original imageenergy in theimage

5. Content-based Image Reduction (2) • Fromeveryrow, removethenpixelswiththesmallestenergyvalues. orignalimage 200 pixels per rowwith minimal energyremoved

6. ColumnwiseReduction • Remove thencolumnswiththesmallestenergyvalues. orignalimage 200 columnswith minimal energyvaluesremoved

7. Seam Carving (1) Seam Carving • Remove a path (“seam”) of pixels with low relevance for the content of an image • Each removed seam reduces the image size by one column or row. A vertical seam changes the image width, a horizontal seam changes the image height. Source: ShaiAvidan and Ariel Shamir: Seam Carving for Content-Aware Image Resizing. ACM SIGGRAPH, 2007

8. Seam Carving (2) • A vertical seam is an 8-connected path from the top to the bottom of the image con-taining exactly one pixel from each row. Definition S: vertical seam H: image height

9. Seam Carving (3) Advantage of seams

10. low energy seams (white) are removed first Seam Carving (4) algorithmreduce image size while (image size > destination size) do Find optimal seam in image Remove pixels of optimal seam

11. Optimal Seams How do we identify the optimal seam? • Use an energy function E to select the pixels • Minimize the energy of the seam • Use dynamic programming to solve this minimization problem • Example Energy function • Absolute gradient magnitude of adjacent path pixels

12. The Example Energy Function The gradientisdefinedasthedifferenceoftheintensity (brightness) of a pixelfromitseightneighbors: Thisenergyfunctionisbased on theassumptionthattheintensityvaries in im-portantregionsoftheimagemorethan in lessimportantregions.

13. 2 5 1 4 1 2 3 4 1 2 3 3 5 4 4 1 Example • Exampleforthecomputationof optimal seams 2 5 1 1 4 3 3 3 4 5 4 5 6 6 7 9 8 9 7 7

14. Back to Our Example image with 200 seamsremoved orignalimage

15. Modified Energy Function Source: Hwang, Daw-Sen and Chien, Shao-Yi. Content-Aware Image Resizing using Percep-tualSeam Carving with Human Attention Model. IEEE Conference on Multimedia and Expo, 2008.

16. Seam Carving for Videos (1) First approach: Use seam carving on each frame separately  video becomes blurred and shaky original seams carved out

17. sourcenode sink node frame N edges: energybetween pixels frame 1 time Seam Carving for Videos (2) Second Approach Video defines a 3D space-time volume • remove 2D seam manifolds (seam pixels are connected in 3D) • use graph cuts (max-flow min-cut) to detect the optimal seam manifold

18. Third Approach: Fast Seam Carving (1) • Idea • Create one image that aggregates the pixel values/energy values of all frames (process each shot separately) • Detect 1D seam in the aggregated image • Map this seam to all frames

19. Fast Seam Carving (2) Approach • Use image registration techniques to estimate camera motionbetween adjacent frames • Compensate camera motion and create background image • Detect optimal seams in the background image • Use inverse camera motion to transform optimal seam back to all frames.

20. Fast Seam Carving (3) • Step 1: Camera Motion Compensation • Identify feature points (Harris with sub-pixel refinement) • Similarity measure: sum of absolute differences • Greedy-based assignment of corresponding features • Calculate camera model parameters (RANSAC) • Randomly draw four corresponding features • Calculate parameters of the camera model • Get the number of inliers and outliers • Go back to step1. • To describe camera motion, use the projective camera model:

21. Fast Seam Carving (4) Step 2: Aggregate frames into one background image

22. Fast Seam Carving (5) Advantages • Seams are robust: pixels of optimal seam represent the same visual back-ground in all frames. • Algorithm is fast: seams are detected in a 2D image (not in a 3D space-time cube). • Problems • Foreground objects • Seams of the background image are not necessarily included in all frames (e.g., when the camera pans)

23. Video Example scaling seam carving

24. Improved Line Preservation • Seamcarvingtypicallyintroducesartefactsforstraightlines orignalimage seamscarved out

25. The EffectofSeamsCrossing a Line a line in an image seamscrossingtheline after removaloftheseams

26. ModifiedEnergyFunction • Idea: Use a modifed energy function around the point where a seam has crossed a line • Increase the energy values in a Gauß curve around that point. seamcutting a line modifiedenergyfunctionandnewseam energyfunctionmodifiedagain thirdseam Themodifed energy function prevents multiple seams from crossing the line next to each other. Quelle: Johannes Kiess, Stephan Kopf, Benjamin Guthier and Wolfgang Effelsberg: Seam Carving with Improved Edge Preservation. Proc. of IS&T/SPIE Electronic Imaging, San Jose, 2010.

27. EffectofImproved Line Preservation original imageseamcarvingseamcarvingwithimprovedlinepreservation

28. Conclusions • The visual quality of images and videos adapted with seam carving is high. • Seam carving is optimal for images and videos with a small number of important regions. • The computational effort and memory requirements of seam carving for videos can be very high. • Two improvements for seam carving were developed by our team in Mannheim.