1 / 27

Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA

Image Enhancement Method via Blur and Noisy Image Fusion. Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA. Outline. Introduction Related work Proposed method Photometric calibration Luminance fusion Color fusion

tea
Download Presentation

Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Enhancement Method via Blur and Noisy Image Fusion Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA

  2. Outline • Introduction • Related work • Proposed method • Photometric calibration • Luminance fusion • Color fusion • Experiments and examples • Conclusions 2 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  3. Introduction • Image capturing in dim light is challenging, especially for miniature cameras • Tuning camera parameters tradeoffs between different quality factors • Aperture increase: ensures more light but reduces depth of field • ISO sensitivity increase: amplifies the noise • Exposure time increase: may result in motion blur 3 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  4. Related work • De-noise a single image captured with short exposure (and high ISO): • e.g., [Donoho and Johnstone, 1994], [Starck, Candes, and Donoho, 2002], [Portilla, et al. 2003] • Due to short exposure and quantization color may not be recovered • De-blurring a single image captured with long exposure time (and small ISO) • Blind image de-convolution [You and Kaveh '96 & '99], [Chan and Wong '98], [Fergus ’06], [Shan ‘08] • Computationally complex, plus reliability problems • Assume spatially invariant blur PSF • Fusing blurry / noisy image pairs [Tico, et al. ‘06], [Yuan. et al. ‘07] • Avoid blind de-convolution by estimating the blur PSF from the two input images • Assume blur PSF is spatially invariant • Using additional hardware • Extra video camera for motion estimation [Ben-Ezra, Nayar ‘04] • Flash: effective only for close objects, and change the mood of the scene • Opto-mechanical stabilizer systems: inefficient for long exposure times 4 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  5. The proposed method Time • Short exposed frame: noisy but not affected by motion blur • Long exposed frame: better colors, less noise, but blurry Short exposed: dark, noisy and less color Long exposed: good color but blurry 5 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  6. The block diagram of the proposed solution Photo- calibration Image fusion Registration 6 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  7. Photometric calibration • Build the joint histogram (comparagram) 7 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  8. Photometric calibration (cont’d) • Identify most likely correspondences(xi,yi)between pixel values in the two images 8 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  9. Photometric calibration (cont’d) • Estimate the Brightness Transfer Function (BTF) 9 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  10. Photometric calibration (cont’d) Calibration curves Calibrated short exposed Short exposed Long exposed 10 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  11. Image fusion Photo- calibration Image fusion Registration 11 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  12. Luminance fusion • Formally we can write the following model for the two images • By applying an orthogonal wavelet transform the model becomes • Taking advantage of the de-correlation in the wavelet domain we propose a MMSE diagonal estimator of the form where • By minimizing the mean square error results the expression for the optimal weight 12 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  13. Luminance fusion (cont’d) • Prefer the noisy image near edges, and the blurry image in smooth areas • Edges can be detected based on the difference between the two images 13 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  14. Different levels of blur 29.81dB (5x5) 24.61dB 25.58dB (11x11) 22.78dB (21x21) 33.76dB (3x3) 35.09dB 34.03dB 33.50dB 33.40dB 14 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  15. Different levels of noise 24.61dB 27.80dB (7x7) 22.12dB 20.21dB 28.13 35.32dB 33.62dB 32.47dB 31.48dB 15 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  16. Comparative simulations Curvelet HT 24. 92dB 27.43dB Wavelet HT 24.73dB 31.19dB 21.02dB (7x7) 29.44dB Wiener 31.05dB [Portilla, et al. 2003] 16 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  17. Comparative simulations (cont’d) • The CPU times measured on Intel Core 2 Duo 2.20GHz Proposed method Portilla, et al. 2003 CPU Time: 0.8 sec 31.05dB CPU Time: 31 sec 31.19dB 17 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  18. Noise variance estimation • Noise variance in the photo-calibrated image = noise variance in the short exposed image = brightness transfer function = short exposed image in pixel Short exposed - calibrated Noise variance 18 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  19. Color fusion • Emphasize colors from long exposed image except the areas where the long exposed image is saturated • Weighting functions • Saturation weight function (left) • Blurriness weight function (right) 19 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  20. Color fusion (cont’d) • Example of color weighting in accordance to the two rules (saturation and blurriness ) Short exposed Long exposed Color weight 20 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  21. Example Short exposed Long exposed Result 21 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  22. Example (local blur) Short exposed Long exposed Result 22 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  23. Example (local blur) De-blurring Long exposed Result 23 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  24. Low Light Imaging Photometric aligned short exposed Long exposed: 1 sec, ISO 200 Short exposed: 1/30 sec, ISO 100 Output 24 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli 24

  25. Conclusions • We proposed a new approach of image restoration by fusing two differently degraded images • Short exposed image affected by noise • Long exposed image that may be affected by motion blur • In contrast to previous blurred/noisy image fusion our approach is not applying de-convolution on the blurry image • The main advantages: • Can deal with spatially variant blur due to parallax, or object motion • Lower computational complexity • Since the proposed approach is not dependent of blur spatial invariance it can be used also for fusing images with different aperture • Small aperture image, affected by noise but capturing a large depth of field • Large aperture image, less noisy but affected by blur due to narrow depth of field 25 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  26. Thank you! 26 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

  27. References • De-noise a single image captured with short exposure (and high ISO) • Several de-noising methods available in the literature, e.g., • D. L. Donoho and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika, vol. 81, pp. 425–455, 1994. • Jean-Luc Starck, Emmanuel J. Candes, and David L. Donoho, “The Curvelet Transform for Image Denoising,” IEEE Trans. on Image Processing, vol. 11, no. 6, pp. 670–684, 2002. • J. Portilla, V. Strela, M. Wainwright, E.P. Simoncelli, “Image denoising using scale mixtures of Gaussians in the wavelet domain”, IEEE Trans. on Image Processing 12, No. 11, 1338–1351, 2003. • Some are too complex a mobile device computational power • De-blurring a single image captured with long exposure time (and small ISO) • Blind image de-convolution • Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image”, SIGGRAPH, 2008. • R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, and W.T. Freeman, “Removing camera shake from a single photograph”, SIGGRAPH, 2006. • High complexity and insufficient robustness for consumer applications • Using additional camera for motion estimation • M. Ben-Ezra, S.K. Nayar, "Motion-based motion deblurring", IEEE Trans. on PAMI, 26, No. 6, 689-698, 2004 • Using specially designed CMOS sensors • X.Liu, A. Gamal, "Synthesis of High Dynamic Range Motion Blur Free Image From Multiple Captures", IEEE Trans. on Circuits and Systems I: Findamental Theory and applications, vol. 50, no. 4, 530-539, 2003. • Fusing blurry / noisy image pairs • Marius Tico, Mejdi Trimeche, and Markku Vehvil¨ainen, “Motion blur identification based on differently exposed images”, ICIP, 2006. • Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum, “Image deblurring with blurred/noisy image pairs,” SIGGRAPH 2007. 27 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

More Related