Introduction to Multimedia

1. Introduction to Multimedia Wen-Shyang Hwang KUAS EE.

2. Outline • What is Multimedia? • Multimedia Authoring and Tools • Graphics and Image Data Representations • Color in Image and Video

3. What is Multimedia? • Ze-Nian Li: • involves multiple modalities of text, audio, images, drawings, animation, and video. • consists of a wide variety of topics: • Multimedia processing and coding: multimedia content analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc. • Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases. • Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems, multi-modal interaction and integration: web-everywhere devices, multimedia education including Computer Supported Collaborative Learning, and design and applications of virtual environments.

4. Multimedia and Hypermedia • HyperMedia: • nonlinearly read, by following links that point to other parts of the document, or to other documents • include text, graphics, images, sound, and video. • World Wide Web (WWW): best example of hypermedia application. • Multimedia: • computer information represented through audio, graphics, images, video, and animation.

5. Multimedia Software Tools • Music Sequencing and Notation • Cakewalk, Cubase, Macromedia Soundedit • Digital Audio • Cool Edit, Sound Forge, Pro Tools • Graphics and Image Editing • Adobe Illustrator, Adobe Photoshop, Macromedia Fireworks, Macromedia Freehand • Video Editing • Adobe Premiere, Adobe After Effects, Final Cut Pro • Animation • Java3D, DirectX, OpenGL, 3D Studio Max, Softimage XSI, Maya, RenderMan, GIF Animation Packages • Multimedia Authoring • Macromedia Flash, Macromedia Director, Authorware, Quest

6. Video Transitions • Video transitions: to signal “scene changes”. • Cut: an abrupt change of image contents. • Wipe: a replacement of pixels in a region. • Cross Dissolve: every pixel is affected gradually. D = (1− (t))A+ (t)B, where A,B are the color 3-vectors for video A and video B. Here,  (t) is a transition function • Dither Dissolve:Determined by (t), increasingly more and more pixels in video A will abruptly change to video B.

7. VRML (Virtual Reality Modelling Language) • VRML is an interpreted language; it was the first method available for displaying a 3D world on the World Wide Web. • It has the capability to put colored objects into a 3D environment. • The basic geometric shapes can be combined to create more complex objects.

8. 8-bit Gray-level Images • Each pixel has a gray-value between 0 and 255. • Bitmap: Two-dimensional array of pixel values to represent the graphics/image data. • Image resolution refers to the number of pixels in a digital image • Frame buffer: Hardware used to store bitmap. (Video card) • 8-bit image: a set of 1-bit bit-planes where each plane consists of a 1-bit representation of the image. Grayscale image of Lena Monochrome 1-bit Lena image

9. Dithering • used to calculate patterns of dots such that values from 0 to 255 correspond to patterns that are more and more filled at darker pixel values, for printing on a 1-bit printer. • to replace a pixel value by a larger pattern, such that the number of printed dots approximates the ink used in analog (newspaper photos). • In the dither matrix, first re-map image values in 0..255 into the new range 0..4 by (integer) dividing by 256/5. • if the pixel value is 0, print nothing • if the pixel value is 4, print all four dots.

10. Image Data Types • The most common data types for graphics and image file formats: 24-bit color and 8-bit color. • Most image formats incorporate some variation of a compression technique due to the large storage size of image files. Compression techniques can be classied into either lossless or lossy. • In a color 24-bit image, each pixel is represented by three bytes, usually representing RGB. (a) High-resolution color and separate R, G, B color channel images. (a) 24-bit color image. (b, c, d): R, G, and B color channels for this image

11. 8-bit Color Images (If space is concern) • use the concept of a lookup table to store color information. • Basically, the image stores not color, but instead just a set of bytes, each of which is actually an index into a table with 3-byte values that specify the color for a pixel with the lookup table index. 3-dimensional histogram of RGB colors in forestre.bmp" The count in each bit in terms of intensity and pseudocolor. 8-bit color image in GIF format

12. Color Look-up Tables (LUTs) • Store only the index or code value for each pixel, e.g., if a pixel stores the value 25, the meaning is to go to row 25 in a color LUT. • A Color-picker consists of an array of fairly large blocks of color such that a mouse-click will select the color indicated. • In reality, a color-picker displays the palette colors associated with index values from 0 to 255. Color-picker for 8-bit color: each block of the color- picker corresponds to one row of the color LUT

13. GIF (Graphics Interchange Format) File Format • Limited to 8-bit (256) color images only, • supports interlacing - successive display of pixels in widely-spaced rows by a 4-pass display process. • Screen Descriptor comprises a set of attributes that belong to every image in the file.

14. Color Science • Visible light is an electromagnetic wave in the range 400 nm to 700 nm. • Eye works with lens focusing an image onto the retina which consists of an array of rods and three kinds of cones. • For lower light levels,rods come into play and produce a image in shades of gray. • For higher light levels, cones each produce a signal, three kinds of cones are most sensitive to red (R), green (G), and blue (B) light. Spectral power distribution (Spectrum). The relative power in each wavelength interval for typical outdoor light on a sunday day.

15. Spectral Sensitivity of the Eye • Eye is most sensitive to light in the middle of visible spectrum. The sensitivity is also a function of wave-length. (Statistically, Blue is the favorite color of humans.) • the overall sensitivity as a dashed line is called luminous-efficiency function. It usually denoted V() and is formed as the sum of Red, Green, and Blue. • The rod sensitivity curve looks like V() but is shifted to the red end of the spectrum. • The achromatic channel produced by cones is approximately proportional to 2R+G+(B/20). Spectral Power Distribution (SPD) or Spectrum of daylight

16. Gamma Correction • The light emitted is in fact roughly proportional to the voltage raised to a power; this power is called gamma, with symbol . • append a prime to signals that are gamma-corrected by raising to the power (1/ ) before transmission. Thus we arrive at linear signals: Darker values are displayed too dark

17. Color-Matching Functions • Without knowing eye-sensitivity curves, a technique in psychology for matching a combination of basic R, G, and B lights to a given shade. • The three basic lights are called color primaries. • To match given color, separately adjust the brightness of three primaries using a set of controls.

18. Color Models in Video • Colors models used for stored, displayed, and printed images. • Video Color Transforms • North America TV uses the matrix transform YIQ. • Europe TV uses the matrix transform YUV • Digital Video uses the matrix transform YCbCr • YUV • Chrominance refers to the difference between a color and a reference white at the same luminance. (color differences U, V) original color image

19. Color Models in Video (YIQ) • YIQ • YIQ is used in NTSC color TV broadcasting. Gray pixels generate zero (I;Q) chrominance signal. I and Q are a rotated version of U and V .

20. Color Models in Video (YCbCr) • often simply written YCbCr - closely related to YUV transform • The YCbCr transform is used in JPEG image compression and MPEG video compression.