1 / 25

Multi-Media Retrieval

Multi-Media Retrieval. by Paul McGlade Modified by Shinta P. What is Multi-Media Retrieval?. The searching and retrieval of various multi-media (image, video, web). Typically consists of a query search against a database, usually called either digital libraries or digital archives.

kisha
Download Presentation

Multi-Media Retrieval

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multi-Media Retrieval by Paul McGlade Modified by Shinta P.

  2. What is Multi-Media Retrieval? • The searching and retrieval of various multi-media (image, video, web). • Typically consists of a query search against a database, usually called either digital libraries or digital archives. • Generally, multimedia databases also consist of textual data types.

  3. Tasks • Multimedia systems must solve at least two different tasks: • First, relevant items have to be identified. • Second, they have to be presented in such a way that the user can relate them to each other, and what is often more complicated, to the query.

  4. Problems • Multimedia data comparison is more difficult than textual data. • Different types of querying raises different types of problems. • The relevance of each aspect in the multimedia data must be weighted.

  5. Approaches / Solutions • Different approaches are explored for the comparison process: • Text-based • Region-based • Object-based • Various solutions have been created: • Query formulation • MRML • Image indexing

  6. Text-based • Index images using keywords or descriptions. • Advantages: • Easier to design and implement. • Uses surrounding text in a web page. • Disadvantages: • Often too expensive. • A picture can sometimes require many words. • Surrounding text may not describe picture.

  7. Region-based • Queries images using regions of the image. • Advantages: • Handles low-level queries. • Many features can be extracted. • Disadvantages: • Cannot handle high-level queries.

  8. Region-based Good  Bad 

  9. Object-based • Extracts objects from images first. • Advantages: • Handles object-based queries. • Reduce feature storage adaptively. • Disadvantages: • Object segmentation is very difficult. • User interface is complicated and not easily implemented.

  10. Object-based (cont’d)

  11. Blobworld • Blobworld is a system for content-based image retrieval. • By automatically segmenting each image into regions which roughly correspond to objects or parts of objects, we allow users to query for photographs based on the objects they contain. • Blobworld Site

  12. Query Formulation • Formulates a query for comparison against a database. • Query Formula example: SIMILARITY: look similar OBJECT: contains a bike OBJECT RELATIONSHIP: contains a dog near a person MOOD: a happy picture TIME/PLACE: Yosemite sunset

  13. MRML • Multimedia Retrieval Markup Language • MRML’s goal is to unify access to multimedia retrieval. • XML-based communication protocol. • Specified to standardize access to Multimedia Retrieval software components.

  14. MRML (cont’d) • Code example: <property  id = "p1"     type = "subset"     caption = "Weighting function"     visibility = "visible"     sendtype = "attribute"     sendname = "cui-weighting-function"     minsubsetsize = "1"     maxsubsetsize = "1" >       <property        id = "p2"           type = "setelement"           caption = "Best fully weighted"           visibility = "visible"           sendtype = "value"           sendvalue = "best-fully"           defaultstate = "selected" />       <property        id = "p3"           type = "setelement"           caption = "Classical IDF"           visibility = "visible"           sendtype = "value"           sendvalue = "classical-idf"           defaultstate = "unselected" /> </property> <mrml >       <get-server-properties /> </mrml> <mrml >       <get-algorithms   collection-id = "collection-1" /> </mrml>

  15. GIFT • GNU Image-Finding Tool is a Content Based Image Retrieval System (CBIRS). • Uses MRML. • Enables the user to query by example on images. • Relies purely on the content of the image. • GIFT Site

  16. Image Indexing • Process which analyzes an image and selects aspects of the image to compare in order to index the image with little user input. • Segments the image into various regions, and attaches words to each region.

  17. Image Indexing (cont’) Computer Predictions - male  cloth  female  fashion  environment   people  industry  fire  face  man  man-made Manual Category Annotation - super model people female cloth Computer Predictions - grass mare tiger horses cat buildings Manual Category Annotation - cat grass tiger

  18. A-Lip • Automatic Linguistic Indexing of Pictures system selects among 600 trained concepts to annotate images automatically. • On-line real-time image annotation demonstration is expected to be developed and made available later this year. • When released, will be able to submit your own images for automatic annotation. • A-Lip Site

  19. High-Level Tools • Some technical approaches to image comparison: • Wavelet comparisons. • Fast Image Segmentation. • IRM (Integrated Region Matching). • Fuzzy Matching.

  20. SIMPLIcity • Semantics-sensitive Integrated Matching for Picture Libraries. • Combine low-level statistical semantic classification with image retrieval. • Wavelet-based feature extraction for fast segmentation. • Integrated Region Matching (IRM). • SIMPLIcity Site

  21. Mengapa Image Retrieval Sulit? • Text Retrieval • Kata Adalah suatu unit, mudah diindex • Kata Memiliki arti semantik • Image Retrieval • Unit pberupa piksel, sulit diindex • Piksel tak memiliki arti • piksel membentuk pola representasi objek, kesulitan dalam segmentasi • Objek gambar tergantung banyak faktor

  22. Mengapa Image Retrieval Sulit? (Cont’) • Image Retrieval • Objek gambar tergantung banyak faktor • Sudut Pandang • Iluminasi • Bayangan • Dan komplikasi lainya (latar belakang, variasi warna, dll)

  23. Pencocokan Citra (Global Similarity) • Histogram Warna • Karakteristik Tekstur (region)

  24. Pencocokan Citra (Local Similarity) • Query By Example • Segmentasi Objek • Pencocokan • Caption Text • Similarity (warna, tekstur, bentuk) • Susunan Spatial (orientasi, posisi) • Teknik Khhusus (eg. Pengenalan Wajah)

  25. Conclusion • Since one query will return many false results, I believe more emphasis should be placed on the weighting of certain aspects of each image. • Some ideas: • Artistic tendencies could be taken into account when determining the relevance of an object in an image. • A textual comparison of an images indexed words, could help in determining how common certain objects are found together.

More Related