1 / 20

Semi-Automatic Image Annotation

Semi-Automatic Image Annotation. Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski and Brent Field Microsoft Research. Outline. Introduction: What, Why, and How Our Approach: Semi-Automatic Processes and Algorithms Automated Performance Evaluation

Jimmy
Download Presentation

Semi-Automatic Image Annotation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Semi-Automatic Image Annotation Liu Wenyin, Susan Dumais, Yanfeng Sun, HongJiang Zhang, Mary Czerwinski and Brent Field Microsoft Research

  2. Outline • Introduction: What, Why, and How • Our Approach: • Semi-Automatic • Processes and Algorithms • Automated Performance Evaluation • Usability Studies • Concluding Remarks

  3. What it is and Why • Image Annotation is a process of labeling images with keywords to describe semantic content • For image indexing and retrieval in image databases • Annotated images can be found more easily using keyword-based search

  4. Image Annotation Approaches • Totally Manual Labeling (Gong et al., 1994) • Enter keywords when image is loaded/registered/browsed • Accurate but labor-intensive, tedious, and subjective • Direct Manipulation Annotation (Shneiderman and Kang 2000) • Drag and drop keywords (from a predefined list ) onto image • Still manual, also limited to predefined keywords (can’t be many) • Automatic Approaches: Efficient but less reliable and not always applicable compared to human annotation---how to grab this when no text context? • By Image Understanding/Recognition (Ono et al. 1996) • By Associating with environmental text (Shen et al. 2000; Srihari et al. 2000; Lieberman 2000)

  5. Our Proposed Approach • Semi-Automatic Approach • User provides initial query and relevance feed back. • Feedback used to “semi-automatically” annotate images • Trade-off between manual and automatic • Achieve both accuracy and efficiency • Increase productivity • Employ Content-Based Image Retrieval (CBIR), text matching, and Relevance Feedback (RF)

  6. CBIR and RF Process and Framework

  7. Algorithms for Matching • Visual Similarity Measurement • Features: color histogram/moments/coherence, Tamura coarseness, pyramid wavelet texture, etc • Distance model: Euclidean distance • Semantic (Keywords) Similarity Measurement • Features: keyword vectors, TF*IDF • Metrics: dot product and cosine normalization • Overall similarity: weighted average of the above two

  8. Algorithms to Refine Search • Image Relevance Feedback Algorithms • There are many algorithms can be used • Cox et al. (1996) • Rui and Huang (2000) • Vasconcelos and Lippman (1999) • Lu et al. 2000 is employed in MiAlbum for text and images • Modified Rocchio’s Formula • Uses both semantics (keywords) and image-based features during relevance feedback

  9. Semi-Automatic Annotation During Relevance Feedback • In each keyword-query search cycle • When positive and negative examples provided, • Increase the weight of the keyword for all positive examples • Decrease the weight of the keyword for all negative examples • Relevance feedback algorithm refines and puts more relevant images in top ranks for further selection as positive examples • Repeat the feedback process

  10. Possible Future Automatic Annotation • When a new image is added… • Find top N similar images using image metrics • Most frequent keywords among annotations of these top N similar images are potential annotations, and could be automatically added with low weight or presented to user as potential annotations • TBD--Need to be confirmed in further RF process

  11. Automated Performance Evaluation • Test Ground Truth Database • 12,200 images in 122 categories from Corel DB • Category name is ground truth annotation • Automatic Experimental Process • Use category name as query feature for image retrieval • Among first 100 retrieved images, those belonging to this category are used as positive feedback examples others as negative • Performance Metrics • Retrieval accuracy and annotation coverage

  12. Image retrieval accuracy and annotation coverage

  13. Usability Studies • Objectives • 2 studies examined overall usability of MiAlbum • The usability of the semi-automatic annotation strategy • Tasks • Import pictures, annotate pictures, find pictures, and use relevance feedback • Questionnaires including but not limited to • Overall ease of entering annotations for images • Impact of annotation on ease of searching for images • Satisfaction of search refinement & relevance feedback

  14. Questionnaire Results • Overall ease of entering annotations: 5.6/7.0 • Ease to search annotated photos: 6.3/7.0 • Intuitiveness of refining search: 4.1/7.0 • Other Comments • Positive on “semi-automatic”: (1) When using the up and down hands the software automatically annotated the photos chosen. (2) The ability to rate pictures on like/dislike and have the software go from there. • Negative: difficulties in understanding the feedback process and how the matching algorithm operated.

  15. Concluding Remarks • A Semi-automatic Annotation Strategy Employing • Available image retrieval algorithms and • Relevance feedback • Automatic Performance Evaluation • Efficient compared to manual annotation? • More accurate than automatic annotation • Usability Studies • Preliminary usability results are promising • Need to improve the discoverability of the feedback process and the underlying matching algorithm

More Related