CBIR Based Expert System an Overview

1. CBIR Based Expert Systeman Overview Subhash Chand Dec. 21, 2010 Indian Agricultural Statistics Research Institute

2. Today’s Agenda • Introduction. • CBIR Techniques • Implementations. • Conclusion.

3. Obsective • Able to design Expert System: • beyond the language bar • Image retrieval (Color, Taxture, Histogram etc.)

4. Introduction Farmers always need satisfactory and easy advice from expert system, especially from agricultural expert system. To get the advice from an expert system, it should have enough knowledge about the domain. Gathering enough knowledge and representing it in a machine understandable format is time consuming and difficult job. Also, representing each and every kind of knowledge is still a research issue.

5. Is Multimedia Solution? • Multimedia integrate image, movie, sound and text in any combination. The use of images in human communication is hardly new – our cave-dwelling ancestors painted pictures on the walls of their caves, and the use of maps and building plans to convey information almost certainly dates back to pre-Roman times. Images now play a crucial role in fields as diverse as medicine, journalism, advertising, design, education and entertainment.

6. Multimedia expert systems provide "a comprehensive set of resources to enhance the presentation and communication of expertise to support users' needs within a particular area of expertise."

7. Since, a single picture is worth a thousand words, it will be a good idea to acquire knowledge also in images rather than only text. Image is an easy way of communication without any boundary of language. Hence there is a need for building an expert system with content based image retrieval which could acquire and deliver the knowledge by searching the image having the similar features that is searched by the user.

8. Can CBIR Expert System help?

9. What is CBIR ? • Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR)is to describe experiments into automatic retrieval of images from a database, based on the colors and shapes. • The term has been used to describe the process of retrieving desired images from a large collection on the basis of syntactical image features. The techniques, tools and algorithms that are used originate from fields such as statistics, pattern recognition, signal processing, and computer vision.

10. What does mean Content in CBIR ? • "Content-based" means that the search will analyze the actual contents of the image rather than the  metadata such as keywords, tags, and/or descriptions associated with the image. • The term 'content' in this context might refer to colors, shapes, textures, or any other information that can be derived from the image itself. CBIR is desirable because most web based image search engines rely purely on metadata and this produces a lot of garbage in the results. Also having humans manually enter keywords for images in a large database can be inefficient, expensive and may not capture every keyword that describes the image. Thus a system that can filter images based on their content would provide better indexing and return more accurate results.

11. Applications • Art Collections e.g. Fine Arts Museum of San Francisco • Medical Image Databases CT, MRI, Ultrasound, The Visible Human • Scientific Databases e.g. Earth Sciences • Trade Mark • The World Wide Web • Expert Systems

12. What is aquery in CBIR? • an image you already have • a rough sketch you draw • a symbolic description of what you want e.g. an image of a man and a woman on a beach

13. Demo Caliph & Emir

14. Definations • MPEG-7 is a multimedia content description standard developed by MPEG (Moving Picture Experts Group). It was standardized in ISO/IEC 15938 (Multimedia content description interface). • A Descriptor represents a Feature of audio-visual material whereas a Feature here is understood as "a distinctive characteristic of the data [that] signifies something to somebody" • MPEG-7 Visual Descriptors :Part three of the ISO/IEC 15938-3 specifies standardized methods for the description of visual content - that are still images, videos and 3D models. The MPEG-7 Visual Descriptors are grouped in five categories by the features of color, texture, shape, motion and localization as well as a domain specific face recognition tool.

15. Low Level Features of Image • Color Feature Descriptors • The color description tools include four descriptors that represent different aspects of color: representative colors (DominantColor), color distribution (ScaleableColor) and spatial distribution of colors (ColorLayout and ColorStructure) and may be extracted from images of arbitrary shape.

16. DominantColor: fits the targets for content based retrieval of either the whole image or arbitrary defined shapes by defining a set of characterizing dominant colors in that area. It contains fields (among others) representing the percentage of pixels having that associated color in the specified region, as well as a spatial coherency per dominant color attribute. This descriptor is most suitable for representing local features where a small amount of colors is enough to characterize the provided information in the region of interest but it also is applicable for whole images for example, flag or color trademark images.

17. ScalableColor: This descriptor depicts a color histogram in the HSV color space, encoded by a Haar transformation (which allows the sorting of data by frequency) and is used to find and compare images by their color characteristics.

18. ColorLayout: Is used for high-speed image retrieval and browsing using image-to-image and sequence-to-sequence matching - and even sketch-to-image matching, by specifying a spatial distribution of colors. This Descriptor makes use of the DCT12 the discrete cosine transformation to compute its YDC, CbDC, CrDC, YAC, CbAC and CrAC coefficients. This compact representation of color layout information allows visual signal matching functionality combined with high retrieval efficiency at very small computational costs and therefore even is utilizable on mobile terminal applications with strictly limited hardware and software restrictions.

19. ColorStructure: This descriptor is similar to the one of a color histogram in the meaning of describing color content - but further also pays respect to the structure of its content. Unlike the color histogram it uses a structuring element, which is composed of 8x8 connected pixels and instead of characterizing the relative frequency of individual image samples it characterizes the frequency of these structuring elements. In that way it is possible to distinguish between two images having an identical amount of the same color, though the structure of the groups of pixels having that amount is different. The main areas of usage are image-to-image comparison in domains of still-image retrieval – even in arbitrary shaped and possibly non-connected regions.

20. Texture Feature Descriptors • Textures in MPEG-7 are understood in the meaning of patterns which are used to describe surface characteristics of objects. Similarity of patterns here is measured in terms of intensity of certain colors in an image or by the degree of uniformity of a structure compared to a certain environmental point. The three standardized Visual Texture Descriptors TextureBrowsing, HomogeneousTexture and EdgeHistogram are very important tools for similarity retrieval and browsing.

21. HomogeneousTexture: Homogeneous texture has emerged as an important visual primitive for searching and browsing through large collections of similar looking patterns. An image can be considered as a mosaic of homogeneous textures so that these texture features associated with the regions can be used to index the image data.

22. TextureBrowsing: This descriptor has the function to represent a perceptual characterization of a texture, in terms of regularity (irregular, slightly regular, regular, and highly regular), coarseness (fine, medium, coarse, very coarse) and directionality (the dominant direction(s) characterizing the texture alignment.

23. EdgeHistogram: Edges play an important role for image perception. The EdgeHistogram tool distinguishes five types of edges in local image regions, four directional (vertical, horizontal, 45 degree, 135 degree) and a non- directional one. This information is provided for every sub-image, which is de.ned by dividing the image space into 16 non-overlapping parts. The Descriptor is able to retrieve images with similar semantics.

24. Using Relevance Feedback • The CBIR system should automatically adjust the weight that were given by the user for the relevance of previously retrieved documents • Most systems use a statistical method for adjusting the weights.

25. Content Based Image Retrieval (CBIR)Techniques Color Correlogram Color Edge Display Descriptor (CEDD) Fuzzy Color and Texture Color (FCTH)

26. Color Correlogram: The color correlogram describes the global distribution of local spatial correlations of colors and Integrates both color information and space information. Color correlogram of an image is a table indexed by color pairs, where the k-th entry for (i,j) specifies the probability of finding a pixel of color j at a distance k from a pixel of colori in the image. Such an image feature turns out to be robust in tolerating large changes in appearance of the same scene caused by changes in viewing positions, changes in the background scene, partial occlusions, camera zoom that causes radical changes in shape, etc. This technique is suitable for the system as image taken by farmer may have variation in image’s properties.

27. Color Edge Display Descriptor: extraction technique is based on algorithm. The unit associated with the extraction of color information is called Color Unit. Similarly, the Texture Unit is the unit associated with the extraction of texture information. The CEDD histogram is constituted by 6 regions, determined by the Texture Unit. Each region is constituted by 24 individual regions, originating from the Color unit.

28. Fuzzy Color and Texture Color (FCTH): extraction technique is based on algorithm, in which the histogram is constituted by 8 regions, as these are determined by the fuzzy system that takes decision with regards to the texture of the image. Each region is constituted by 24 individual regions, as these results from the second fuzzy system. Overall, the output that results includes 8 × 24 = 192 bins. Based on the content of the bins the respecting final histogram is produced.

29. Basic Components of CBIR • Feature Extractor • Create the metadata • Query Engine • Calculate Similar

30. Implementation • Can Use Java API called LIRE http://www.semanticmetadata.net/lire

31. User Interface User Domain Expert Farmer Insert Image Image Processing Search for Image Delete Image Feature Extraction Indexing Mechanism Database Management Database Handler Database Typical System Design Diagram of Multimedia Enabled Expert System using CBIR

32. Image Uploading Searching Retrieval on textual descriptors and /or similarity measures of features Correlogram Image Feature Extraction CEDD FCTH XML Document Storage Yes Visualisation Indexing No No image found; Submiteed to domain experts to characterize the image Typical Flow Chart of the Multimedia CBIR based Expert System

33. Demo • Project • Lire • http://www.anaktisi.net/ • Image Search Engine • http://www.gazopa.com • http://mipai.esuli.it/ • http://www.tineye.com/ compare

34. Case Study Sample image

35. Results • Results_Corologrm • Result_CEDD • ResultFCTH

36. Conclusion • Now, We have many features (too many) • Not all features are always important. • Can choose one or combination of many techniques • CBIR based application can be integrated with existing text based expert system. • It’s wide acceptability beyond the regional language bar. • Where images are involved in your research.

37. Refferences • http://en.wikipedia.org/wiki/List_of_CBIR_engines • http://orpheus.ee.duth.gr/anaktisi/ • http://http://cvit.iiit.ac.in/projects/cbir/ various approach of CBIR • Huang, J.; Kumar, S. R.; Mitra, M.; Zhu, W. & Zabih, R. "Image Indexing Using ColorCorrelograms", IEEE Computer Society(2007).

38. Thanks

39. Experience the following and expand your ideas • Project • http://www.semanticmetadata.net/ • http://www.anaktisi.net/ • Image Search Engine • http://www.gazopa.com • http://mipai.esuli.it/ • http://www.tineye.com/ compare