1 / 22

Text ové Datab á zy

Text ové Datab á zy. Ján GENČI PDT. Obsah. Literat úra Terminol ógia Vymedzenie pojmu textové databázy Typy dotazov Fulltextové vyhľadávanie Lingvistick é korpusy. Literatúra. Pokorný J. : Datab ázové systémy 2, Nakladatelství ČVUT, 2007

xuxa
Download Presentation

Text ové Datab á zy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Textové Databázy Ján GENČI PDT

  2. Obsah • Literatúra • Terminológia • Vymedzenie pojmu textové databázy • Typy dotazov • Fulltextové vyhľadávanie • Lingvistické korpusy

  3. Literatúra • Pokorný J.: Databázové systémy 2, Nakladatelství ČVUT, 2007 • Pokorný J., Snášel V., Kopecký M.: Dokumentografické informačné systémy, Nakladatelství Karolinum, 2005. • Laura C. Rivero, Jorge H. Doorn, Viviana E. Ferraggine: Encyclopedia Of Database Technologies And Applications. Idea Group Publishing, 2005 (heslo Text Databases, p. 688) • Erickson J.: Database Technologies:Concepts,Methodologies,Tools, and Applications. IGI Global, 2009. ISBN 978-1-60566-058-5 (pp. 931-939)

  4. Literatúra (cont.-1)

  5. Literatúra (cont.-2) • Oracle Text. http://www.oracle.com/technology/products/text/index.html • Oracle Text. An Oracle Technical White Paper. June, 2007 (prečítať) http://www.oracle.com/technology/products/text/pdf/11goracletexttwp.pdf

  6. TXT DB - Terminológia • Textové databázy (informačné systémy) • Dokumentové databázy (Document databases) • Dokumentografické informačné systémy

  7. Text Databases – definition • A text is any sequence of symbols (or characters) drawn from an alphabet. • A large portion of the information available worldwide in electronic form is actually in text form (other popular forms are structured and multimedia information): • natural language text (e.g., books, journals, newspapers, jurisprudence databases, corporate information, the Web), • biological sequences (e.g., ADN and protein sequences), • continuous signals (e.g., audio and video sequence descriptions, time functions), • and so on. • A text database is a system that maintains a (usually large) text collection and provides fast and accurate access to it. These two goals are relatively orthogonal, and both are critical if one is to profit from the text collection.

  8. TXT DB - Type of queries • Syntactic search (expressed in the sequence of characters preseted in the text): • String matching (the simplest query, cely rad algoritmov – Knut-Morris-Pratt first O(n)) • Regular expression • Approximate searching (to recover from different kinds of errors that the text collection (or the user query) may contain - simple error model is edit distance) • Semantic search (great value) - user expresses an information need and the system retrieves portions of the text collection (i.e., documents) that are relevant to that need, even if the query words do not directly appear in the answer. System ranks the documents and offers the highest ranked documents to the user. There are no right or wrong answers, just better and worse ones.

  9. Fulltext search • In the traditional database management systems (DBMS), text manipulation is restricted to the usual string manipulation facilities (the exact matching of substrings) • The traditional string-level operations are very costly for large documents - traditional DBMS engine is inefficient for these operations, they are usually extended with a special full-text search (FTS) engine module.

  10. Fulltext search (cont.) • There is a significant demand on the market on the usage of free text and text mining operations, since information is often stored as free text (text analysis in medical systems, analysis of customer feedbacks, and bibliographic databases) • Simple character-level string matching would retrieve only a fraction of related documents

  11. Alternatives for implementingan FTS engine • Built-in FTS engine module (Oracle, Microsoft SQLServer, Postgres, and mySQL; Informix Text Datablade; ) • DBMS-independentFTS engine (SPSS LexiQuest, SAS Text Miner, dtSearch, and Statistica Text Miner)

  12. Ways of processing • Text mining • Full text search

  13. Text mining • Subfield of document management that aims at processing, searching, and analyzing text documents • The goal – to discover the non-trivial or hidden characteristics of individual documents or document collections • Interdisciplinary field of machine learning which exploits tools and resources from computational linguistics, natural language processing, information retrieval, and data mining

  14. General application schema of text mining

  15. Information Extraction • Includes i.e. following subtasks: • named entity recognition – recognition of specified types of entities in free text, • co-reference resolution – identification of text fragments referring to the same entity, • identification of roles and their relations – determination of roles defined in event templates

  16. Text Categorization • Aim - sorting documents into a given category system; e.g..: • document filtering –spamfiltering, or newsfeed; • patent document routing – determination ofexperts in the given fields; • assisted categorization – helping domainexperts in manual categorization with valuablesuggestion; • automatic metadata generation.

  17. Document Clustering • Groupping elements of a document collection based on their similarity. • Documents are usually clustered based on their content. • Document Clustering is applied for e.g.: • clustering the results of (internet) search for helping users in locating information, • improving the speed of vector space based information retrieval, • providing a navigation tool when browsing a document collection.

  18. Summarization • Automatic generation of short and comprehensible summaries of documents

  19. FULL-TEXT SEARCH (FTS)ENGINES

  20. Fulltext indices • A crucial sub-problemin the information retrieval area is the designand implementation of efficient data structures andalgorithms for indexing and searching informationobjects that are vaguely described. • The most commonly used indexing structures are: • inverted files, • signature files, • bitmaps.

  21. Informix • Excalibur Text DataBlade Module provides text search capabilities that include: • phrase matching, • exact and fuzzy searches, • compensation for misspelling, • synonym matching.

  22. Lingvistické korpusy • Kolekcie textov v konkrétnom jazyku určené primárne pre lingvistický výskum • Značkované texty • Príklady: • British National Corpus (100 mil. slov) • Slovenský národný korpus (530 mil. tokenov) • Český národný korpus (300 mil. slov) • Paralelné korpusy

More Related