1 / 19

CLEF 2008 Multilingual Question Answering Track

CLEF 2008 Multilingual Question Answering Track. UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner. QA 2008 Task and Exercises. QA Main task (6th edition) Pilot: QA WSD, English newswire collections with Word Sense Disambiguation

saburo
Download Presentation

CLEF 2008 Multilingual Question Answering Track

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CLEF 2008Multilingual Question Answering Track UNED Anselmo Peñas Valentín Sama Álvaro Rodrigo CELCT Danilo Giampiccolo Pamela Forner

  2. QA 2008 Task and Exercises • QA Main task (6th edition) • Pilot: QA WSD, English newswire collections with Word Sense Disambiguation • Answer Validation Exercise – AVE (3rd edition) • QA on Speech Transcripts – QAST (2nd edition)

  3. Main Task QA 2008Organizing Committee • CELCT (D. Giampiccolo, P. Forner): Italian • UNED (A. Peñas): Spanish • U. Groeningen (G. Bosma): Dutch • U. Limerick (R. Sutcliff): English • DFKI (B. Sacalenau): German • ELDA/ELRA (N. Moreau): French • Linguateca (P. Rocha): Portuguese • Bulgarian Academy of Sciences (P. Osenova): Bulgarian • IASI (C. Forascu): Romanian • U. Basque Country (I. Alegria): Basque • ILSP (P.Prokopidis): Greek

  4. Evolution of the Track

  5. 200 questions • FACTOID • (loc, mea, org, per, tim, cnt, obj , oth) • DEFINITION • (per, org, obj, oth) • CLOSED LIST • Who were the components of The Beatles? • Who were the last three presidents of Italy? • LINKED QUESTIONS • Who was called the “Iron-Chancellor”? • When was he born? • Who was his first wife? • Temporal restrictions by date, by period, by event • NIL questions (without known answer in the collection)

  6. 43 Activated Language Combinations(at least one registered participant)

  7. Activated Tasks 7

  8. Submitted runs 8

  9. Participant groups

  10. List of Participants (random order) Bulgaria

  11. Groups per year and target collection Natural selection? Task Change Above 20 groups

  12. Groups per target collection

  13. 2008 participation: Comparative evaluation? Lack from evaluation perspective: 4 languages without comparison between different groups Breakout session

  14. Results: Best and Average scores

  15. Best scores by language

  16. Best scores by participant

  17. Results depend on type of questions • Definitions • Almost solved for several systems 80%-95% • Factoids • 50%-65% for several systems • Temporal restrictions • Same level of difficulty as factoids for some systems • Closed lists • Still very difficult • Linked questions • Still very difficult • Now wikipedia provides more answers

  18. Conclusion • Same task as 2007 • Same level of participation (slightly better) • 11 target languages (9 with participation) • 43 activated subtasks • 21 participants • 51 runs • Same results (slightly better)

  19. Future direction • Less participants per language • Poor comparison • Change methodology: one task for all • Critics to QA over wikipedia • Easier to find questions with IR • No user model • Change collection • QA proposal for 2009 • SC and breakout

More Related