1 / 19

Dialogue Act Tagging

Dialogue Act Tagging. Discourse and Dialogue CMSC 35900-1 November 4, 2004. Roadmap. Maptask overview Coding Transactions Games Moves Assessing agreement. Maptask. Conducted by HCRC – Edinburgh/Glasgow Task structure: 2 participants: Giver, follower 2 slightly different maps

alain
Download Presentation

Dialogue Act Tagging

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dialogue Act Tagging Discourse and Dialogue CMSC 35900-1 November 4, 2004

  2. Roadmap • Maptask overview • Coding • Transactions • Games • Moves • Assessing agreement

  3. Maptask • Conducted by HCRC – Edinburgh/Glasgow • Task structure: • 2 participants: Giver, follower • 2 slightly different maps • Giver guides follower to destination on own map • Forces interaction, ambiguities, disagreements, etc • Conditions: Familiar/not; Visible/not

  4. Dialogue Tagging • Goal: Represent dialogue structure as generically as possible • Three level scheme: • Transactions • Major subtasks in participants overall task • Conversational Games • Correspond to G&S discourse segments • Conversational Moves • Initiation and response steps

  5. Basic Dialogue Moves • Initiations and responses • Cover acts observed in dialogue – generalized Initiations: Instruct: tell to carry out some action; Explain: give unelicited information; Check: ask for confirmation; Align:check attention; Query-yn: Query-wh Responses:Acknowledge: signal understand & accept; Reply-y; Reply-n; Reply-wh; Clarify Ready:Inter-game moves

  6. Game Coding • Initiation: • Identified by first move • Purpose – carry through to completion • May embed other games – Mark level • Mark completion/abandonment

  7. Interrater Agreement • How good is tagging? A tagset? • Criterion: How accurate/consistent is it? • Stability: • Is the same rater self-consistent? • Reproducibility: • Do multiple annotators agree with each other? • Accuracy: • How well do coders agree with some “gold standard”?

  8. Agreement Measure • Kippendorf’s Kappa (K) • Applies to classification into discrete categories • Corrects for chance agreement • K<0 : agree less than expected by chance • Quality intervals: • >= 0.8: Very good; 0.6<K<0.8: Good, etc • Maptask: K=0.92 on segmentation, • K = 0.83 on move labels

  9. Dialogue Act Tagging • Other tagsets • DAMSL, SWBD-DAMSL, VERBMOBIL, etc • Many common move types • Vary in granularity • Number of moves, types • Assignment of multiple moves

  10. Dialogue Act Recognition • Goal: Identify dialogue act tag(s) from surface form • Challenge: Surface form can be ambiguous • “Can you X?” – yes/no question, or info-request • “Flying on the 11th, at what time?” – check, statement • Requires interpretation by hearer • Strategies: Plan inference, cue recognition

  11. Plan-inference-based • Classic AI (BDI) planning framework • Model Belief, Knowledge, Desire • Formal definition with predicate calculus • Axiomatization of plans and actions as well • STRIPS-style: Preconditions, Effects, Body • Rules for plan inference • Elegant, but.. • Labor-intensive rule, KB, heuristic development • Effectively AI-complete

  12. Cue-based Interpretation • Employs sets of features to identify • Words and collocations: Please -> request • Prosody: Rising pitch -> yes/no question • Conversational structure: prior act • Example: Check: • Syntax: tag question “,right?” • Syntax + prosody: Fragment with rise • N-gram: argmax d P(d)P(W|d) • So you, sounds like, etc • Details later ….

  13. Recognizing Maptask Acts • Assume: • Word-level transcription • Segmentation into utterances, • Ground truth DA tags • Goal: Train classifier for DA tagging • Exploit: • Lexical and prosodic cues • Sequential dependencies b/t Das • 14810 utts, 13 classes

  14. Features for Classification • Acoustic-Prosodic Features: • Pitch, Energy, Duration, Speaking rate • Raw and normalized, whole utterance, last 300ms • 50 real-valued features • Text Features: • Count of Unigram, bi-gram, tri-grams • Appear multiple times • 10000 features, sparse • Features z-score normalized

  15. Classification with SVMs • Support Vector Machines • Create n(n-1)/2 binary classifiers • Weight classes by inverse frequency • Learn weight vector and bias, classify by sign • Platt scaling to convert outputs to probabilities

  16. Incorporating Sequential Constraints • Some sequences of DA tags more likely: • E.g. P(affirmative after y-n-Q) = 0.5 • P(affirmative after other) = 0.05 • Learn P(yi|yi-1) from corpus • Tag sequence probabilities • Platt-scaled SVM outputs are P(y|x) • Viterbi decoding to find optimal sequence

  17. Results

  18. From Human to Computer • Conversational agents • Systems that (try to) participate in dialogues • Examples: Directory assistance, travel info, weather, restaurant and navigation info • Issues: • Limited understanding: ASR errors, interpretation • Computational costs: • broader coverage -> slower, less accurate

  19. Dialogue Manager Tradeoffs • Flexibility vs Simplicity/Predictability • System vs User vs Mixed Initiative • Order of dialogue interaction • Conversational “naturalness” vs Accuracy • Cost of model construction, generalization, learning, etc • Models: FST, Frame-based, HMM, BDI • Evaluation frameworks

More Related