Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems

1. Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems Kate Forbes-Riley, Diane Litman, Scott Silliman, Amruta PurandareUniversity of PittsburghPittsburgh, PA, USA

2. Outline • Introduction • WOZ-TUT System • Experimental Design • Uncertainty Corpus Description • Uses of the Uncertainty Corpus

3. Automatic Detection: promising across affective states and applications, e.g. (Craig et al., 2006; Litman & Forbes-Riley, 2006; Lee & Narayanan, 2005; Vidrascu & Devillers, 2005; Batliner et al., 2003) Larger goal is automatic adaptation, but results are sparser More public affect-annotated corpora of human-computer dialogues could help, e.g. HUMAINE project SYMPAFLY, AIBO (Batliner et al., 2004) (German) Communicator (Walker et al., 2001; Ang et al., 2002) (English) Uncertainty Corpus: new complex domain: spoken dialogue tutoring new affect annotation: student uncertainty Overview: Towards Affect-Adaptive Spoken Dialogue Systems

4. WOZ-TUT:Adaptive Wizard of OZ Tutoring System modified version of ITSPOKE (Litman & Silliman, 2004) system responses based on combined uncertainty and correctness human recognizes speech, annotates uncertainty and correctness Why uncertainty? Most frequent in ITSPOKE corpora (Forbes-Riley & Litman, 2007) Most systems respond only to correctness, but literature suggests uncertain and incorrect answers signal learning impasses What uncertainty adaptation? Treating uncertain+correct answers as incorrect should provide additional knowledge to bridge impasse Uncertainty Corpus Collection: WOZ-TUT System

5. WOZ-TUT Screenshot

6. 3 Conditions: used parameterized WOZ-TUT dialogue manager Experimental: treat all uncertain+correct turns as incorrect First Control: ignore uncertainty (logged) Second Control: ignore uncertainty (logged), but treated a percentage of random correct answers as incorrect Experimental Design

7. TUTOR: What will the velocity of the object be a second after that (where the initial velocity is 9.8m/s and the acceleration is 9.8m/s2)? STUDENT: Nineteen point six meters per second?? [uncertain+correct] TUTOR in First Control Condition moves on: Good. So at every point in time during the fall of the man and his keys, how do their velocities compare with each other? TUTOR in Experimental Condition remediates: Okay. As we have seen, if a falling object has an acceleration of 9.8m/s2, its velocity changes by 9.8m/s every second. So if a second after it began falling its velocity is 9.8m/s, a second later its velocity will be 9.8m/s + 9.8m/s = 19.6m/s. So what will its velocity be a second after that? Corpus Excerpts

8. 60 subjects randomly assigned to 3 conditions (gender-balanced) Native English speakers with no college physics Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZ-TUT, 4) took posttest, 5) worked isomorphic test problem with non-adaptive WOZ-TUT Experimental Procedure

9. 120 dialogues from 60 students (.ogg format) 20 total hours of dialogue Student turns manually transcribed, including disfluency and non-syntactic question annotation Tutor turns and Wizard annotations in log files Corpus Description

10. One-way ANOVAs showed no significant differences: number of correct, uncertain, or uncertain+correct turns number adapted-to turns (EXP vs CTRL2) Student Answer Attributes

11. Uses of the Uncertainty Corpus I • Compare student performance across conditions to isolate impact of uncertainty adaptation • No significant differences in learning • We are comparing dialogue-based metrics in the isomorphic test problem (Forbes-Riley, Litman and Rotaru, 2008) - Feedback confound identified and rectified in larger follow-on study

12. Uses of the Uncertainty Corpus II • Resource for analyzing linguistic features of naturally-occurring user affect in human-computer dialogue • Models built from elicited emotions generally transfer poorly to naturally-occurring dialogue (Cowie and Cornelius, 2003; Batliner et al., 2003) • Uncertainty Corpus provides a new resource for modeling naturally-occurring dialogue • Large number of features in speech, transcript, log files

13. Summary and Current Directions • The Uncertainty Corpus is a collection of tutorial dialogues between students and an adaptive Wizard-of-Oz spoken dialogue system • Corpus (speech, transcripts, uncertainty and correctness annotations) publicly available by request through the Pittsburgh Science of Learning Center: https://learnlab.web.cmu.edu/datashop/index.jsp • Follow-on experiments and corpora • Larger WOZ study just completed, with learning results! • Fully automated study to begin Fall 2008

14. Questions? Further Information? web search: ITSPOKE or PSLC Thank You!