Building Robust Grammars to Handle Unexpected Responses

1. Building Robust Grammars to Handle Unexpected Responses Sunil IssarDirectorConvergys Corporation

2. Convergys at a Glance Enabling organizations to enhance the value of their relationships with customers and employees Solutions and Service Offerings • Customer Care, Human Resources, Billing Services • Solutions, Software, Outsourcing, Consulting Worldwide capabilities • Industry Strengths Include: Financial, Communication, Consumer Package Goods, Technology, Health Care, Retail, Pharmaceutical • 73 contact, service and data centers worldwide • Over 575 clients in 70+ countries Leading public company • Listed on NYSE, S&P 500, Fortune 1000 • A Fortune Most Admired Company for seven consecutive years • $2.8 billion in revenues

3. Agenda • Designing Speech Grammars for High Recognition Accuracy • Missing Science • Achieving High Recognition Accuracy with Robust Speech Grammars • Summary

4. Designing Grammars • Grammars define rules describing callers responses • Airport Grammar • <city> • Boston • <city> <state> • Boston Massachusetts • <airport name> • Logan International Airport • <city> <airport> • Boston Logan • <airport code> • B O S • Define rules for expected responses • Use menu options listed in VUI • Use synonyms, prefixes and suffixes • Use external data sources to ensure completeness • Postal database for city/state

5. Measuring Grammar Performance • Use Speech Recognition Accuracy to measure grammar performance • Correct if recognition result matches callers response • Table below shows recognition results for a digits grammar • Accepts 2-16 digits • Expects single digits • In grammar accuracy measures performance on responses listed in grammar • Perceived speech recognition performance is based on total accuracy

6. Identifying Speech Recognition Issues • High out of grammar rate leads to low overall accuracy • Noise (Peru 800) • Hello (help 900) • Conversational: Give me a agent please (city 600) • Background talk (international from city 600) • Correction: March 3rd 19 or 2007 (March 19th 2007 750) • Restart: December December 14th 2006 (Sat Dec 14th 2006, 700) • Large grammars with uniform probabilities lead to recognition errors • Causes substitutions • Noise is easily misrecognized as “help” • Causes insertions • Recognizer incorrectly inserts short words, digits or letters • Easy to insert “O” for zero or “a” for eight

7. Missing Science • Caller does not know the grammar rules • Follows conversational behaviors and uses unexpected events • Restarts, Corrections, Side conversations • Around 20% of the responses include unexpected events • It is difficult even for people to understand unfamiliar words

8. Building Robust Grammars • Focus on improving recognition accuracy metrics • Conduct recognition experiments • Classify recognition errors • Out of Grammar • Substitution • Insertion • Deletion • Confidence scores

9. Use Appropriate Grammars Based on Caller Responses • Review caller responses • Finite State grammars are appropriate for most speech applications • Short well-formed responses • Use Finite State Grammars (FSG) • Use optional prefixes • I want to find an ATM in Reston, Virginia. • I want <um> I am looking for an ATM in Reston, Virginia. • Use optional suffixes • Reston please • Spontaneous Speech, Large vocabulary • Use Statistical Language Models (SLM) • SLM models n-grams • Example bigram (2-gram): Pr (Virginia | Reston) • SLM are automatically trained from callers responses • Need to assign semantic interpretation before training

10. Use Weights to Improve Accuracy for Frequently Used Phrases • Table below shows accuracy improvements based on limiting digit grammar to frequent digit lengths • Need to look at the data source or actual responses to identify frequent phrases • Yes, No are the most frequent responses for a Yes/No question • Adding synonyms to reduce out of grammar rate causes substitution errors • Talk by Vaibhav Goel from IBM at SpeechTek West 2007 describes accuracy improvements using weights

11. Reduce Out of Grammar (OOG) Rate • Lower Out of Grammar rate will improve overall accuracy • May not reduce number of turns • Need to reprompt even if we recognize side conversations • Will improve caller perception of speech application • Classify Out of Grammar Responses • Noise • Background talk • Data • Format • Restart • Correction • Multiple slots • Model most common OOG responses • Garbage model for noise and background talk • Include rules for missing data elements • Ignore less obvious formats • Date: the second two thousand six

12. Use Completion Techniques to Reduce Out of Grammar Rate • Completion techniques better model callers responses • Responses may not be presented as options • Responses follow conversational behaviors • Menus • Frequent responses include “none of these” or synonyms • “None of these” is not an option typically presented to the caller • Grammar should include “none of these”

13. Summary • Grammars need to model conversational behaviors to improve callers perception of speech applications • In-grammar accuracy is high • Clients and callers perceptions don’t match in-grammar accuracy • Reviewed several techniques for building robust grammars • Use appropriate grammar type • Use weights • Reduce out of grammar rate • Use garbage models