Mark Hawley et al Barnsley District General Hospital and University of Sheffield

1. STARDUST – Speech Training And Recognition for Dysarthric Users of Assistive Technology Mark Hawley et al Barnsley District General Hospital and University of Sheffield

2. STARDUST • To develop speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers

3. Research Team • Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital(Mark Hawley, Simon Brownsell, Stuart Cunningham) • Institute of General Practice and Primary Care, University of Sheffield(Pam Enderby, Mark Parker, Rebecca Palmer) • Department of Computer Science, University of Sheffield(Phil Green, Nassos Hatzis, James Carmichael) • Project funded by Dept of Health New and Emerging Applications of Technology (NEAT) programme

4. Dysarthria • A neurological motor speech impairment • characterised by slow, weak, imprecise and/or uncoordinated movements of the speech musculature. • Speech is often difficult to understand (unintelligible) and variable (inconsistent) • Frequently associated with other physical disabilities • Severe = <40% intelligible

5. Speech-input writing programmes • Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999) • Mild dysarthric speech - 10-15% lower recognition rates(Ferrier, 1992) • Recognition declines as speech deteriorates - by 30-50% for single words(Thomas-Stonell, 1998, Hawley 2002)

6. Performance of a commercial speaker-dependent recogniser(in ‘ideal’ conditions)

7. Difficult speech recognition problem • Dysarthric speech • different to ‘normal’ models • more variable than ‘normal’ speech, both between and within speakers • difficult to collect large corpus of speech

8. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers(ie improve consistency) and improve recognition

9. Intelligibility and Consistency • ‘Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent). • ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent) • but may show consistency of key elements which will make it more intelligible to the familiar listener. • STARDUST is concerned with consistency

10. Training program • Visual feedback to improve consistency at word level • Quantitative • Real time • To be used by the client alone or with carer or therapist • Training tool records speech - used to build recogniser

11. Training program set-up • Record 10 examples of each word to be trained • Program builds models of words based on examples • For each word, program selects example that best matches its model (the best-fit recording) • Program feeds back a measure of the match between last utterance and model

12. Switch PC Recogniser Microphone Score calculation

13. Outcome of speech training(preliminary data) In a group of 5 users, 3 showed an upward trend, 2 showed no upward trend

14. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers

15. Recognition technology • Small vocabulary • Speaker dependent • uses hidden Markov models • based on HTK (University of Cambridge)

16. STARDUST recogniser performance(N=number of words used for training)

17. STARDUST • To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria • To develop a reliable small vocabulary speech recogniser for dysarthric speakers • To develop a computer training program to help to stabilise the speech of dysarthric speakers

18. Switch PC Recogniser Speech synthesiser or recording Microphone Look-up table

19. Vocabulary mapping • One to one (word to phrase) mapping • ‘Want’ = I need something, could you help me? • Pseudo-grammatical combinations • ‘Want ... drink’ = Could I have a drink, please? • Coding • ‘3…6…4’ = I went to Spain for my holidays • nmpossiblecombinations, where n is no of words in vocab, m is length of vocabulary string

20. Switch PC Recogniser Microphone Look-up table Infra-red

21. Work in progress • Test systems in home-based field trials • acceptability • usability (eg speed of access) • accuracy • reliability • practicality

22. Work in progress • Remove switch activation of recogniser • Increase vocabularies to test limits of recogniser • Develop tools for clinicians to build and test individual configurations

23. STARDUST - conclusions • Recogniser that recognises severely dysarthric speech • Computer-based training program to improve recognition and consistency • word level (and sub-word level in future) • collects lots of speech data for recogniser • Developed demonstrators of environmental control and voice-output device • next step to test in real usage