Applying New Voice Recognition Technology to Formative Assessment

1. Applying New Voice Recognition Technology to Formative Assessment Margaret Heritage UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST) Markus Iseli Henry Samueli School of Engineering, UCLA CRESST Conference, Los Angeles, CA September 8th, 2005,

2. Overview • Project Aims and Components • Features of the Program • Automatic Speech Recognition Technology • Interface Demo • Assessment Framework • Looking to the Future

3. Project Components • Develop speech recognition technology for children • Apply technology to create an on-demand, easy-to-use system of assessment in reading for students in grades K-2 • Develop system capacity to present auditory, text, graphical stimuli, and to score, analyze and adapt to responses • Develop query-based data mining to monitor students’ progress • Develop easy-to-understand displays of data analysis

4. Specific Aims Develop assessment system that : • Is helpful for teachers (i.e. has instructional utility and saves time) • Reduces variability (e.g., consistent instructions, consistent delays, consistent scoring) • Automatically scores and analyzes children’s performance on reading assessment tasks

5. Distinguishing Features • Strong interdisciplinary interactions among electrical engineering, computer science, education, psychology and linguistics • Collaboration with expert teachers • Focus on bilingual (Mexican-Spanish accented English) students • Validation of the system

6. Lens of Project • Information that teachers can use next day in their instruction? • Sensitivity to English Language Learners(ELLs) • Sensitivity to Language Knowledge

7. Instructional Utility Effective classroom is assessment-centered (NRC, 2000) • ongoing assessment of students’ learning that provides the day-to-day fuel for instruction Formative assessment • ‘ used to adapt the teaching work to meet the learning needs’ (Black, Harrison, Lee, Marshall, & Wiliam, 2003, p.2).

8. Reading error or pronunciation difference?

9. Academic Language Among the factors contributing to non-comprehension of text: • inadequate knowledge of the words used, • lack of familiarity with the syntactic structures (Lyon, 1998)

10. Automatic Speech Recognition (ASR) • Teach the computer to understand human speech OR • Teach the human being how to talk to be understood by a computer Three main challenges: • Speaker: gender, pronunciation, health, dialect, language • Environment: noise, other speech • System: devices, program

11. Child vs. Adult Speech • Children don’t talk just to be understood by the computer, they just talk! • Children cannot yet control their articulators as well as adults • Children have different anatomical features (shorter vocal tract), and these features change fast • Children have very high pitch frequency

12. Designing an ASR System • Questions • Who is going to use the ASR system? • In what environment will it be used? • Implementation • Collect “a lot” of appropriate data • Train the system • Test the system and make changes

13. Our Assessment System • The system will: • Assess each student in the same manner (no dependence on teacher) • Produce visual stimuli (letters, words, phrases) and record the child’s vocal response • Measure response times very accurately • Analyze the recorded audio and other data to generate reports which are useful to teachers (ASR) • Be easily accessible (internet) • Handle multiple users at the same time

14. System Architecture Interface Client Side Server Side

15. More detailed…

16. Flash Interface: Live Demo http://kittychan.icsl.ucla.edu/tball Flash interface design by Larry Casey

17. System Design Benefits • Accessible • Software: common web browser • Hardware: standard microphone • Flexible • Command structure is open-ended • Allows for any audio-visual testing set-up • Stable • Constant audio stream: everything is captured • Stimulus/response data is recorded in real time • Scalable • Content, display, navigation are independent

18. Assessment Framework • Recall the lens • Guiding questions: • Are the assessments embedded in an instructional framework? • What is the instructional value of the information? • How much assessment is too much?

19. Assessment Framework • Guiding questions privilege a hierarchical rather than a uniform approach to assessment. • All students take benchmark assessments as a check on progress • Some students take 'drill down' assessments related to specific skills on an as-needed based for diagnosis • Teachers have guidance on what to assess

20. Assessment Framework Skills Assessed: • Phonemic Awareness • Word recognition • Oral reading (accuracy and rate) • Comprehension • Syntax

21. Assessment Framework Links to: • English Language Development Standards • English Language Arts Standards

22. Basic Monitoring Spine of Reading Assessment Framework Narrative Oral Reading Narrative Oral Reading Narrative Oral Reading NarrativeReading Comp NarrativeReading Comp NarrativeReading Comp

23. Basic Reading Assessment Framework - Kindergarten Narrative Oral Reading NarrativeReading Comp After student demonstrates mastery of letter sound and naming tasks, begin assessing regular and irregular word reading. After student demonstrates mastery of listening comprehension and word reading tasks, begin assessing in connected text skills. Begin the framework with screening assessments in listening comprehension, letter sound, and naming tasks . 3. 2. Repeat assessing connected text skills throughout year as needed. If problem arises, recheck word reading development 1. 4. Narrative Listening Comp. Narrative Oral Reading Letter Sound Comp. The Dam (decodable word list) NarrativeReading Comp BPST K/1High Frequency Word List Rapid Naming

24. Reading Assessment Framework with Interventions - Kindergarten I.I. I.I. Vocab and Topic Knowledge I.I. I.I. Oral Lang. Comp. The Dam (decodable word list) Irregular Word list Narrative Oral Reading I.I. I.I. NarrativeReading Comp Written Lang. Comp. (Syntax) Vocab and Topic Knowledge Phonemic Awareness I.I. I.I. I.I. I.I. I.I. If student demonstrates skill level below mastery, provide instructional intervention and reassess. Narrative Listening Comp. Narrative Oral Reading Letter Sound Comp. The Dam (decodable word list) NarrativeReading Comp BPST K/1High Frequency Word List Rapid Naming Continue assessing connected text skills throughout year as needed.

25. 1st and 2nd Grades 2. After student demonstrates mastery of listening comprehension, letter sound, and word reading tasks, begin assessing in connected text skills. 3. Begin the framework with screening assessments in listening comprehension, letter sound, and word reading tasks . Repeat assessing connected text skills throughout year as needed. If problem with progress arises, recheck word reading development. 1. Oral Language Comp. Narrative Oral Reading Narrative Oral Reading San Diego High Frequency Word List NarrativeReading Comp NarrativeReading Comp BPST

26. Reading Assessment Framework with Interventions- First and Second Grades I.I. I.I. I.I. I.I. Vocab and Topic Knowledge Oral Lang. Comp. (Syntax) The Dam (decodable word list) Irregular Word list I.I. I.I. K/1High Frequency Word List Written Lang. Comp. (Syntax) The Dam (decodable word list) Vocab and Topic Knowledge Rapid Naming Letter Sound Comp. Phonemic Awareness I.I. I.I. I.I. I.I. I.I. Narrative Listening Comp. Narrative Oral Reading NarrativeReading Comp San Diego High Frequency Word List BPST Continue assessing connected text skills throughout year as needed.

27. ExampleKindergarten/1st grade BPST Blending Words with: Short map rip met rub mop Final-e fine rope rake cute kite Long soap leak pain feed ray r-control fur sort sir tar serve OVD coin moon round lawn foot 2 syllable silent ladder napkin locate cactus

28. Reporting

29. Looking to the Future • Validation of assessment system • Development of ASR to include discourse level performances • Leverage other CRESST technology( QSP) • Applications to other domains ( e.g., math and science) • Applications to other grade levels