Exploring Assistive Technology Issues in Reading Assessment

1. Exploring Assistive Technology Issues in Reading Assessment ATIA 2010 Orlando: Jason Altman, NCEO

2. Statement of Problem • Little evidence that technology improves validity of tests for students w/ VI • One reason is that students’ experience and exposure to AT varies • TARA seeks to develop a test of student proficiency with AT • For use prior to using AT on large-scale reading assessments

3. Preliminary Studies: The State of AT Use with VI Students • Study #1: TVI survey (presented at ATIA ’09) • Study #2: TVI follow-up interview • Study #3: Student Observational Interview

4. Revisiting Teacher Survey • AT use is a large part of instruction for students with VI • TVIs take a “blended” approach to teaching reading (e.g., using a variety of modalities) • Standardization of assessments may be a challenge because students use different technology for different purposes

5. More on Teacher Survey • Reading and technology are intertwined activities for students with VI • The larger the caseload, the less likely students will have individualized AT experiences • Teachers who focus on foundational reading skills tend to provide fewer technology opportunities for students.

6. Overview: Results of Teacher Interview Study (#2) • 1st study showed “accommodations” or “modifications” are common for students w/ VI • Both high- and low-tech approaches to teaching reading are used by TVIs • Most likely for different purposes and at different times • Such flexible approaches to reading and situational use of technologies create challenges for standardization of assessments

7. Method • Interviewed 27 TVIs • At least two TVIs were interviewed in the following categories: • TVIs who work in Schools for the Blind • TVIs who are itinerant teachers • TVIs who teach students with low-vision • TVIs who teach students who are completely blind • TVIs who do not teach braille • TVIs who teach braille

8. Instrument and Procedures • All interviews were conducted by telephone • In most cases, there were two (and up to four) researchers present during telephone interviews • Research participants were asked to answer a series of structured interview questions • Interviews generally lasted 30 minutes, although some interviews lasted as long as one hour

9. Results • All 27 TVIs indicated that they would use a technology-based assessment if one were readily available • All TVIs who taught students with low-vision used at least some form of magnification equipment • TVIs who taught students who use braille readers also reported using a variety of technologies such as audio devices and braille note takers

10. Specific Products • Alpha Smart • Bookport • Braille and Speak • DAISY Readers • Extreme Reader • Freedom Box • IPod • JAWS • Kurzweil • MAGIC • OpenBook • OutSpoken • PacMate • Zoomtext

11. Technology and reading • Some TVIs used technology fluidly in instruction • Those who had wide-ranging knowledge of the tools available (and had access to such tools) match instructional tasks with technology tools. • 5 TVIs spent most of their time teaching students how to use CCTVs and were hesitant to engage in teaching students using computer-based tools (especially those that added voicing to files) • In all cases, however, TVIs used some form of technology to assist teaching

12. Formal AT assessment protocols • About 1 of 4 teachers were using formal assessment protocols/tools to assess students’ assistive technology use • Among those most frequently used are the SETT Protocol and the Texas School for the Blind Assistive Technology Assessment Summary for Students with Visual Impairments • One teacher who used the Texas assessment also mentioned using the Georgia Project for Assistive Technology’s (n.d.) Assistive Technology Evaluation and the Wisconsin Assistive Technology Initiatives (2004) WATI Assistive Technology Checklist

13. Assessment: Proficient Reader Who Uses AT • Teachers recognized that “advanced proficiency” in technology-assisted reading meant the ability to move fluidly between technologies in order to optimize the potential of various technologies • A proficient user of technology-assisted reading would be a reader who read and understood grade level material • Most TVIs were concerned that their students be assessed on grade-level, but with a fair opportunity to demonstrate grade-level skills • TVIs were concerned about the validity of tests

14. Discussion • There is a desire for a formal assessment or evaluation of student skills • Should discriminate between various levels of proficiency • There are a small number of assessments/frameworks in use

15. Activity TVIs in this room: Had you been participants would have findings changed

16. Student Observational Interview: Study #3

17. Method: Sample • Students with visual impairments in grades 6-10 • In total, we interviewed 18 students for this study from 5 states across the US • We sampled students from both general education school systems (n=9) and state schools (n=5) for the blind • Four additional students were educated at state schools through general education classes in a nearby public school

18. Method: Instrument and Procedures • Students participated in “observational interviews” facilitated by three researchers on the project • Observational interviews were a hybrid between verbal interviews, where respondents describe phenomenon (Bogdan & Biklen, 1998) and cognitive interviews, where interviewees participate in an activity and describe their thoughts and actions (Ericsson & Simon, 1993)

19. Procedures (cont.) • During these interviews, students were asked to complete reading tasks using AT • Afterward, students were asked how they use AT in the reading process, including how to download files, retrieve information from printed material, and to explain preferences • Each interview lasted between 30 and 60 minutes

20. Results of Student Study • Demands of a rigorous high school curriculum at times left students without enough time to explore and become proficient in new and possibly more efficient technology strategies • For students with visual impairments reading large print or Braille, the time it takes to complete a task is often far greater than that for their peers with full functioning vision

21. Results • 11 of the 18 students could read large print – 3 of them regular print as well, and many were audio book users. • The remaining 7 students all read braille – and 3 of the large print readers also read braille. • 13 students used audio books to access print no matter what their primary method of print reading was

22. Results • 8 students used technology for reading Braille – 5 used Braille Note • For magnification, students used a variety of technologies, from simple handheld magnifiers to computer-based products • Zoomtext (most often also with speech), CCTV • Technology use for 7 depended on the reading situation

23. Results • Technologies are sometimes unreliable (e.g., not all formats work for internet-based text), take time to learn, and do not always accompany the student beyond the school walls (e.g., many students cannot bring AT devices home to assist with homework)

24. Student characteristics and independence • Students varied from completely relying on help from sighted teachers, para-professionals, and peers to being assertively independent • Most students described their dependence on others as situational • One student mentioned that he is not afraid to ask his friends or family to read things for him • Another student, said that help is not needed

25. Technology Choices • Depended on availability of technology in student location • For example, students at a state school typically would have more access to technology than a student in a rural public school that did not serve a large population of students with visual impairments • Access at home may differ from access at school • One student, who was on the academic honor roll at school, often stayed after school and came in before school every morning to finish homework

26. Choice of specific technology • The student’s TVI or para-professional generally was very involved in the decision • Some students attended technology conferences • It is important to note that another person (teacher, parent, or para-professional) needed to learn the technology along with the student • TVIs had to be careful not to over-extend themselves with multiple new technologies

27. Role of expected change in situation • Decisions were impacted by the nature of the visual impairments • Changes in student vision and forecasts for future change often called for a change in products • For example, one student used a CCTV up until 6th grade but stopped using it because there was a decrease in his field of vision and it was no longer practical

28. Difficulty with AT • The amount of time that must be dedicated to training and practice in using the equipment sometimes outweighed the potential benefit of the technology. • Another complication was consistent access to technology • Students were sometimes forced to share technology with several other students, including CCTVs

29. Other Hangups • There was also difficulty in transporting, setting up, and stowing equipment • Upon learning a particular technology platform, students were hesitant to move to other platforms • Equipment malfunction, or general hang-ups and glitches also were sometimes troublesome for students, teachers, and paraprofessionals alike. • One student stated that the Web sites that contained many graphics can be most problematic

30. Testing and Large-Scale Assessment • Multiple students reported using enlarged text to access the items and then having the paraprofessional fill in the bubble answer sheet • At least one student was able to take a math test using a computer; however, he was required to take the reading test using a paper and pencil version • Other students used standard question and answer forms, accessing them via CCTVs

31. Discussion • What do these findings mean for TVIs? • Does the situation play out similar in your classroom? • Would you agree that your student’s participation in large-scale assessment could be more meaningful?

32. Conclusions • As statewide assessment policy is still in flux (Thurlow et al., 2006), it is important for research to continue in the area of technology-assisted reading to provide stakeholders with a clearer picture of the avenues that students with visual impairments take to access reading anddemonstrate their reading skills

33. Also at this Conference • Also see today from 2:45-3:45 Bonaire 8 EDU-15 Development of an Accountability Test to Measure a Student’s Ability to Access Text with AT Presenting: Laitusis, Hall

34. Next Steps • Field Testing the new Technology Based Reading Assessment in school classrooms • We need your help! • Administrators of the field test • Students who can participate

35. Thank you! For more information… National Center on Educational Outcomes University of Minnesota612-626-1530 http://www.nceo.info Jason Altman altma014@umn.edu