Multifaceted Approaches for the Evaluation of an Educational Exhibit

1. Multifaceted Approaches for the Evaluation of an Educational Exhibit Arlen Rosenthal, M.A. Adrienne Semidey, M.A. Wanda Parham, M.P.A. and Ewa Carlton, M.A. ORC Macro Jean Horrigan National Eye Institute October 23, 2001

2. THE EYE SITE-A Traveling Exhibit on Low Vision for Shopping Centers • Developed by the National Eye Institute • Target audience: • young people to senior citizens • Hispanic/Latino • African American • Consists of 5 colorful kiosks and includes • interactive multimedia touchscreen • information on low vision services and resources • display of aids and devices • Piloted at a shopping mall in Birmingham, AL in January-February 2000.

3. Objectives of The EYE SITE • To increase knowledge about low vision • To increase referrals of people with low vision to eye care professionals and from local eye care professionals to specialists in low vision. • To increase the use of adaptive devices among people with low vision. • To increase the use of vision rehabilitation services and devices by people with low vision. • To increase knowledge about NIH, NEI, and its medical research.

4. Study Methodology • THE EYE SITE was evaluated for its • ability to reach its intended audiences • effectiveness • behavior change messages • A multifaceted approach was used to evaluate the exhibit • suitability assessment • focus groups • observations • intercept surveys • lessons learned

5. Suitability Assessment of TheEYE SITE Kiosk Panels and Interactive Multimedia Program Purpose: to provide NEI with information on the audience suitability of educational materials presented in the exhibit.

6. Suitability Assessment continued • Process— • Identified experts in health communications, marketing, interactive health media, and low vision • Geographic representation • One group of experts reviewed the content of the 16 panels • Another group of experts reviewed the interactive multimedia program (CD-Rom)

7. Suitability Assessment continued • Modified an evaluation instrument—Suitability Assessment of Materials • Pre-tested instruments • Prepared 2 suitability assessment guides—one for the assessment of the kiosk panels, the other for the assessment of the interactive multimedia program. • Guides contained open ended questions, rating scales, and space for additional comments.

8. Kiosk Panels Content Layout Writing and conversation style Text and font Learning stimulation/ motivation Headers and topic captions Illustrations, pictures, graphics Cultural match Interactive Program Purpose Navigation Layout Content Motivation Calls to action Appropriateness of illustration, graphics, and video clips Writing and conversation style Source of information Sequence of information Appropriateness for all audiences Suitability Assessment continued

9. Suitability Assessment continued • Definitions and guiding questions were provided to help experts formulate a rating • Using a 5-point rating scale, experts rated each factor— 1=very poor, needs major work 2=poor, need some changes 3=just okay, needs minor changes 4=good, fine as is 5=excellent, perfect • Space was provided for comments

10. Focus Groups • Allow for in-depth probing and discussion of the specific components of THE EYE SITE • Provide information about the content of the kiosk panels • 2 focus groups • Elderly/low vision population aged 65 and older • General population aged 18 to 64

11. Focus Groups continued • Focus group participants provided information about— • Purpose of the exhibit • Comprehension of the material • Intended audience • Color, lighting, size, and layout • Interactive multimedia program • Interactive vs. written text

12. Focus Groups continued • Panels • most and least favorite • amount of text • use of audio buttons • Assistive devices • Exhibit’s sponsor

13. Observation and Intercept Survey • Conducted onsite at the shopping mall over a 5-day period at different time intervals • Trained staff observed and recorded the behavior and reactions of individuals • Nonuser intercepts • User intercept (self administered survey) • Incentives • Data captured on usage of the interactive multimedia program • Comment cards

14. Observation and Intercept SurveyContinued • From the data collected onsite, we were able to provide— • Demographics of exhibit users and nonusers • Users’ rating of the exhibit • What users’ intend to do with the information • Use of the interactive multimedia program • Nonusers’ reasons for not stopping at the exhibit • Panels viewed and amount of time spent • Users’ order of preference • Summary of users’ comments • Report of the usage of the interactive multimedia program

15. Lessons Learned • Conducted debriefing with host organization/ committee and shopping mall management • Topics discussed— • Planning for THE EYE SITE • Length of time exhibit should be displayed • Placement in the shopping mall • Advertising and media coverage • Use of volunteers • Outcomes/benefits • Challenges

16. Benefits of Employing a Multifaceted Approach to Evaluate an Educational Exhibit • Comprehensive • Information from a variety of sources • Refine/modify exhibit and interactive multimedia program • Continual feedback • Use results to market the exhibit • Validates exhibit is meeting its goals/objectives

17. Next Steps... • Future host organizations will be asked to collect data on— • Media activity (amount/type/reach) • Number/type of events sponsored by host organization • THE EYE SITE visitor survey • Comment cards • Number of referrals to low vision clinics/providers • Interactive multimedia program usage • Observations

18. For more information contact— Arlen Rosenthal ORC Macro 11795 Beltsville Drive, Suite 300 Calverton, MD 20705 301.572.0222 rosenthal@macroint.com