Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey

1. Innovative Use of Multiple Methods to Assess Health Informatics System StandardsAPHA 2013 -- Boston, MANovember 6, 2013 Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey

2. Presentation Outline • Background • Purpose • Framework • Evaluation Data Collection • Results • Conclusion

3. BACKGROUND

4. NCS Vanguard Study • The National Children’s Study (NCS) • Large-scale, dynamic health study • 20-year time horizon • NCS Vanguard Study • Pilot study for the NCS main study • Uses multiple information management systems (IMS) to support one research data collection effort

5. NCS Vanguard Study IMS Evaluation • Assessment of IMS to see how well meeting the study’s needs • Current needs • Future projected needs

6. PURPOSE

7. IMS Evaluation • Develop and implement an evidence-based evaluation approach • Answer fundamental questions: • Does each IMS support the study’s basic functional requirements? • Does each IMS perform as expected? • Leverage evaluation framework and methodology

8. Evaluation Phases • Phase 1 • Short-term: 2-3 months • High level assessment of basic functionality of IMS solutions • Completed in Fall 2012

9. Evaluation Phases (continued) • Phase 2 • Longer-term: 9-12 months • Identify/refine preferred practices, processes, and functionality of an IMS solution used to support NCS Vanguard protocol • Identify gaps in IMS solutions with respect to evaluation criteria

10. FRAMEWORK

11. Phase 1 Evaluation Framework

12. EVALUATION DATA COLLECTION

13. Phase 1 IMS Evaluation Benchmarking System Characteristics Real-Case Scenario-Testing • Reviewed system and user documentation • Performed system stand up for available solutions • Developed standardized scenarios • Conducted testing using host-provided environment

14. Example: Role-Based Scenario Functions Tested

15. Phase 1 Overview: IMS Functional Requirements Tested

16. RESULTS

17. Benchmarking • Benchmarking results showed where systems were strongest and where they needed additional functionality

18. Review of Documentation on IMS Characteristics *Few sites collecting biospecimens at the time.

19. Real-Case Scenario Functional Testing • Results provided: • Objective information on functionality • Qualitative input from IMS testers

20. Data Collector User Role Scenario Testing

21. Functionality of IMS Solution Characteristics

22. Translating Results into Use • Results used to: • Identify high-level areas already meeting requirements • Identify specific gaps or areas needing improvement • Inform Phase 2 assessment

23. CONCLUSION

24. NCS Vanguard Study IMS Evaluation • Framework • Grounded in theory • Collaboration • Involving and communicating with multiple stakeholders • Critical when testing systems • Innovation • Moved beyond traditional IT evaluation to include aspects of program evaluation • Implemented ways to test systems, not users, using protocol-driven real-life standardized scenarios

25. Thank You! Authors • Linda Piccinino, Senior Evaluator, SSS, lpiccinino@s-3.com • Andrew Westdorp, Center Director, SSS, awestdorp@s-3.com • Jennifer Kwan, Evaluation Team Co-Lead, Program Office, National Children’s Study, Eunice Kennedy Shriver National Institute of Child Health and Human Development, jennifer.kwan@nih.gov • Lydia Rogers, Project Director, SSS, lrogers@s-3.com • Jean Wilson, Senior Study Director, SSS, jwilson@s-3.com • Sue Griffey, Vice President and Director, Evaluation Center, SSS, sgriffey@s-3.com