Usability and Human Factors

1. Usability and Human Factors Designing for Safety Lecture c This material (Comp 15 Unit 10) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003. This material was updated by The University of Texas Health Science Center at Houston under Award Number 90WT0006. This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/.

2. Designing for SafetyLecture c – Learning Objectives • Design a workflow analysis study (Lecture c) • Identify common sources of error documented in research studies in medicine (Lecture c) • Apply principles underlying the design of healthcare systems for safety (Lecture c)

3. Medical Error Reporting • Use of the taxonomy • MAUDE • Classified reporting better than current free text • Current systems may be overly specific to domain, task • Should allow ID of problems, solutions

4. Koppel (2008, 2009, 2010)Patient Safety • Sociologist, studying work in hospitals • CPOE is a major source of errors • Sociological causes of errors: • Vendors sell early (investment makes it less likely hospital will switch) • Less incentive to be responsive, make changes, feedback / adjustments

5. Koppel (2008, 2009, 2010)Patient Safety(Cont’d – 1) • Hospital administration: • Validate investment • Little incentive to discuss errors • Contracts may prohibit open discussion • Joint Commission rules: hospital and its clinicians are prohibited from not talking about patient safety-related errors in relation to HIT • Impossible to tell how system will work in situ until deployment

6. Koppel (2008, 2009, 2010) • Example of interface-created error: • “The impossibility of entering allergies, or when you enter an allergy, it wipes out the previous allergy. So if the first allergy was anaphylactic shock and the second was a mild rash to latex, anaphylaxis dies, disappears, and you get the mild rash to latex coming up”.

7. Koppel (2008, 2009, 2010)(Cont’d – 1) • Inability to talk, screenshots • Vendor in control of information, chooses which flaws to remedy • Realistic testing scenarios rarely used in purchasing, but should be standard

8. Koppel (2008, 2009, 2010) – Study of Workaround to Barcode Medication Administration • BCMAs intended to ensure 5 rights: • Patient • Drug • Dose • Route • Time • Observation, interviews, failure analyses, log data at 5 hospitals • Found overrides 4.2% of patients, 10.3% of medications • Possible consequences: • wrong admin, doses, times, formulations • Need for attention to in-situ use to ensure safety

9. Koppel (2008, 2009, 2010) (Cont’d – 2) • Medication administration stage 26-32% of adult med errors; 4 – 60% pediatric errors • Less likely to be intercepted • Barcode systems: integrate MAR with admin process, ensure protocols followed • Workarounds: omission of process steps (e.g. omit visual check of medication list, name, dose) • Steps out of sequence (e.g. document before admin)

10. Koppel (2008, 2009, 2010) – Medication Workarounds • Unauthorized BCMA process steps (e.g. barcode placed on another object, not patient) and scanned • Alert overrides: cause due to smudged/torn medication label, wristband missing, different dose; alert confirming scan does not match med order or patient ID • Causes: technology-related, e.g. software or hardware problems

11. Probable Causes of Workarounds (Koppel, 2008, 2009, 2010) • Task-related: user unfamiliar with protocols, believe it slows performance, e.g. several patients’ meds on one tray • Organizational: policies incompatible with safety, e.g. patients or meds without barcodes, codes covered by label, pharmacy sending partial doses. • Patient-related: e.g. refusal, vomiting, sleeping, agitated, central lines, contact isolation, home-bought meds without barcodes • Environmental: e.g. lack of wireless BCMA, room or doorway hinder bedside access of BCMA, meds stored remote from scanner (e.g. in refrigerator) require trips, loud noise may prevent hearing alarms

12. Prevention of Errors From Workarounds • Key to impact is the multiplicity of relationships: many causes, each causing many workarounds • Most due to time-saving efforts • Staffing inadequacy • Best done study of system design, workflow integration, integration improvements • Make easier to follow correct procedures, harder to workaround • Not obscure, rare, or secret

13. Koppel – JAMA 2005 – Role of CPOE in Facilitating Errors • Widely discussed mixed methods study of house staff CPOE at tertiary-care hospital 2002-2004; older system • CPOE facilitated 22 types of medication error risks • Fragmented displays, misinterpretations, function separation->dosing errors, incompatible orders, inflexible order formats-> wrong orders. • 75% of house staff report observing these, weekly error risks

14. Walker - 2008 • Interim, pragmatic framework for assuring EHR safety • Use EHR as tool for process improvement • Requires new organizational policies, processes, work systems, job descriptions, education • Explicitly recognize safety as a goal

15. Walker (Cont’d – 1) • Reporting to national clearinghouse • For reporting and viewing other institutions’ data in anonymized form • Vendors likewise required to report • Rapid notification to multiple stakeholders when significant flaws/incidents identified

16. SAFER Guides • Designed to help healthcare organizations conduct self-assessments to optimize the safety and safe use of EHRs • Developed based on the best evidence available including literature review, expert opinion, and field testing at a wide range of healthcare organizations ("Health IT and Patient Safety Goals | Policy Researchers & Implementers | HealthIT.gov", 2014)

17. Error Reporting: ECRI Institute • Encourages healthcare providers, patients, and manufacturers to report medical-device-related incidents and deficiencies • Free, voluntary, and confidential • Informs manufacturers and encourages them to correct the problem • Publishes resulting safety information ("Report a Device Problem", n.d.)

18. Error Reporting: FDA • Medical device reports are submitted to the FDA by • mandatory reporters (manufacturers, importers and device user facilities) • voluntary reporters (health care professionals, patients, caregivers and consumers) using Medwatchweb site • Note that EHRs are not (yet) considered medical devices by FDA("Medical Device Reporting (MDR)", 2015)

19. Searching for Reported Errors: FDA MAUDE Search: Problems listed under “Medical computers and software” ("MAUDE - Manufacturer and User Facility Device Experience", 2016)

20. Recommended Prevention/Management Strategies • Local implementation teams should notify EHR vendor • Vendor provides software enhancement • If not, organization changes process or policy • If not, organization removes function from EHR • If not, users warned during training (and if necessary, repeatedly in practice) • Organizations track mitigation efforts, incidents

21. Prevention (Cont’d – 1) • 2 teams: shadow trainers observe users, answer questions, JIT education, report flaws to 24-hour command center • Command center: makes rapid fixes, communicates them to users • 2nd team: clinical ops leaders, EHR support, informaticians, security, patient safety, risk management, PR • Meet on few hours notice to correct, warn users, remove flaw from system • Vendors could provide incident management for smaller orgs.

22. Canada – Health Infoway (the Integrated Centre for Care Advancement through Research (iCARE), Canada Health Infoway (Infoway), and the Canadian Patient Safety Institute (CPSI)., n.d.)

23. Problem – Lack of Knowledge • Large scale studies not done due to lack of concept, technology standardization • Largely anecdotal, often confidential reports • Lack of measurement, pre, post • Canada: jurisdictional differences

24. Design Patterns and New Methods • Formulate library of design patterns addressing common or identified problems (e.g. similar drug names) • Dedicated HIT labs attached to hospitals • Invite the participation of a broad base of end-users (e.g. via web 2.0) • (a) Users may submit “bad” designs and “good” designs • (b) Users may vote for design candidates to elect the ones that make make most sense to their work • Simulation/gaming exercises (e.g. OR virtual game for team coordination) • Sandboxes for users • Permit free discussion

25. Bad Informatics Examples and Solutions • The VA's medical data -- vital signs, lab results, active meds -- sometimes popped up under another patient's name • Failed to clearly display a doctor's stop order for a treatment, leading to reported cases of unnecessary doses of intravenous drugs such as blood-thinning heparin • Baby boy died after an untrained doctor pressed the wrong button on his bypass machine because it was a less "horrid" color – doctor did not know correct button

26. Multiple Name Formats 1.1 Table: Senathirajah, Y. (2010).

27. Methods of Preventing Error – Tall Man Lettering Senathirajah, Y. (2010).

28. Regulation? • The United States Food and Drug Administration (FDA), which regulates medical devices of all sorts, has started to look at EHRs from a product-safety standpoint • Current ONC regulations mandate functionality, not safety standards

29. ONC Certification Standards for Safety-Enhanced Design (SED) • Certification requires designers: • Use a formal User Centered Design (UCD) process during development • Perform summative usability testing on specific areas of the product ("Safety Enhanced Design_EHR - NCCD - The University of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics", n.d.)

30. Requirement 1: User Centered Design (UCD) • A formal UCD procedure must be followed during design and development, and the procedure must be identified or described as part of the certification process. • Several ISO standards • ISO 9241-11 • ISO 9241-210 • ISO/TR 16982 • IEC/ISO 62366 • ISO/IEC 25062 • NISTIR 7741 & 7742 ("Safety Enhanced Design_EHR - NCCD - The University of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics", n.d.)

31. Requirement 2: Summative Testing, 1 • Summative usability testing: described in Procedure for §170.314(g)(3) Safety-enhanced design • NISTIR 7742 Customized Common Industry Format Template for EHR Usability Testing: outlines how the ONC requires documentation of the results of summative usability testing. • SED Use case scenarios and testing methods are available at SBMI Website (Link provided in reference section) ("Safety Enhanced Design_EHR - NCCD - The University of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics", n.d.)

32. Requirement 2: Summative Testing, 2 • In the test procedure, real users run the EHR with a series of fundamental tasks. • Goal: to measure the usability and safety of the EHR by documenting: • its effectiveness in performing each task • its efficiency in performing that task, and • the user’s satisfaction from performing the task • Each task: • is a fundamental EHR capability • each must be tested and reported separately

33. Requirement 2: Summative Testing, 3 • The tasks cover these capabilities (areas of focus): • Computerized provider order entry system (CPOE) • Drug- drug, drug-allergy interaction checks • Medication list • Medication allergy list • Clinical decision support • Electronic medication administration record • Electronic prescribing • Clinical information reconciliation

34. Safety Enhanced Design Briefs • Developed by The National Center for Cognitive Informatics and Decision Making in Healthcare • Available at SBMI Website (Link provided in reference section)

35. NCCD Safety Enhanced Design Briefs General Briefs • Making effective use of color • Effective table design • Reducing wrong patient selection errors • Result management Specific Meaningful Use Cases • Drug-drug, drug-allergy interaction checks • Medication list • Medication allergy list • Clinical decision support • Electronic prescribing • Clinical information reconciliation • Medication reconciliation • Problem reconciliation • Computerized practitioner order entry

36. Safety Enhanced Design Brief:Making Effective Use of Color • Carefully used colors can: • improve efficiency and safety of health information systems by drawing attention to important items • Make it easier to perceive differences and trends • Incorrectly used colors can: • make a display hard to use • hard to interpret • mislead

37. Safety Enhanced Design Brief: Color • To maximize the communication benefits of color, design: Use gray scale, then add color sparingly • To group items into different categories: Use no more than 7 colors (4 recommended) ("Safety Enhanced Design_EHR - NCCD - The University of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics", n.d.)

38. Safety Enhanced Design Brief: Color (Cont’d – 1) • To show sequential ranges of quantitative values • Use 1 color (for sequential) and 2 colors (for diverging) values • Vary color intensity from pale (low values) to darker (extreme values) ("Safety Enhanced Design_EHR - NCCD - The University of Texas Health Science Center at Houston (UTHealth) School of Biomedical Informatics", n.d.)

39. Safety Enhanced Design Brief: Color (Cont’d – 2) 4. To ensure consistency, learnability, and to prevent misinterpretation, create rules for: • Colors for critical values • Colors for severity of warnings and alerts, etc. • Colors for different categories of items • Colors combined with differentiators (tooltips, symbols, icons, positions) 5. To ease understanding and learnability of colors • Use text, tooltips or legends (National Center for Cognitive Informatics and Decision Making in Healthcare, 2016)

40. Safety Enhanced Design Brief: Color (Cont’d – 3) • Use color-blind friendly colors • 10% of men and 5% of women are color-blind • Combine color with an image, shape, position, or text to convey same meaning. • To select appropriate color schemes: • Match schemes to data types and support color-blind safe choices (http://colorbrewer2.org/) • Preview design as it would be seen by a color-blind user (http://www.color-blindness.com/coblis-color-blindness-simulator ) (National Center for Cognitive Informatics and Decision Making in Healthcare, 2016)

41. Designing for SafetySummary – Lecture c • Designing for safety in healthcare IT is a difficult, ongoing, and rapidly changing process in which research and careful planning are required.

42. Designing for SafetyReferences – Lecture c Reference Food and Drug Administration. (n.d.) Medical Device Reporting. Retrieved June 24, 2016 from http://www.fda.gov/MedicalDevices/Safety/ReportaProblem/default.htm/ Food and Drug Administration. (n.d.) MedWatch Online Voluntary Reporting Form. Retrieved June 24, 2016 from https://www.accessdata.fda.gov/scripts/medwatch/index.cfm?action=reporting.home. Health IT.gov. (2014). Health IT and Safety. Retrieved June 24, 2016 from https://www.healthit.gov/policy-researchers-implementers/health-it-and-safety Interview with Ross Koppel http://histalk2.com/2009/08/12/histalk-interviews-ross-koppel/ Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human. Committee on Quality of Healthcare in America Institute of Medicine. Washington, DC: National Academy Press; 1999. Koppel, R. (2010). EMR entry error: not so benign. AHRQ Morbidity and Mortality Rounds on the Web. Case & Commentary: Medicine, April 2010. Available at http://www.webmm.ahrq.gov/case.aspx?caseID=199 Koppel, R. (2008). Wetterneck T, Telles JL, Karsh BT. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc. 2008 Jul-Aug;15(4):408-23. Koppel, R., and Kreda, K. (2009). Health Care Information Technology Vendors’ “Hold Harmless” Clause: implications for patients and clinicians., JAMA.2009; 301:1276-1278. Koppel, R., Metlay, J.P., Cohen, A., Abaluck, B., Localio, A.R., Kimmel, S.E., et. al. (2005). Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005 Mar 9;293(10):1197-203.

43. Designing for SafetyReferences – Lecture c Reference Koppel, R., Metlay, J.P., Cohen, A., et. al. (2005). Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293:1197-1203. National Center for Cognitive Informatics & Decision Making in Healthcare (n.d.) What is Safety Enhanced Design?. Retrieved June 29, 2016 from https://sbmi.uth.edu/nccd/SED/. National Center for Cognitive Informatics & Decision Making in Healthcare: Safety Enhanced Design Briefs: Retrieved June 29, 2016 from https://sbmi.uth.edu/nccd/SED/Briefs/ National Center for Cognitive Informatics & Decision Making in Healthcare: Inspired EHRs: Designing for Clinicians eBook: Retrieved June 29, 2016 from https://sbmi.uth.edu/nccd/SED/ebook/ National Center for Cognitive Informatics and Decision Making in Healthcare: SED Use Case Scenarios and Testing Methods: Retrieved June 29, 2016 from https://sbmi.uth.edu/nccd/SED/Scenarios/ Walker, J.M., Carayon, P., Levesond. N., Paulus, R.A., Tookere, J., Chin, H., Bothe, A. Jr., Stewart, W.F. (2008). Perspectives on Informatics EHR Safety: The Way Forward to Safe and Effective Systems. JAMIA 2008 Zhang, J., Patel, L.V., Johnson, R. T., &. Shortliffe, H.E. (2004). A cognitive taxonomy of medical errors. Journal of Biomedical Informatics 37:193–204

44. Designing for SafetyReferences – Lecture c (Cont’d – 1) Images Slide 16: Health IT.gov. (n.d.) Health IT and Safety. Retrieved June 24, 2016 from https://www.healthit.gov/policy-researchers-implementers/health-it-and-safety Slide 19: Food and Drug Administration. (n.d.) MAUDE - Manufacturer and User Facility Device Experience. Retrieved June 24, 2016 from http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfMAUDE/search.CFM Slide 22: Electronic Health Records and Patient Safety A joint report on Future Directions for Canada. Canada Health Infoway. Retrieved on September 8th, 2010 from https://www2.infoway-inforoute.ca/Documents/EHR-Patient%20Safety%20Report.pdf Slide 27: Senathirajah, Y. (2010). Department of Biomedical Informatics, Columbia University Medical Center, New York, NY. Slide 37: National Center for Cognitive Informatics & Decision Making in Healthcare. Retrieved June 29, 2016 from https://sbmi.uth.edu/dotAsset/8ba0b036-780f-49cf-a936-a13aa39464c0.pdf Slide 38: National Center for Cognitive Informatics & Decision Making in Healthcare. Retrieved June 29, 2016 from https://sbmi.uth.edu/dotAsset/8ba0b036-780f-49cf-a936-a13aa39464c0.pdf

45. Usability and Human FactorsDesigning for Safety Lecture c This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC000003. This material was updated by The University of Texas Health Science Center at Houston under Award Number 90WT0006.