The Line Operations Safety Audit: Validating Behavioural Markers in Aviation

1. The Line Operations Safety Audit: Validating Behavioural Markers in Aviation GIHRE - Zurich The University of Texas Human Factors Project Austin, Texas 8 July, 2001

2. LOSA � Normal Operations Monitoring Collect data benchmark safety: Crew performance � strengths and weaknesses Threat and error management CRM performance System performance � strengths and weaknesses Culture Airspace System � airports and navigational aids Aircraft design / automation Standards / Training / Safety / Maintenance Crew support � ATC, Cabin, Ground, and Dispatch

3. The Evolution of LOSA and Behavioural Markers Initial definition of markers from analysis of accidents and incidents Validation of specific markers from correlations with expert (global) ratings of crew performance LOSA focus on process assessment LOSA shift (1996) to include specification of threats, errors, and their management Evolution in analysis of LOSA data to include relationships between markers and threat and error and undesired aircraft state management

4.  Pending/in progress Ansett Australia COPA Emirates Frontier Continental Express Middle East and European carriers (TBD) Threat and Error Management LOSA Continental Latin America Continental Express Gulfstream Express Air New Zealand Air Micronesia Continental Delta US Airways Cathay Pacific EVA (Taiwan)

5. Global Consolidation � Regional Seminars North America � ATA Ops Forum (Marcb 2001) Asia � Hong Kong � 15 countries Latin America � Panama Nov. 01 � Central & South American Europe & Middle East � Dubai, Jam. 02

6. Threats Defined Environmental Threats Adverse WX Terrain Airport conditions Heavy traffic / TCAS events ATC Threats Command events / errors Language difficulties Aircraft Threats Aircraft malfunctions Automation events

7. A New Typology of Crew Error (tentative) 1. Task Execution � Unintentional physical act that deviates from intended course of action Wrong altitude setting dialed into the MCP 2. Procedural � Unintentional mental (cognitive) slip, lapse, or mistake when trying to follow required course of action Slip � skipping an item on a checklist Lapse � forgetting a required briefing Mistake � choosing the wrong checklist in an abnormal situation 3. Communication � Failure to transmit information, failure to understand information, failure to share mental model ex) Miscommunication with ATC 4. Decision � Choice of action unbounded by procedures that unnecessarily increases risk ex) Unnecessary navigation through adverse weather 5. Intentional Noncompliance � violations ex) Performing a checklist from memory

8. Consequential errors lead to another error or to an UNDESIRED AIRCRAFT STATE 

9. Decision Error Decision that increases risk in a situation with multiple courses of action possible time available to evaluate alternatives no discussion of consequences of alternate courses of action no formal procedures to follow

10. Undesired Aircraft State Defined

11. Threat Analysis

12. Threat Management Analysis Goal � Build a threat profile for flight operations Analytical Questions What is organization�s exposure to threats and how are they managed? How does organization compare to the LOSA Archive? What type of threats do flight crews most frequently encounter? What type of threats are most difficult to manage? Are there phase of flight effects on threat management? Are there operations or fleet differences in threat management?

13. Threat Baselines

14. LOSA error frequencies

15. Error Analysis

16. Error Management Analysis Goal � Build a crew error profile for organization�s flight operations Analytical Questions What is organization�s baseline of errors and how are they managed? How does organization compare to the UT LOSA Archive? What were the most frequently crew errors committed? What types of crew error are the most difficult to manage? What were the most frequent undesired aircraft states and how were they managed? Are there phase of flight effects on error management?

17. Error Baselines

18. LOSA error frequencies

19. The Importance of Violations Airlines cannot allow violations to normalize Why? Violations cultivate complacency and a disregard of rules Crews that commit at least one intentional noncompliance error are twice as likely to: Commit unintentional errors (Procedural, Communication �..) Commit consequential errors that lead to additional errors or undesired aircraft states

20. Error Responses and Outcomes

21. Undesired State Analysis

22. Undesired Aircraft States

23. Error Management by Phase of Flight

24. The Blue Box

25. The Blue Box

26. Behavioural Marker Validation

27. An Expanded Definition of CRM

28. Threat and Error Countermeasures CRM skills are best defined as threat and error countermeasures. The following have been validated as critical in LOSA Different skills play different roles in threat, error management, and undesired state management Team Climate � critical in all areas of crew performance Leadership, communication environment, and flight attendant briefing Planning � critical in threat management SOP briefings, plans stated, workload assignment, and contingency mgmt Execution � critical in error management Monitor / cross check, workload mgmt, vigilance, and automation mgmt Review and Modify � critical in undesired aircraft state management Evaluation of plans, inquiry, and assertiveness

29.  LOSA Data Lead to a Model of Threat and Error Management (UT-TEMM)

30. Why Develop Such a Model? Accurately describe and investigate threat and error and their management Determine latent factors that lead to system failures Go beyond root cause analysis to determine how complex situations are managed effectively Data to teach from good examples as well as system failures Used for analysis of accidents and incidents

31. Latent systemic threats Latent systemic threats Culture (National, Professional, and Organizational) Airports and navigational aides Aircraft design, automation, and maintenance Regulations, policies, and procedures Training curriculum and implementation Flight crew support (ATC, MX, Ground, Dispatch, & Cabin) Latent threat often detected only after an incident or accident Normal operations data identify latent systemic threats before they become consequential

33. Predicting Line Performancefrom Safety Culture

34. Forming a Safety Culture Scale Pilots observed in LOSA also completed a survey with items from the UT Flight Management Attitudes Questionnaire Items related to organizational culture regarding safety were summed to form a Safety Culture Scale

35. Safety Culture Scale Items Read the itemsRead the items

36. Classifying the Pilot Population Low Safety Culture Crews had captains and first officers in the bottom 33% of distribution Average Safety Culture Crews had captains and first officers in the middle 33% of distribution High Safety Culture Crews had captains and first officers in the top 33% of distribution Safety indices for each group were contrasted

38. Safety Indices Violators make more errors � link to BobViolators make more errors � link to Bob

39. Applying lessons from aviation to medicine

40. Major issues in the OR Threat and error are common The OR is a complex environment (more than aviation) multiple teams part of a larger organization Unclear lines of authority who is in charge? Communication across disciplines Multi-cultural environment Professional culture Denial of vulnerability

41. Observable Medical Error 1. Task Execution � Unintentional physical act that deviates from intended course of action Nicking of artery during surgery 2. Procedural Countermeasure � Unintentional failure to follow mandated procedures Failure to follow required treatment protocols 3. Communication � Failure to transmit information, failure to understand information, failure to share mental model Failure to communicate critical patient information during shift change 4. Decision � Choice of action unbounded by procedures that unnecessarily increases risk Failure to use temperature probe during surgery 5. Intentional Noncompliance � violation of formal procedures or regulations Surgeon elects not to sign surgical site

42. Procedures as Countermeasure and Problem Standard Operating Procedures (SOP) were aviation�s first countermeasures against threat and error Aviation is arguably over-proceduralized Tombstone regulation Medicine is under-proceduralized Example: checklists are a critical error countermeasure

43. Adapting the Threat and Error Management Model to Medicine

46. Where are we going?

47. Priority Items for Medicine Behavioural markers have been refined and limited in aviation We now know that markers are differentially related to situations (i.e., threat, error avoidance, undesired state) It is likely that the potent markers from aviation will apply in specific medical situations Operating theatre ER Obstetrics (simulation) Markers need to be defined and validated in simulation as well as the OR TEMM has been adopted by Kaiser-Permanente as a teaching tool and a means of analyzing adverse and sentinel events (10,000 doctors) The University of Texas has also committed to a comprehensive patient safety programme (6 medical schools)