1 / 13

Reliability of a Procedural Checklist as a High Stakes Measurement of Advanced Technical Skill

Sarah E. Peyré, EdD, Christian G. Peyré, MD, Jeffrey A. Hagen, MD, Maura E. Sullivan, PhD Department of Surgery, Keck School of Medicine University of Southern California. Reliability of a Procedural Checklist as a High Stakes Measurement of Advanced Technical Skill. Disclosure.

astrid
Download Presentation

Reliability of a Procedural Checklist as a High Stakes Measurement of Advanced Technical Skill

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sarah E. Peyré, EdD, Christian G. Peyré, MD, Jeffrey A. Hagen, MD, Maura E. Sullivan, PhD Department of Surgery, Keck School of Medicine University of Southern California Reliability of a Procedural Checklist as a High Stakes Measurement of Advanced Technical Skill

  2. Disclosure • 2006 SAGES Endoscopic-Laparoscopic Research Grant: Advanced Laparoscopic Skills Competency: Developing an Assessment and Feedback Evaluation Tool for Laparoscopic Nissen Fundoplication

  3. Scientific Testing of Assessment Instruments • Validity and reliability of technical skill assessment instruments need to be measured before implementation • APA/AERA/NCME “Standards” provide rigorous, well-proven guidance on determining the validation and reliability of assessments

  4. Measuring Technical Skill • Previous work has focused on measurement within surgical skill laboratory settings and video recordings of basic operative procedures • Psychometric Properties of Technical Skills Assessment • Validity – Instrument measures the underlying construct that it is meaning to measure • Reliability – An estimate of the dependability of a measurement device or test

  5. Aim • To determine the reliability of a previously validated laparoscopic Nissen fundoplication procedural checklist as a measurement of advanced technical skill when using multiple raters reviewing the same operative performance

  6. Methods • Five surgeon evaluators independently assessed video recordings of two expert surgeons performing a laparoscopic Nissen fundoplication • Inter-rater reliability was analyzed by percent agreement and Fleiss Kappa coefficient

  7. Procedural Checklist • A task analysis and modified Delphi technique was used to develop a 65 step procedure specific checklist for laparoscopic Nissen fundoplication • Checklist was validated using a cohort of junior and senior surgical trainees • Peyre, et al. Surg Endoscopy. Epub 2008.

  8. Checklist Components

  9. Checklist Components

  10. Percent Agreement

  11. Fleiss Kappa Coefficient • The Fleiss (1971) Kappa produces a coefficient that is the mean probability of all the possible rater combinations

  12. Discussion • Objective assessment and feedback on technical performance is critical for training surgeons • Current efforts focus on global rating scales and general procedural checklists • Benefits: Ease of use, requires little time, no special training • Drawbacks: Poor inter-rater reliability, lack of concrete performance feedback

  13. Conclusions • The percent agreement and Kappa coefficients indicate a high degree of reliability (>.80) • Results support the use of checklist for high stakes assessment of a laparoscopic Nissenfundoplication • A task analysis and modified Delphi is effective in creating valid and reliable assessment tools to measure advanced surgical skill

More Related