1 / 31

Petrucci AM. MD 1 , Nouh T. MD 1 , Boutros M. MD 1 , Gagnon R. PhD 3 , Meterissian S. MD 1, 2

Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key. Petrucci AM. MD 1 , Nouh T. MD 1 , Boutros M. MD 1 , Gagnon R. PhD 3 , Meterissian S. MD 1, 2 Department of Surgery, McGill University 1

eryk
Download Presentation

Petrucci AM. MD 1 , Nouh T. MD 1 , Boutros M. MD 1 , Gagnon R. PhD 3 , Meterissian S. MD 1, 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM. MD1, Nouh T. MD1, Boutros M. MD1, Gagnon R. PhD3, Meterissian S. MD1, 2 Department of Surgery, McGill University 1 Center for Medical Education, McGill University 2 Faculty of Medicine, University of Montreal 3 Association for Surgical Education Paper Session 1 March 22, 2012

  2. Disclosures The authors have no disclosures to declare

  3. Current Assessment Tools • MCQ • Rich context MCQ • Orals • Short answer questions • Long answer questions The perfect assessment tool?

  4. Graduating general surgery residents Multiple choice examination + Oral examination

  5. What is Reasoning? Institute of Applied Research in Health Sciences Education http://www.script.md/portal/en/script_en.html

  6. SCT: from memorizing to reasoning • New tool intended to assess mental processes in medical practice • Can be used to evaluate a trainee’s approach to ill-defined problems including those encountered by expert surgeons

  7. Example -2 Contra-indicated or almost totally -1 Not useful or even detrimental 0 Nor less nor more useful +1 Useful +2 Necessary or absolutely necessary

  8. The Scoring Grid(Modal Experts’ Choice) Nouh, T., et al. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. The American Journal of Surgery, In press

  9. What have we done in the past?

  10. Preliminary Study:McGill Universityn= 36 Meterissian, S. et al. (2007). Is the script concordance test a valid instrument for assessment of intraoperative decision-making skills? The American Journal of Surgery, 193, 248-251.

  11. Pan Canadian Studyn= 202 Nouh, T., et al. (2012). The script concordance test as a measure of clinical reasoning: A national validation study. The American Journal of Surgery, In press

  12. Purpose To determine whether using a specialty-specific scoring key would result in a progression of scores from R1 to R5, with the R5 residents having the highest scores and therefore improving the validity of the scoring key

  13. Methods

  14. Participants n= 202 (Nouh, T., et al, 2012) n= 25 McGill n tot= 227

  15. Specialty Specific Scoring Key Colorectal Hepatobiliary Endocrine EXPERTS Surgical Oncology Thoracic Trauma/ ACS

  16. Creation of amalgamated expert exams EXPERT 1 EXPERT 2 EXPERT 3 EXPERT 4 EXPERT 5

  17. Statistical Analysis • Reliability: Cronbach alpha coefficient • Analysis of item-to-total item correlation used to select the best items for the final analysis • Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts • All p values at alpha<5% were considered significant

  18. RESULTS

  19. 153-question test Cronbach α = 0.67 Elimination of items with a negative item-to-total item correlation Final exam 100-question test Cronbach α = 0.81

  20. Results ? Scores rising with increasing level of residency: p< 0.001

  21. Number of expert panel members matters! Gagnon et al. Medical Education, 2005: • The mean of residents’ scores increased with increasing panel size. “…the number of points earned on an SCT is influenced by the number of members used in the panel”.

  22. Mean Score Graph Scores rising with increasing level of residency: p< 0.001

  23. Juniors vs Seniors The average score of junior residents proved to be significantly lower than the average score of senior residents: p< 0.001

  24. Discussion • The results of this study suggest that the scoring key should be developed through the use of specialty- specific experts • This may be due to the increasing subspecialization of General Surgery • This has important implications in the application of the SCT on a wider level

  25. Limitations • We used only 5 amalgamated general surgery experts for each surgical discipline instead of the recommended >10 Gagnon et al. Medical Education 2005 • The resident pool consisted mostly of the same residents from our last study with the addition of 25 new residents from our institution

  26. Future implications… National wide study using the specialty-specific expert scoring key. The SCT might be ready for a national in-training formative examination.

  27. Acknowledgements • Dr. Sarkis Meterissian • Dr. Robert Gagnon • Dr. Thamer Nouh • Dr. Marylise Boutros

  28. Thank you

More Related