Residents In Trouble

1. Residents In Trouble M. Rebecca Hoffman, MD, MSPH December 8, 2008 Academy for Scholarship in Education

2. Overview • Scope of the Problem • Literature Review • Study Design • Study Results (in progress)

3. Scope of the Problem • The goal of Family Medicine Residency • Residents’ backgrounds • No guidance

4. What does the RRC say? • Must have advising system • Must give regular feedback BUT… no real guidance regarding what/how

5. Consequences of RIT • For the individual RIT • Psychological stress • Financial stress • Eventual job security • For the program • Adequate coverage of duties • Morale • Faculty time/energy • Dismissal

6. Consequences… • Other consequences • Recruiting • Impact on patient care

7. Literature Review • Not much on the prevalence of RIT • Reamy et al (1): 25 year in-depth study of one family medicine residency program • 9.1% • Yao and Wright (2): Survey of Internal Medicine PDs– • 6.9% • Williams et al (3): SIU Gen Surgery Residency • 22%

8. Kinds of Problems • Knowledge Deficits • Attitudinal Problems • Interpersonal Conflict • Psychiatric Illness • Substance Abuse • Family Stress • Relationship Disruption

9. Other Classification • Academic Performance • Classroom, test • Clinical Performance • Applied knowledge/skills • Professional Behavior • Workplace behavior, relationships with other professionals, etc • Others • Difficult to classify

10. So we can name it… • But what do we do with the information? • Literature reviews: • “Need to do better” • Core content review, quizzes • Counseling • Treatment of illnesses • Repetition (rotation, year) • Probation • Dismissal • Others

11. Summary of the problem • We know it’s there, but we aren’t sure exactly how common it is • We aren’t good at predicting who will struggle • Once we identify them, we don’t know what to do with them

12. RIT Family Medicine Study Primary Aims 1) Determine the prevalence of RIT in the SIU Family Medicine Residency Programs 2) Describe and categorize the types of troubles encountered in the FCM residency programs

13. RIT Aims... 3) Identify and describe the kinds of processes the programs employ to address deficiencies with RIT 4) Describe outcomes of the residents who underwent remediation 5) Identify early indicators to predict RIT

14. RIT Study Design • Chart review/resident record -based • Prevalence over significant timeframe • 10 years of entering classes • Entering class of 1993-entering of 2003 • Why? • De-identified data from all 4 programs • To further assure confidentiality • No one program singled out

15. RIT Study • Initially modeled after Reamy et al study (1) after discussion with Dr. Reamy about the process in his residency program • Assistance from Reed Williams, PhD and Nicole Roberts, PhD

16. Methods • Rosters of all entering residents for the 10 years from all programs • Random ID numbers assigned • Records from each program reviewed in their entirety • Mostly paper • Some electronic

17. Methods… • Data collection instrument created based on personal experience, literature, and discussion with Reed Williams and Nicole Roberts • Feedback from academic faculty • Suggestions incorporated as needed

18. Collected for all residents: • US Medical School Grad (Y/N) • USMLE or NBOME or COMLEX scores • Step 1, Step 2, Step 3 (when available) • Gender (M/F) • ITE Scores • PGY1, PGY2, PGY3 (when available) • Did the resident have substantial difficulties? (Y/N)

19. For those who struggled • Time frame- when first identified • Categorization of the main problem • Academic, Clinical, Professionalism, Other • Main performance area involved • See list • Breadth of performance areas • Academic, Clinical, Professional, Other • All performance areas involved • See list

20. For those who struggled • What remediation attempted? • See list • What was the final resolution? • Graduated with concerns • Graduated without concerns • Voluntarily left • Dismissed • Finished on probation • Not documented

21. Severity Rating • Compared to other residents with performance problems, this resident’s problems were: • Likert scale, 1-5 • 1 among the least serious • 5 among the most serious • Narrative descriptions of overall case, time course, etc

22. Example of Survey Instrument • Survey Monkey for data entry

23. Data Collection • 2 raters • Independently reviewed same 5 charts at first program to establish inter-rater reliability, then 3 charts at each other program

24. Data Collection… • Then independent record review • Any case the reviewer identified as a RIT was then reviewed by the other reviewer independently, ratings discussed after each case • “Borderline cases” also reviewed by both

25. Data Analysis • Data entered into Survey Monkey for ease of collection • Analysis: descriptive and basic statistics • When all data complete, analysis using SPSS

26. Results • 96 Residents reviewed (as of 12/4/2008) • US Med School Grads: 69% (66) • Male: 65% (62) • Female: 35% (34) • Prevalence of documented RIT: 38.5% (37)

27. RIT (N=37) • US graduates: 59.5% (22) • Foreign graduates: 40.5% (15) • Male: 70.3% (26) • Female: 29.7% (11) • Male, US: 40.5% (15) • Male, Non-US: 29.7% (11) • Female, US: 18.9% (7) • Female, Non-US: 10.8% (4)

28. RIT

29. Most Important Single Problem Area? • Knowledge (10) • Putting everything together (3) • Data interpretation/diagnosis (2) • Data collection (2) • Incomplete paperwork/charts (2) • Lack of motivation/interest (2) • Treatment/management (2)

30. Breadth of Problems

31. Breadth of problems • Number of performance areas involved ranged from 1 to 23 • Average: 7 • See breakdown of individual RIT on handout

32. Remediation • Ranged from “none” or “told to improve” to 12 or more interventions for a single resident • Most commonly reported: Increased meetings with advisor/mentor

33. Outcomes of RIT (residency) • 26 graduated from the program (70.3%) • 11 did not graduate (29.7%) • Left voluntarily: 6 • Changed specialty: 2 • Dismissed: 3 • More complete follow up not complete • Have initiated licensure status and board certification F/U but too little data to report

34. Outcomes

35. Conclusions? • RIT more common than suspected based on lit. • Increased prevalence compared to other studies • ? Over-diagnosis • For “mild cases” • ? Truly higher prevalence • Due to population? • Due to program characteristics? • All RIT who had concerns in more than 8 areas had ongoing concerns at graduation or did not complete program • RIT who struggled in all 3 performance areas also either did not complete or graduated with ongoing concerns • Professionalism issues very common

36. Problems Encountered • Information in files varies between programs • Missing data • Esp. test scores (fortunately, may be able to acquire) • Even within programs, varying data • Classification of single most important area for major problem residents • We collected data for an additional group of residents: those with ITE performance less than 10th %ile, even if no other problems, will follow those as well (n=4 right now)

37. To come! • Full data reporting (approx 270) • Including USMLE and ITE data • Predictive model? • Follow up data (post-graduation)

38. Special Thanks to: • Nicole Roberts, PhD data spelunker extraordinaire • Reed Williams, PhD • Steve Verhulst, PhD • Jerry Kruse, MD, MSPH • SIU Family Medicine Residency Program Directors: • Janet Albers, MD • John Bradley, MD • Tom Miller, MD • Penny Tippy, MD • Wiley Jenkins, PhD

39. Referenced Articles • 1. Reamy BV, Harman JH. Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency. Fam Med 2006;38(4):252-7. • 2. Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA 2000;284:1099-1104. • 3. Williams RG et al. The nature of general surgery resident performance problems (publication pending)