Assessment of Clinical Competencies

1. Assessment of Clinical Competencies Zubair Amin MD MHPE Associate Professor; Dept of Pediatrics Yong Loo Lin School of Medicine Senior Consultant; Dept of Neonatology National University Hospital

2. Overview of clinical assessment • Basic psychometric principles on clinical assessment • Bias • Sampling • Context specificity • Basic educational principles about clinical assessment • Educational impact • Feedback • Quality assurance • General properties of innovative instruments

3. “ Assessment Drives Student Learning.” George E Miller 1919-1998

4. “Assessment drives learning in at least four ways: its content, its format, its timing and any subsequent feedback given to the examinee.” vander Vleuten, C. (1996) The Assessment of Professional Competence: Developments, Research and Practical Implications, Advances in Health Sciences Education, 1, pp. 41–67.

5. Professional authenticity Knowledge and Performance Does Shows how Knows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

6. Examiner Question

7. Examiner Question

8. Examiner Question

9. Examiner 1 Examiner 2 One student One case: a patient with diabetes mellitus Scenario A 30 minutes for long case Decision: Pass Total Examiners’ Time: 60 minutes (30X2)

10. Examiner A Examiner B Examiner C Examiner D Diabetes Mellitus Rheumatoid Arthritis Asthma Stroke Scenario B 15 minutes for each long case Decision: Pass Total Examiners’ Time: 60 minutes (15X4)

11. You are the Chairperson of the examination committee. The particular candidate is known to be weak. In which situation (Scenario A or Scenario B) you are more comfortable in examiners’ decision to pass the candidate? What are the comparative advantages and disadvantages between Scenario A and Scenario B?

12. Context Specificity • There is no generic problem solving skills Geoff Norman Management of Diabetes Ketoacidosis = Outpatient management of diabetes Management of COPD = Management of rheumatoid arthritis Diagnosis of asthma = Counseling of asthma patient Typical correlation of “skills” across problems is 0.1 – 0.3

13. “The ‘law’ of educational cause and effect states that: for every evaluative action, there is an equal (or greater) (and sometimes opposite) educational reaction.” Schuwirth, L.W.T. (2001) General Concerns About Assessment. Web address: www.fdg.unimaas.nl/educ/lambert/ubc “Assessment drives learning in the direction you wish.”

14. Unintended Consequences of Assessment Evolution of Medical Students Website by NUS students:http://medicus.tk

15. Educational Impact and Feedback • Any test is anxiety provoking for the students and (staff) • Test has potential positive and negative steering effects on learning and professional development • We tend to assess more, but provide considerably less feedback • Provision for feedback should be strongly considered in performance-based assessment

16. Linking Learning and Assessment

17. Concept of Mastery ‘All or none state’ – not really

18. Continuum of Clinical Competencies Student Trainee Doctor in Practice

19. A examination that attempts to test students’ competency at a given point of time is less preferable than one that tests the competency over a span of time

20. Some Common Features in Assessment of Clinical Competency • Multiple examiners • Multiple cases (patients, problems) • Context free • Over a span of time • Feedback and improvement of learning

21. Traditional Clinical Examination • One/two examiners evaluate a candidate • Candidate takes one patient • Tasks • Take history • Complete physical examination • Interpret data • Generate differential diagnosis • Discuss the patient problem with examiners

23. Issues • Single patient: context specificity • Single examiner: bias • Patient: mostly in-patients • “Exam” patients • Artificial • Unlike most physician-patient encounters • Limited or no opportunity for observation of communication • Limited or no opportunity for feedback

24. Instruments for Clinical Competency Assessment • OSCE • Mini-CEX • DOPS • 360 degree evaluation • Portfolio • Many others

25. Mini-CEX • Assessor observes a trainee in any setting • Takes into account of contexts • Complexity • Focus of visit (diagnosis, management, follow-up) • Uses standard form (not customized to particular patient/problem) • Completes 6-10 encounters

26. Implementation Issues • Generally reliable and reproducible • At least 4-6 encounters • Different assessors, different patient, different context • Feedback

27. The mini-clinical evaluation exercise ( mini -CEX) form Norcini, J. J. et. al. Ann Intern Med 2003;138:476-481

28. Direct Observation of Procedural Skills • Mini-CEX equivalent for procedural skills assessment • Similar psychometric properties • Observation of procedures with real patients • Minimum of 4-6 procedures, different assessors • Provision of feedback

29. 3600 Assessment / Multi-Source Feedback • Involves evaluation of the candidate by multiple individuals within the sphere of influence • Self, superior, peers, co-workers (nurse, ward clerk) • Assessment of professionalism

30. 3600 Assessment • Multiple observations by multiple observers • Over a period of time • Provide evidence, as opposed to impression • Highly valued as a developmental tool

31. 3600 Assessment • Focus on behavior: action taken by the individual • Not used for knowledge or skills • Nurses often are better discriminators • Difficult to standardized grades Unsatisfactory Satisfactory Superior • Customize to your needs

32. Fundamental Principles • Assessment should be driven by purpose • Consider both psychometric and non-psychometric parameters of assessment tool • Create a backbone of assessment with few high quality, practical tools • Strive towards holistic profiling of a candidate