1 / 50

Competency Based Assessment

Competency Based Assessment. Baraa Alghalyini SAPIC Dec 1 st 2012. Today. Define Competency Based Assessment Discuss CBA goals, tools, limitations and, unintended effects. What is Competence in Medicine?. Epstien RM, JAMA 2002

michi
Download Presentation

Competency Based Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Competency Based Assessment Baraa Alghalyini SAPIC Dec 1st 2012

  2. Today.. • Define Competency Based Assessment • Discuss CBA • goals, • tools, • limitations and, • unintended effects

  3. What is Competence in Medicine? • Epstien RM, JAMA 2002 “The habitual and Judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served”

  4. What is Competence in Medicine? • Frank et al 2010 • Competency Based Education is • “Approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It deemphasizes time-based training and promises greater accountability, flexibility, and learnercentredness”

  5. Goals of Assessment • 1) Optimize Learners capabilities • 2) Protect the Public • 3) Determine bases of applicants’ selection for advanced training

  6. Summative Assessment • Provide overall impression of learners’ • performance • Competence • Qualification for advancement to higher education • Assessment of learning

  7. Formative Assessment • Reinforce internal motivation • Guide future learning • Promote reflection • Assessment for learning

  8. Good Assessment Criteria • Validity or Coherence • does the assessment measure the real world? • Reproducibility or Consistency • assessment results the same if repeated under similar circumstances

  9. Good Assessment Criteria • Equivalence • Different repetitions of same assessment yield same results • Educational effect • motivates those who takes it to prepare for it in a way that promotes learning

  10. Good Assessment Criteria • Catalytic effect • provides feedback that enhances education • Acceptability • assessors and learners found it credible • Feasibility • practical and realistic

  11. Assessment Tools

  12. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

  13. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

  14. Commonly used assessment tools Bandiera G, Sherbino J, Frank J. The CanMEDS Assessment Tools Handbook: An introductory guide to assessment methods for the CanMEDS Competencies. Ottawa (ON): Royal College of Physicians and Surgeons of Canada; 2006.

  15. Assessment Tools • Multisource Feedback (360° Evaluation) • Supervisor report of direct observation • Peer assessment: Residents, Medical students • Allied Health Professionals • Patients, their family members • Self assessment • Summative and Formative

  16. Assessment Tools • Multisource Feedback: • Advantages: • Encourage self reflection • Evaluate certain behaviors: professionalism, communication skills • Disadvantages: • Time consuming • Survey to be meticulously designed to ensure validity

  17. Assessment Tools • ITER/FITER In Training Evaluation Report/Final In Training Evaluation Report • Report completed based on several observation opportunities in different clinical settings

  18. Assessment Tools • ITER/FITER • Advantages • Immediate feedback is possible • High face validity if structured and standardized • Disadvantages: • Different observers might observe different things • Documentation not timely or not done

  19. Assessment Tools • Encounter Cards: • Daily Evaluation Cards, • Daily Encounter Cards, • Daily Operative Cards, • Daily Teaching Evaluation Cards • Capture clinical competence from brief encounters • Increasingly popular in ICU, Emergency Medicine, and Surgical rotations

  20. Assessment Tools • Encounter Cards: • Advantages: • Frequent, timely, formative and grounded feedback • Method of tracking procedures performed/ clinical exposure • May contribute to ITER/FITER • Disadvantages: • Frequent feedback meetings • Assessment fatigue • Faculty development is required to train assessors use cards effectively

  21. Assessment Tools • Portfolio • Collection of assessment tools: log book, performance diaries, multisource feedback, others?

  22. Assessment Tools • Portfolio • Advantages: • Learner centered • Longitudinal • Authentic • Flexible • Disadvantages • Require on going monitoring by reviewers • Learners’ poor compliance • Difficult to be standardized

  23. Assessment Tools • Simulation: • Artificial recreation of clinical environment or circumstances

  24. Assessment Tools • Simulation: • Advantages: • High risk procedures in safe environment • Measure performance in emergency situations • Standardized • Disadvantages: • Cost • Resources

  25. CanMEDS Assessment Matrix

  26. CanMEDS implementation 2001-2009 Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

  27. CanMEDS Roles Implementation 2001 VS 2009 A little Somewhat Mostly Completely Not at all Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee 30

  28. Which of the following methods do you use to teach the CanMEDS competencies of your residents? All CanMEDS Roles Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

  29. Which of the following methods do you use to assess the CanMEDS competencies of your residents? All CanMEDS Roles Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJason Frank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

  30. Barriers in ensuring teaching and assessment of CanMEDS (N = 374) % of Program Directors indicating barrier Faculty Adoption of a Nationwide Competency Framework 2001-2009: The CANMEDS Roles in Residency EducationJasonFrank, C. Abbott, G. Bourgeois, S. Hyde, A.C. Lee

  31. Assessment Pitfalls, Challenges&Unintended Outcomes

  32. Pitfalls of Assessment • (Ginsbur, 2010) • More emphasize on objectivity of assessment method • Discount subjective measures thinking they are unreliable • Misconception that eliminating human inference from judgment process minimize error

  33. Pitfalls of Assessment • ( Kogan, 2010) • Faculty’s own clinical skills may affect their evaluation of trainees

  34. Challenges with Current Assessment Methods What are your challenges?

  35. Challenges with Current Assessment Methods Assessors: • (Laeeq et al, 2010) • Poorly motivated • Lack of resources :time, financial, personnel • Inadequate training • Solutions.. • Embrace CBE curriculum and concepts • Internalize it, teach it, assess it • Self reflection!

  36. Challenges with Current Assessment Methods Trainees • Not involved in the process of curriculum development and its assessment tools • Not being well oriented to the assessment tools or entrustable professional activity(EPA) • Not providing feedback to program directors and faculties on the assessment process

  37. Challenges with Current Assessment Methods Curriculum • Doesn’t meet specific “competencies” standards • Evaluated with different assessment methods

  38. Challenges with Current Assessment Methods Tools • Newly introduced without proper training to faculty and trainees • Emphasis on what learners have done instead to what they demonstrated • Impractical • Invalid

  39. Unintended Effects of Assessment • What are the negative effects of assessment? • Learners focus on the test, not on improvement • De-emphasizes things that are not ‘testable’

  40. Unintended Effects of Assessment • Episodic anxiety and stress • May not assess what it is meant to assess • Unwanted bias to affect assessment of progress (afraid to give negative feedback) • May take time away from effective learning

  41. What is Entrustable Professional Activity ( EPAs)? • What the public trust clinicians are capable doing • Represent what trainees are expected to master doing based on their specialty and subspecialty

  42. What is Entrustable Professional Activity ( EPAs)? • Cate, Scheele 2007 • “ a practitioner has demonstrated the necessary knowledge, skills, and attitudes to be trusted to independently perform this activity”

  43. Conditions of Entrustable Professional ActivitiesConditions of Entrustable Professional Activities (Cate, Scheele 2007) • 1. Part of essential professional work in a given context. • 2. Must require adequate knowledge, skill, and attitude. • 3. Must lead to recognized output of professional labor. • 4. Should be confined to qualified personnel. • 5. Should be independently executable. • 6. Should be executable within a time frame. • 7. Should be observable and measurable in its process and outcome (well done or not well done). • 8. Should reflect one or more competencies

  44. Could you suggest an EPA for your Learners?

  45. Home take message • Every key competency MUST be ASSESSED somewhere in the program • Not every competency needs to be assessed in every activity/rotation • Choose the best methods for the competency • Be careful to match expectations to level

  46. Home-take message • Try to combine competencies into assessment strategies (such as an OSCE station that reviews multiple competencies at once) • Choice of assessment tool should depend on educational context rather than a blind desire to use objective or standardized assessment tools • EPAs, EPAs, and EPAs!!

  47. Questions?

More Related