1 / 22

Portfolios for Lifelong Competence Assessment

Portfolios for Lifelong Competence Assessment. Assessment of Competence - More than One Approach. Traditional (primarily external scrutiny) Emerging (more internal, self-directed) Research (educationally-driven). Assessment of Competence - The Traditional Approach.

albertu
Download Presentation

Portfolios for Lifelong Competence Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Portfolios for Lifelong Competence Assessment

  2. Assessment of Competence - More than One Approach • Traditional (primarily external scrutiny) • Emerging (more internal, self-directed) • Research (educationally-driven)

  3. Assessment of Competence - The Traditional Approach • Tests standardized where possible to maximize reliability (e.g. simulations - MCQ/ virtual, SP) • Narrowly focused on knowledge and reasoning • High stakes, summative • May focus on individuals (“assessment”) or programs (“evaluation”) • All about minimal competence • Preferred approach of regulators (viz. societal protection)

  4. Assessment of Competence - The Emerging Approach • Tests tailored to the actual practice and learning needs of the individual • Assessment beyond knowledge (e.g. includes skills and behaviors) • Low stakes, formative, abundant feedback • Usually focuses on the individual • All about driving continuous improvement • Impelled by innate professionalism (internal locus of accountability), not just external scrutiny

  5. Assessment of Competence - The Research Approach • Seeking better pedagogies, best practices • Broadening base of assessment, shaping learning • Finding effective balance between traditional and emerging approaches • Focus may be on individual, but more often programmatic • Seeking balance between baseline competence and continuing improvement • Primarily the realm of professional educators • Driven by need to define effective educational approaches

  6. Availability of Data Sources andMethodologies • For the Traditional approach - Good • For Emerging & Research approaches - Poor

  7. Needed: Access to Many New DataSets - Some Examples • Work learning experience - scope of clinical practice and changes over time • Clinical performance - processes of care, outcomes, management • Behaviors and skills • Achievement of competency milestones • Longitudinal research data set

  8. Comes the Portfolio aka eFolio - Definition • An mechanism to collect electronic data sets in near real time (i.e. A Process) • A particular collection of data and data sets specific to purpose - “Assessment of competence” (i.e. A Thing) • Both a Process and a Thing (i.e. A “Proing”)

  9. The Good News: Bad News Debate • Healthcare professionals leave a massive contrail of electronic data in their daily work • BUT relevant data are very fragmented, scattered across a galaxy of repositories • HIT now allows the potential for sourcing and compilation of data in near real time • BUT the relevant repositories have scant tradition nor appetite for data-sharing

  10. eFolio ConnectorSuite of apps compiling data reportsfor the learner, and for other viewers authorized by the learner Data Commons A collection of Portfolios and Repositories that can provide data for many different applications The eFolio Connector Vision Data Sourcing

  11. eFolio ConnectorSuite of apps compiling data reportsfor the learner, and for other viewers authorized by the learner eFolio ConnectorSuite of apps compiling data reportsfor a variety of viewers authorized by the learner identified Data Commons A collection of Portfolios and Repositories that can provide data for many different applications Data Commons Bridging Many Portfolios and Repositories, and providing data for many different applications eFolio Connector: Initial Data Sets Planned Data Sourcing

  12. eFolio ConnectorSuite of apps compiling data reportsfor the learner, and for other viewers authorized by the learner eFolio ConnectorSuite of apps compiling data reportsfor a variety of viewers authorized by the learner identified Data Commons A collection of Portfolios and Repositories that can provide data for many different applications Some Other Important Issues Reporting out to authorized viewers specifications Trusted Agent Hub “VirtualRepository” Data Sourcing

  13. The “Virtual Repository”: Affiliated Repositories • Sign standard data-sharing agreements (Commons likely to become separate entity) • Generically willing to share data, but decide on case by case basis • Follow data custodian best practices • Expose data in XML, comply with technical specifications for each data set • May consume data, with authorization of the learner identified • Might charge fee to Apps for data provided

  14. More on Trusted Agent Hub • Provides a backbone of connectivity across Data Commons infrastructure • Includes array of System Tools (Apps) for • ID matching • De-identification • Authentication • Business rules • Permissions • Transaction tracking and auditing

  15. Channels of Distribution Viewers (authorization by learner or de-identification for identified data) Apps and Data Compilations Data Commons More on Report-Out to Viewers Report Out

  16. Compiling Data Sets: How Sausage Really Gets Made • Clearly define purpose (e.g. Understanding my practice) • Devise necessary schema (e.g. to codify narrative) • Do use cases to define content, context, use of data set • Find reliable sources of data required • Execute agreement with repositories to share data • Build technical specification to facilitate interoperability • Source data from repositories using web services • Pay any agreed-upon fees to repositories providing data

  17. eFolio Data Commons Examples of New Data Reports In Development Data Sourcing

  18. What If? • There is no current source for data required? • Add needed data to existing repository(ies) • Build a new repository, affiliate with Commons • Permanent access to data required, but repository(ies) unwilling or unable to maintain • Arrange with repository to provide archiving • Build central archiving function outside Commons • Learner says that data in a report is wrong? • Build central process for reconciliation of data

  19. What if? -2 • Legal discovery is attempted on one of the data sets generated? • Compiled data sets are expunged by policy immediately after report-out (cf. Mandalas) and no data values are stored in the system • A request is received for identified data from someone other than the relevant learner • No date are moved, compiled or reported out without authorization of the learner

  20. Evolution of the eFolio Connector - Feasibility Work • 2007 Invitational Conference (sponsored by AAMC, ACGME, FSMB and NBME) • 2008 Pilot Organizational Meeting; User advisory group established • 2010 Feasibility Pilot - built ANSI-conformant international technical specification for “Educational Trajectory” data set

  21. Evolution of the eFolio Connector: Pre-Implementation • 2011 • At least two more specifications (likely “Assessment of Milestones” and “My Practice”) • Marketing, pricing, and business model • Prototype featuring 3-4 reports for feedback

  22. Conclusions • The eFolio Connector can source data from a diverse galaxy of repositories affiliated with the Data Commons and provide potentially useful data reports that • bridge the continuum • provide feedback on competence and • inform lifelong learning • Reports are available in near real time to learner and any other viewers authorized

More Related