1 / 37

An implementer’s Perspective: An evaluation and compensation pilot

An implementer’s Perspective: An evaluation and compensation pilot. October 17, 2012 Richard Bowman, Ph.D. Context Proposal Results Buy-in Logistics Metrics Issues. Context Proposal Results Buy-in Logistics Metrics Issues. Context. Reform efforts

finian
Download Presentation

An implementer’s Perspective: An evaluation and compensation pilot

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. An implementer’s Perspective: An evaluation and compensation pilot October 17, 2012 Richard Bowman, Ph.D.

  2. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  3. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  4. Context • Reform efforts • Teacher Evaluation Task Force (TETF) • Race To The Top (RTTT) • Planned legislation • Prior legislation • Shared urgency • Four schools • SIG application • “Funds have been set aside to support performance pay.” • $1,000 per teacher assignedto the school. (~ $300K) • Strategic Data Project Data Fellows • Highly trained and capable personnel pressed into service

  5. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  6. The Proposal • Performance compensation • Teacher evaluation • Pilot program • Multiple measures • Student Surveys • Observations • School Value Added • Individual Value Added • PLC Student Learning Goals (SLOs) • Individual Student Learning Goals (SLOs) • In-depth research

  7. The Proposal at 50,000 feet

  8. The Proposal at 10,000 feet

  9. Student Surveyson the ground I wonder who translated this?

  10. Student Surveyson the ground

  11. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  12. Proposal Results

  13. Most Valued: Student Surveys

  14. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  15. Union Leader Buy-in (Negotiations) • Voluntary participation • Student Learning Goals (SLOs) • Hold-harmless evaluation • Bonus, not pay • Percentages • Ferguson’s research

  16. Teacher Buy-in • Emphasize shared background • Time – personal time spent • Focus on concerns • Negative consequences • Time taken • Talk context • Responsiveness • Transparency

  17. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  18. Logistics - Survey • One friendly Denver SDP fellow • Two local SDP fellows • Four schools • Eight survey versions • 16 survey days • 93 participants • ~20,000 student surveys • Blank photocopied surveys • Rough class count stuffed in envelopes • Envelopes sorted by period and teacher • ~20,000 student barcode labels • Cut by guillotine and bagged by class period

  19. Logistics - Software • Excel (Report production) • Lookup tables • VB Macros • RDBMail (Outlook Integration) • PDF generation • Google Docs/Drive (Collaboration) • Stata (Data analysis and management) • Remark Office OMR (Optical recognition)

  20. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  21. Student Surveyson the ground

  22. Metrics • Survey • Average responses assuming a 1 - 5 valued scale. • Average for each of the “Seven Cs” • Composite average of the “Seven Cs” • Reporting • Average for each of the “Seven Cs” and composite • Individual • School-wide • Pilot • Histogram of the composite • Top two relative strengths and weaknesses • No item-level detail • Evaluation • Thirds of composite score within school

  23. Metrics • Value-added Model • Includes student, teacher, and school covariates • 1-year estimates for both teachers and schools • Best Linear Unbiased Predictor (BLUP) aka. Bayesian or shrunken estimates • Relative to (normalized) district-wide student test scores • By year, standard deviations • Random effects model estimated by REML • Estimates produced four times, based on SCA and SBA • Reporting • Limited numbers • Concern about meaningfulness of numbers • Conversion to months of learning • Confidence level • Visual • Range of months of learning in text • Histogram • School estimate and teacher estimates • Evaluation • Three possible ratings • Significantly above, significantly below, and not significantly different than average • 95% significance

  24. Metrics • Student Learning Goals • Teacher developed and measured and reported • Individual goals and PLC goals • “Reach” goal – goal that would be a stretch to achieve • “Expected” goal – goal expected to be achieved • Evaluation • Three possible ratings • Met Reach Goal, Met Expected Goal, Did not meet either goal

  25. Metrics • Observations • Administrator evaluates teachers based on three domains of Danielson’s FFT • Reported electronically • Ratings converted into a four-point scale and averaged • Evaluation • Teachers reported whether or not observation happened

  26. Context • Proposal • Results • Buy-in • Logistics • Metrics • Issues

  27. Observation Data

  28. Issues • Training and Professional Development • Observation • Principal time • No consequences • Length • Student Learning Goals • Ensuring rigor and relevance • Professional development • Surveys • Special Populations • Proctoring • Value-added • Understanding, fairness, and use • School-wide evaluations • Universally disliked • Joint or shared evaluations • Largely disliked

  29. Issues • How can this information inform instruction? • Evaluation Metrics • Selection bias due to volunteers • Relative classifications • With or without base year, creates problems • Absolute classification

  30. Thank you!Questions or Comments?

More Related