Value-Added as a Measure of Teacher Impact on Student Achievement

1. Value-Added as a Measure of Teacher Impact on Student Achievement 2012 Center of Excellence Research Consortium Francis Marion University March 22, 2012 Andy Baxter & Jason Schoeneberger Charlotte-Mecklenburg Schools Center of Excellence Research Consortium

2. Section 1 A Beginning Illustration Center of Excellence Research Consortium

3. What We Mean by Value-Added The next slides were created by a 7th Grade Math teacher in Wisconsin in an attempt to explain the rationale for the type of growth measure being adopted by CMS. The slides can be found at: http://varc.wceruw.org/tutorials/oak/index.htm Center of Excellence Research Consortium 3

4. Let’s evaluate the performance of two gardeners. We keep a yearly record to keep track of the height of the trees for evaluation. For the past year, they have been tending to their oak trees trying to maximize the height of the trees. 4 Center of Excellence Research Consortium

5. To measure the performance of the gardeners, we will measure the height of the trees today (1 year after they began tending to the trees). Using this method, Gardener B is the superior gardener. This method is analogous to a Proficiency Model. 5 Center of Excellence Research Consortium

6. 6 Using our records, we can compare the height of the trees one year ago to the height today. By finding the difference between these heights, we can find how much the trees grew during the year of gardener’s care. We can see that Oak B had superior growth this year. This is analogous to a Simple Growth Model . Center of Excellence Research Consortium

7. Oak A is in a region that experiences a high level of rainfall. Oak B is in a region with very low rainfall. 7 Oak A is in a region with poor soil richness. Oak B is in a region with very rich soil. Oak A is in a region infested with insect pests. Oak B is in a region with very few insect pests. Center of Excellence Research Consortium

8. 8 To calculate our new adjusted growth, we need to start with simple growth. Now we will adjust for environmental conditions to give an “apples to apples” comparison of the two oak trees. This is analogous to an Adjusted Growth Model – another name for Value-Added. +14 Simple Growth +20 Simple Growth + 5 for Rainfall - 3 for Rainfall - 2 for Soil + 3 for Soil - 5 for Pests + 8 for Pests _________ +18 inches Adjusted Growth _________ +22 inches Adjusted Growth Center of Excellence Research Consortium

9. Section 2 Not all gardeners are the same. Are all teachers? Center of Excellence Research Consortium

10. The Widget Effect Center of Excellence Research Consortium

11. But teachers differ in impact on students. Center of Excellence Research Consortium

12. Center of Excellence Research Consortium

13. Center of Excellence Research Consortium

14. Center of Excellence Research Consortium

15. This difference matters for students later in life. For one year, difference for a student between having a 50th and 84th percentile teacher. Raises probability of attending college at age 20 by 0.5 percentage points.. Difference in lifetime earnings is equivalent to giving each student $4,600. Decreased probability of teenage child-bearing http://obs.rc.fas.harvard.edu/chetty/value_added.html Center of Excellence Research Consortium

16. Section 3 Measurement of Value-Added Center of Excellence Research Consortium

17. Key Question Value-Added Attempts to Answer 2 Teachers 1 Classroom What’s the Difference for the Students? Center of Excellence Research Consortium

18. How Value-Added is Calculated (1 of 3) Step 1: Calculate a student’s expected score.

19. How Value-Added is Calculated (2 of 3) Step 2: Measure difference in actual score and expected score.

20. How Value-Added is Calculated (3 of 3) Step 3: Determine how much of the difference is due to the teacher.

21. Goal of Value-Added Models • Identify the portion of the residual, or error that is due to the teacher • To accomplish this, we have to reduce the influence of, and account for differences associated with other factors: student prior achievement, school climate, etc.

22. Basic Requirements • Longitudinal data, with test scores from multiple grades for each student • Tests with good psychometric properties measuring attribute of interest • Databases linking students, teachers and schools, with student, neighborhood, family, teacher, etc. characteristics

23. Accounting for Differences • Differences exist between students, classrooms and schools • Inclusion of covariates (school-level % FRL) or fixed effects? • How do we intend to use the results of the model?

24. Example of Adjusting for Differences Center of Excellence Research Consortium

25. Model Types • Covariate Adjustment • Gain-Score • Layered (EVAAS)

26. Key Data Issues • Teacher-Student Matches • Teacher mobility • Student mobility • Dynamic models of team teaching • Missing Data • Classification based on patterns (MCAR, MAR, MNAR) • Estimate accuracy depends on model used Center of Excellence Research Consortium

27. Key Data Issues • Omitted Variables • We can’t measure everything • Confounding occurs when other influences are incorrectly modeled, or not modeled at all • Extent of confounding dependent upon the clustering of students/teachers with different characteristics Center of Excellence Research Consortium

28. Key Data Issues • Achievement Tests • Scaling and construction of tests • Timing of tests • Measurement error present and heteroskedastic Center of Excellence Research Consortium

29. Section 4 Value-Added Results Center of Excellence Research Consortium

30. Teacher Effect Distributions Center of Excellence Research Consortium

31. Teacher Effect Distributions Center of Excellence Research Consortium

32. Less Variation in Some Subjects than Others Center of Excellence Research Consortium

33. Teacher Effect Distributions Center of Excellence Research Consortium

34. Teacher Effect Model Similarity Center of Excellence Research Consortium

35. Section 5 Threats to Validity and Use Center of Excellence Research Consortium

36. 1. How stable are the estimates? Center of Excellence Research Consortium

37. 2. Does the test instrument bias the results? Center of Excellence Research Consortium

38. 3. Does sorting bias the effects? • High confidence • Students to teachers on observables • Moderate confidence • Students to teachers on unobservables • No confidence • If truly highly effective teachers are going to affluent schools then we are docking them . Center of Excellence Research Consortium

39. Remember the Status Quo • In what ways, and to what extent, might principal observations be biased? • What is the width of the confidence interval you’d place around a summative evaluation? • How reliable are principal observations? (i.e., inter-rater reliability?) • How connected are observation results to student achievement results? Center of Excellence Research Consortium

40. Section 6 Potential Policy Uses Center of Excellence Research Consortium

41. It’s Just one Measure Center of Excellence Research Consortium

42. Recruiting Value-Added of Teachers by Undergraduate Institution First 5 Years of Teaching Notes: 4th-8th grade math and reading teachers with five or fewer years of experience between 1998-99 to 2008-09 Center of Excellence Research Consortium

43. Equity Center of Excellence Research Consortium

44. Development Center of Excellence Research Consortium

45. Feedback How am I doing over time? How am I doing with different types of students? Center of Excellence Research Consortium

46. Evaluation Center of Excellence Research Consortium

47. Retention Center of Excellence Research Consortium

48. Compensation Actual CMS Data Random Data Percentile of Teacher Value-added Score Center of Excellence Research Consortium

49. They are measuring my performance. Center of Excellence Research Consortium

50. I don’t understand how they get this number. Center of Excellence Research Consortium