Production Functions and Measuring the Effect of Teachers on Student Achievement With Value-Added

1. Production Functions and Measuring the Effect of Teachers on Student Achievement With Value-Added HSE March 18, 2011

2. Education Production Functions • The school production function is typically specified in the following way: O = a + bP + cT + dS + µ where O = school output (usually pupil test score gain) P = pupil characteristics T = teacher characteristics, S = school characteristics µ = error term

3. What Are the Standard Inputs of Schooling? • Student characteristics are usually student family background, books in the home, ethnicity, whether the pupil works outside the home. • Teacher characteristics have traditionally included formal education, experience in teaching, and in-service training. • School characteristics include physical conditions, availability of textbooks, class size, school size, whether private or public, urban or rural, violence in the school.

4. Issue 1: Are All the Relevant Classroom and School Inputs Specified? • The unspecified variables in this type of model are usually: time on task or opportunity to learn (teacher effort, student attendance), pedagogical style (teacher teaching capacity), teacher content and pedagogical content knowledge (teacher teaching capacity), and teacher expectations. • The kind of curriculum used is also usually not observed. • Teacher and student attendance may be important. This is another type of opportunity to learn. • Unless these unspecified variables are highly correlated with the usual teacher and school specified variables (such as teacher education and experience or class size), we may not be picking up the school effects on output.

5. Issue 2: Are School and Student Variables Highly Correlated? • An important argument is that school variables are highly correlated with pupil social characteristics (higher SES students go to schools with more resources, including peers of higher SES), so that it is not easy to separate school effects from SES effects on academic output. • The counter-argument is that much, if not most, of the variation of academic output is within, not between schools.

6. Issue 3: Are Inputs Really Exogenous to Output? • One question we have to ask when we estimate the relationship between school inputs and outputs is whether the school input (treatment variable) of interest (say, teacher “quality”) is “assigned” independently of the output variable. • For example, if we find that teachers with greater content knowledge are associated with classrooms and schools in which students score higher, or even have higher test score gains, are we sure that the higher content knowledge teachers did not “choose” to be in classrooms and schools where pupil gain was likely to be higher? Or that pupils with higher gain trajectories did not choose to be with more knowledgeable teachers?

7. Worrying About Causality and Effect Size • Economists are at the forefront of trying to reduce the bias in estimating the relationship between classroom/school inputs and outputs controlling for student characteristics. • The best technique to control for selection bias is random assignment of students and teachers to treatment and control groups. • There are other techniques as well. These attempt to correct for selection bias. • Economists also focus on effect size, but since they can only study variables they can measure, they often estimate “unbiased” effect sizes but miss out on variables that may have larger effect sizes.

8. Do Teachers Make a Difference? Evidence from the U.S. • A number of studies are able to link students and their achievement scores to particular teachers across grades and over time (Hanushek, Kain, O’Brian and Rivkin, 2005; Rowen, Correnti, and Miller, 2002; Nye, Konstantopoulos, and Hedges, 2004). They show that teacher variation explains a significant part of the variation in student outcomes. • Some studies show greater teacher effects within school and some across schools.

9. The Value Added Model • The standard form for estimating class size and teacher effects is a model that estimates student achievement gains as a function of class size or some measure of the quality of the student’s teacher, student characteristics and school characteristics. • The “best” models observe multiple cohorts of students across several grades. • One way to estimate value added is to assume “decay” in knowledge from time t-1 to time t: A(it) = A(it-1) + T(it) + . Another avoids possible serial correlation of A(it-1) with the error term by assuming that =1: Ait) -A(it-1) = T(it) + . But this might bias teacher or other “treatment” effects downward if prior achievement is correlated with teacher effects, or, more generally, correlated with the treatment.

10. Issues that Emerge in Estimating Classroom Treatment Effects • The main problem in estimating classroom treatment effects from value-added models is that students are not randomly assigned to the treatment--for example, better teachers might attract the “better” students (those who get the higher value added no matter which teacher they get), or higher value-added student may crowd into the “better” classrooms, creating larger class sizes. • One way to get around this is to control for student fixed effects--time invariant characteristics of students (observable and unobservable) that are related to their achievement gains. This takes care of the non-random sorting problem if students are assigned to teachers on the basis of their average gains.

11. What Do the VA Estimates Show About Teacher Attributes? • Some studies suggest that teacher credentials are not important in explaining why some teachers are more effective than others (Hanushek et. al’s Texas studies, for example). • But other US studies do show significant relations between value added and teacher credentials (Hill et al, 2005; Clotfelter, Ladd, and Vigdor, 2007; Boyd et al, 2008). • A study of rural schools in Guatemala also shows significant relation between teacher knowledge and student achievement gains.

12. Results of Some US Teacher Credential Studies • Most value added gain models that estimate the effect of teacher credentials use test scores that are normalized by year, grade, and subject. • So they report results of effects in fractions of a standard deviation of student test score variation. • To illustrate these issues, here are the results of a value added study by Boyd et al, for New York City with student, grade, and time fixed effects.

13. Boyd, Grossman, Lankford, Loeb, and Wyckoff Study

14. Clotfelter, Ladd, and Vigdor Studies • CLV used a large data set in North Carolina to estimate the effects of teacher credentials on student achievement in elementary and high school. • They employed various models, including those controlling for student fixed effects, which take care of the problem of “static” tracking, and others with school fixed effects, which estimate teacher differences across classrooms within schools.

15. Clotfelter, Ladd, and Vigdor Studies

16. Is It Teacher Skills or Other School Factors Related to Teacher Skills? • Teacher quality, as measured by various indicators such as experience, subject content knowledge, pedagogical content knowledge, classroom teaching skills, and teacher commitment to student learning, are generally considered to have some impact on how much students learn. • Issue 1: Is teacher quality as measured by these skills significantly related to student learning gains? • Issue 2: Are there other factors related to student learning gains that are more important than individual teacher skills, such as class size, the quality of the curriculum, opportunity to learn (amount of time devoted to the subject during the year and the pace of coverage of the curriculum), and school climate (violence, absenteeism; lack of parent involvement; student and teacher turnover) or the aggregate of teacher skills in a school or principal leadership? • Issue 3: Are these other factors related to teacher skills? For example, is OTL related to teacher skills? Is the quality of the curriculum related to teacher skills?

17. Studies Outside the U.S. • The advantage of the U.S. studies is the vast amount of data on students across grades, classrooms, and schools. • However, some studies outside the U.S., using smaller data sets, are able to get better data on teachers in terms of teacher subject matter teaching knowledge, and teaching skills, and some are able to control for opportunity to learn in terms of curriculum completion or observed number of lessons taught and lessons taught on tested topics. • For example, Marshall and Sorto in Guatemala studied rural schools, using one cohort of elementary school students. The study focuses on measuring teacher knowledge and opportunity to learn. Reeves (2009) estimates teaching skill and OTL effects in a sample of Cape Town schools. Carrasco is estimating the effects of teaching skills in Chilean secondary schools using school fixed effects. He finds a small but significant effect of teacher “quality,” as measured by an index of teaching skills.

18. What Should We Measure Better? • One of the key issues is what teacher skills we should measure in trying to contribute to school improvement. • An important factor may be teaching skills themselves, which in turn may be related to subject knowledge and to subject pedagogical knowledge. These are important factors to identify, because they are factors that can be taught to teachers. • Videotape analysis of teacher teaching skills, teacher tests of CK and PCK, and measures of how much curriculum teachers cover in class should contribute to better value added studies.

19. South Africa: Cross-Section and Value Added Estimates of Teacher and Other Classroom Variables on Learner Mathematics Achievement