A Guide to Education Research in the Era of NCLB

1. A Guide to Education Research in the Era of NCLB Brian Jacob University of Michigan December 5, 2007

2. How has the environment for education research changed? • NCLB: evidenced-based programs • Accountability • Tight state and local budgets • Heightened oversight by foundations

3. Outline • What are the different types of education research, and what are the goals of each? • What distinguishes good evaluation research from bad research? • What are some of the common challenges to collaboration between education researchers and practitioners?

4. Types of Education Research • Basic/Descriptive • What teacher characteristics do principals value most highly? • How does the gender gap in student achievement change over the course of schooling? • Program Evaluation • Formative (Process Analysis) • Which aspects of a curricular intervention work well and which aspects need to be refined? • Summative (Impact Analysis) • Does a new curriculum have a positive impact on student achievement?

5. How to Tell a Good Impact Analysis when You See It • Good intervention • Well-defined question • Coherent and explicit theory of action • Good research design • Causal inference • Good implementation • Attrition, contamination • Good analysis/interpretation • Magnitude • Generalizability

6. The Problem of Causal Inference • What is the casual impact of a particular program or policy on the outcome of interest? • Do teacher induction/mentoring programs reduce teacher attrition? • Does computer-aided curriculum improve student achievement? • We need to know what would have happened in the absence of the program (i.e., the “counterfactual”) • We often start with a correlation • Students who attend magnet schools score higher than students in traditional schools. • But this correlation may not reflect a “causal” impact • Many potential “confounding” factors • Students who attend magnet schools are more motivated and have more supportive families than those who attend traditional schools.

7. Threats to Causal Inference • Selection • Students/teachers/schools who participate in a program are systematically different than those who do not participate. • Example: families that choose to send their children to charter schools; teachers that engage in professional development. • Concurrent events • Concurrent events may have been responsible for the effects that are attributed to the program. • Example: High school reforms in Chicago implemented at the same time as accountability provisions. • Maturation • Naturally occurring changes over time may be confused with a treatment effect • Example: Schools that were improving because of new professional development adopt a tutoring program.

8. Common Research Designs • Example: Success for All (SFA) is implemented in 12 out of 32 elementary schools in a district in 2001-02. • Matched Comparison Group • Compare test scores in SFA schools with “similar” non-SFA schools in years after 2001-02. • Pre/Post Design • Compare test scores in SFA schools after 2002 with test scores in the same schools before 2002. • Pre/Post with Matched Comparison Group • Compare test score changes from, say, 1996 to 2005 in SFA schools with changes over the same time period in the “similar” non-SFA schools.

9. Randomized Control Trials (RCTs): The Gold Standard • Randomly assign some students/teachers/schools to receive the treatment, and others to not receive the treatment. • Randomization assures that the treatment and control groups are equivalent in all ways - even in ways that one cannot observe, such as “motivation,” life circumstance, etc. • Avoids concurrent event/maturation concerns since both treatment and control group should experience these effects • Decision about the level of random assignment depends on the nature of the treatment, and the possibility of “contamination” of the control group. • One-on-one tutoring: student level random assignment • Curriculum or pedagogy: classroom or school random assignment • Some policies/programs cannot be evaluated via RCTs. • Example: the competition effects of school choice

10. Some Concerns with RCTs • Ethical Concerns: Should we deny treatment to some students/schools? • Assumes that we know that the program is effective • If there are limited resources, then randomization is arguably the “fairest” method for allocating the treatment. • Many research designs can ensure equitable distribution of the program while, at the same time, maintaining random assignment. • Group 1: 3rd grade classes get the curriculum in year 1, and then 5th grade classes get the curriculum in year 3 • Group 2: 5th grade classes get the curriculum in year 1, and then 3rd grade classes get the curriculum in year 3. • Logistical Concerns: How disruptive would it be for schools/districts to conduct random assignment? • Depends on the context, but often not very disruptive • Professional development program with existing student tests • Requires evaluations to be planned at the same time that the program is implemented, and not merely attempted after the fact.

11. Good Research Worries about the Problem of Attrition • Attrition occurs when participants (e.g., students, teachers) leave the school/district prior to the conclusion of the study. • Difficult to collect outcome data for these individuals. • Differential attrition is when members of the treatment group leave the study at a different rate than members of the control group. • If those who “attrit” are different in important ways, this can bias the results of the study - even in RCTs. • Example: “lottery” studies of magnet/charter schools • Many students who lose the lottery leave the public school system. If the most “motivated” lottery losers leave, one would overstate the benefits of attending a magnet/charter school.

12. Good research also … • Pays attention to the magnitude of the effects, and not just the statistical significance. • Looks at how effects change over time. • Addresses the generalizability of the results (external validity) • Explores variation across students, teacher and/or school subgroups • Discusses the limitations of the study

13. Common Challenges for any Research Collaboration • Planning for evaluation in advance rather than conducting it after the fact • Concern about denying treatment • Obtaining consent from principals, teachers and parents • Financial incentives; give information back to schools • Respecting the organization of schools • Be aware of school schedules and the academic calendar • Respecting the time constraints of school and district staff • Use existing tests when possible; limit survey length

14. Conclusions • Good research is becoming increasingly important in education • And can be very useful for practitioners (case studies) • Good research requires advance planning and collaboration between researchers and practitioners • Collaboration is possible, as shown in the following case studies