WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS?

1. WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS? David Myers Senior Vice President IES 2006 Research Conference

2. Presentation Road Map • Characteristics • Benefits relative to small-scale RCTs • Implementation and design challenges

3. Characteristics • Random assignment of schools/classrooms: • Multiple school districts (Reading Comprehension has about 10 districts; P4K has 27) • 10 to 20 schools per “intervention arm” • 200+ classrooms per “intervention arm”

4. Characteristics (continued) • Random assignment of students to treatment or control status with many schools/sites: • Upward Bound evaluation included a probability sample of 67 grantees and more than 3,000 students. • Charter school evaluation includes about 40 schools and 4,000 students.

5. Characteristics (continued) • Multiple years of data collection: • Baseline • One or more follow-up assessments • Multiple interventions: • Reading Comp. and P4K: four each • Math Curriculum: three to four

6. Benefits • Impacts of school interventions, curricula, instruction, and professional development implemented in typical schools: • Often smaller impacts in large-scale studies (P4K) than in efficacy trials • Impacts from a wider range of contexts than generally found in efficacy trials: • Students • Teachers • Schools • Greater opportunity for subgroup analyses

7. What Questions Can an RCT Address? • Intention to treat – yes • Treatment on the treated – yes, with assumptions • Ingredients and dosage – no design answers particularly well

8. Intention to Treat • Impact of the opportunity to participate for all schools/teachers/students • Simply, the difference in the means for the treated group and the control group • Control group represents what we expect for treated group in absence of intervention • Most policy relevant • Subgroup analysis shows who benefits most

9. Treatment on the Treated • More assumptions • Impact of intervention on those who actually receive treatment • Adjusts for those who did not participate and control who did • Less policy relevant because it does not show the impact of an implemented policy – TOT separates the impact on participation from the effect of the treatment • Subgroup analysis shows who benefits most

10. Ingredients/Dosage • Many assumptions, some unrealistic • Like other designs, cannot address these questions with the same level of rigor • Random assignment to different “intervention arms” of ingredients/doses needed for defensible impact estimates

11. Study Implementation Challenges • Objections involve interventions, not design, so negotiation teams must have substantive expertise • More innovation in schools in recent years – less inclined to adopt yet another approach • Districts want consistent instruction across schools • Alignment of curriculum with state testing • NCLB waivers – will schools and districts be held accountable (AYP) if the intervention fails?

12. Study Implementation Challenges (cont.) • How much “intervention from the evaluator” is OK when it comes to fidelity? • Let policy and practice dictate decisions – what would developer or school do in the absence of the evaluation? • Maintain normal school, program, and intervention operations – make the design accommodate the program • When oversubscription is needed, programs and schools often overestimate student interest.

13. Generic Design Challenges • External validity – random effects design requires larger samples for same precision as fixed effects – demos vs. “national evaluations” • Spillover or contamination from random assignment of schools to interventions • Effect of giving some students treatment on the outcomes for the control group (P4K) • Identifying the counterfactual

14. Design Challenges: Case Study • Charter schools evaluation: • Research question concerns the impact of attending a charter school • How would a child attending a charter school have performed in the absence of that school? • Design is random assignment to charter/regular public schools (lottery) • All designs are limited here: • A nonrandom sample of students left the public school, which may change what is taught there • Different than the thought experiment of asking what would happen with no charter schools

15. Another Case Study: P4K • What is the impact of being assigned to a pullout program? • How would students perform in the absence of the pullout program? • Pullout program: • RA of schools to one of four interventions • Within schools, RA of eligible students to the pullout program or the regular classroom • Change in the composition of the regular classroom and perhaps, change in curriculum and instruction when some struggling readers are removed