1 / 15

WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS?

WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS?. David Myers Senior Vice President IES 2006 Research Conference. Presentation Road Map. Characteristics Benefits relative to small-scale RCTs Implementation and design challenges. Characteristics . Random assignment of schools/classrooms:

cameo
Download Presentation

WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. WHY LARGE-SCALE RANDOMIZED CONTROL TRIALS? David Myers Senior Vice President IES 2006 Research Conference

  2. Presentation Road Map • Characteristics • Benefits relative to small-scale RCTs • Implementation and design challenges

  3. Characteristics • Random assignment of schools/classrooms: • Multiple school districts (Reading Comprehension has about 10 districts; P4K has 27) • 10 to 20 schools per “intervention arm” • 200+ classrooms per “intervention arm”

  4. Characteristics (continued) • Random assignment of students to treatment or control status with many schools/sites: • Upward Bound evaluation included a probability sample of 67 grantees and more than 3,000 students. • Charter school evaluation includes about 40 schools and 4,000 students.

  5. Characteristics (continued) • Multiple years of data collection: • Baseline • One or more follow-up assessments • Multiple interventions: • Reading Comp. and P4K: four each • Math Curriculum: three to four

  6. Benefits • Impacts of school interventions, curricula, instruction, and professional development implemented in typical schools: • Often smaller impacts in large-scale studies (P4K) than in efficacy trials • Impacts from a wider range of contexts than generally found in efficacy trials: • Students • Teachers • Schools • Greater opportunity for subgroup analyses

  7. What Questions Can an RCT Address? • Intention to treat – yes • Treatment on the treated – yes, with assumptions • Ingredients and dosage – no design answers particularly well

  8. Intention to Treat • Impact of the opportunity to participate for all schools/teachers/students • Simply, the difference in the means for the treated group and the control group • Control group represents what we expect for treated group in absence of intervention • Most policy relevant • Subgroup analysis shows who benefits most

  9. Treatment on the Treated • More assumptions • Impact of intervention on those who actually receive treatment • Adjusts for those who did not participate and control who did • Less policy relevant because it does not show the impact of an implemented policy – TOT separates the impact on participation from the effect of the treatment • Subgroup analysis shows who benefits most

  10. Ingredients/Dosage • Many assumptions, some unrealistic • Like other designs, cannot address these questions with the same level of rigor • Random assignment to different “intervention arms” of ingredients/doses needed for defensible impact estimates

  11. Study Implementation Challenges • Objections involve interventions, not design, so negotiation teams must have substantive expertise • More innovation in schools in recent years – less inclined to adopt yet another approach • Districts want consistent instruction across schools • Alignment of curriculum with state testing • NCLB waivers – will schools and districts be held accountable (AYP) if the intervention fails?

  12. Study Implementation Challenges (cont.) • How much “intervention from the evaluator” is OK when it comes to fidelity? • Let policy and practice dictate decisions – what would developer or school do in the absence of the evaluation? • Maintain normal school, program, and intervention operations – make the design accommodate the program • When oversubscription is needed, programs and schools often overestimate student interest.

  13. Generic Design Challenges • External validity – random effects design requires larger samples for same precision as fixed effects – demos vs. “national evaluations” • Spillover or contamination from random assignment of schools to interventions • Effect of giving some students treatment on the outcomes for the control group (P4K) • Identifying the counterfactual

  14. Design Challenges: Case Study • Charter schools evaluation: • Research question concerns the impact of attending a charter school • How would a child attending a charter school have performed in the absence of that school? • Design is random assignment to charter/regular public schools (lottery) • All designs are limited here: • A nonrandom sample of students left the public school, which may change what is taught there • Different than the thought experiment of asking what would happen with no charter schools

  15. Another Case Study: P4K • What is the impact of being assigned to a pullout program? • How would students perform in the absence of the pullout program? • Pullout program: • RA of schools to one of four interventions • Within schools, RA of eligible students to the pullout program or the regular classroom • Change in the composition of the regular classroom and perhaps, change in curriculum and instruction when some struggling readers are removed

More Related