1 / 27

Individual vs. Group Randomized Trials

Individual vs. Group Randomized Trials. Jens Ludwig University of Chicago, Brookings Institution and NBER. Three things to consider. Realizing randomization Power Nature of the intervention. Realizing randomization. “Should I randomize at the individual or the group level?”.

davin
Download Presentation

Individual vs. Group Randomized Trials

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Individual vs. Group Randomized Trials Jens Ludwig University of Chicago, Brookings Institution and NBER

  2. Three things to consider • Realizing randomization • Power • Nature of the intervention

  3. Realizing randomization • “Should I randomize at the individual or the group level?”

  4. Realizing randomization • “Should I randomize at the individual or the group level?” • Sort of like asking: • Should I take my $30 million Powerball prize all at once, or spread it out in installments?

  5. Why are so many American public schools not performing better? • Four major hypotheses: • Inadequate resources • Inefficient production technologies (curriculum, etc.) • Unmotivated teachers • Unmotivated students • Education researchers usually can only randomize what they can pay for • Severely limits which of these we can study with randomized experimental designs of any sort

  6. Keep our eye on the prize • Our real goal has to be to convince governments to do more randomization • Tennessee STAR • Progressa • Even Roland Fryer w/ Eli Broad at his back is taking on just a modest slice (pay for grades) • First consideration: Choose unit to randomize you need to in order to be able to randomize

  7. Me: “Can we randomly assign the intervention?” City official in very large Midwestern city: The rhetoric of randomization

  8. Me: “Can we randomly assign the intervention?” City official in very large Midwestern city: “No way.” The rhetoric of randomization

  9. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” City official in very large Midwestern city: “No way.” The rhetoric of randomization

  10. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” The rhetoric of randomization

  11. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” The rhetoric of randomization

  12. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” The rhetoric of randomization

  13. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” Me: “Could we flip a coin, which would be the fair thing to do?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” The rhetoric of randomization

  14. Me: “Can we randomly assign the intervention?” Me: “Well, could we do a pilot program?” Me: “How do you decide who gets the pilot program if there is excess demand?” Me: “Could we flip a coin, which would be the fair thing to do?” City official in very large Midwestern city: “No way.” City official: “Sure. We do pilot programs all the time.” City official: “Good question.” City official: “Ah, now I get it.” The rhetoric of randomization

  15. The rhetoric of randomization • Never use term “randomized experiment”

  16. The rhetoric of randomization • Never use term “randomized experiment” • Acceptable talking points: • “Pilot program” • “Excess demand” • “Fair, random lotteries” • If there would be a natural unit for doing “regular” pilot program, randomize that • I.e., we’d just be implementing an unusually informative pilot program • For many education interventions would seem to argue for group randomization (ex: pay for grades)

  17. Second consideration:Power • There is the standard statistics version: • More power from 1,000 kids distributed across 1,000 schools than 1,000 kids distributed across < 1,000 schools • Due to non-independence of student observations within schools

  18. Then there is the real-world version of this issue • We live in a resource-constrained world • Are there economies of scale in data collection? • Cluster randomization could reduce data collection costs for same reason that population surveys use two-phase sampling • Are there economies of scale in delivering the intervention itself?

  19. For a given budget, cluster randomization could in principle generate more power • Imagine rapidly declining marginal costs of “treating” and studying kids w/in a school • Suppose school-based self-administered student and teacher surveys plus use of student administrative school records • Suppose we have a DARE-like intervention (“Don’t do drugs kids, look what happened to me!”)

  20. Power considerations in a resource-constrained environment • It’s conceivable you could get more power out of a clustered sample of given size N, even with non-trivial intra-class correlations (ICCs) • Significant information requirements: • Need to know ICCs for your sample & outcome • Need school system to help you think about average cost & marginal cost schedules for intervention • Need very good survey subcontractor to help you think about economies of scale in data collection

  21. Power considerations • On the other hand, you can run out of observational units quickly in clustered experiments • Imagine randomizing schools: • Even in Chicago, “just” 116 high schools, 483 elementary schools • Imagine half of schools meet your eligibility criteria, then half of principals agree to cooperate in experiment, then you randomize half to T and C • That would be 14 treatment high schools, 14 controls • Or 60 treatment elementary schools, 60 controls

  22. Nature of the intervention • Previous two issues are shamelessly pragmatic • Individual vs. group choice could also hinge on substantive considerations

  23. Spillover effects • Stable unit treatment value (SUTVA): • Your treatment effect is independent of how many others get treated • If violated, then your treatment effect estimates generates only to situations with similar take-up rates • Sometimes you want to study intervention independent of these spillovers, while sometimes spillovers key part of treatment

  24. Spillover effects example:Moving to Opportunity (MTO) • Initial concerns: • Generic SUTVA concern • Groups of public housing families moving in to new areas might generate backlash • Also didn’t want families to recreate any unproductive baseline social ties • New concerns: • Families lose access to productive social ties, so should we have randomized in groups?

  25. Spillover effects example number 2 • Roland Fryer, paying kids for grades in Chicago, DC, NYC • Wants to change whole school climate around academic achievement • Generic adolescent (American?) anti-intellectualism • Plus specific “acting white” concerns • Only by paying everyone (or offering to pay everyone) can you change peer norms (or try to change peer norms)

  26. Interventions as public goods • MTO suggests neighborhood safety key factor for parental mental health • Maybe for kids, too • Could also affect learning in other ways, too • If you reduce crime in neighborhood, every kid in neighborhood will benefit • This is sort of another way of talking about economies of scale in providing intervention

  27. Bottom lines • Clustered experiments might help realizing randomization • Power considerations complex in a resource-constrained environment • Research community needs to develop infrastructure to meet informational requirements for decisions • Substantive considerations about role of spillovers and “public good” interventions • ‘Tis better to have randomized at the wrong level than to have never randomized at all?

More Related