Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle

1. Experimental Evidence on the Early Effectiveness of Intervention in Childhood Orla Doyle (UCD School of Economics & UCD Geary Institute) Conference on Irish Economic Policy 1st February 2013

2. Importance of Early Childhood Investment Targeted early intervention programmes effective way of reducing socio-economic inequalities in children’s skills Well-designed interventions can generate a return to society ranging from $1.80 to $17.07 for each dollar spent YET primarily US-based evidence only on the effectiveness & cost-effectiveness of such interventions Argument for early intervention in more generous welfare systems is not rooted in strong empirical evidence

3. Irish Context • Prevention & Early Intervention Programme (The Atlantic Philanthropies) & Government , DCYA) initiated in 2004 • Aim: to support the development, implementation and evaluation of strategies for disadvantaged children • All programmes evaluated using experimental or quasi-experimentaldesign • Little tradition of social experimental evaluation in Ireland • Comparing the effectiveness of multiple programmes will enhanced resource allocation and policy design

4. Preparing for Life Programme • Community-led initiative: operated by Northside Partnership in highly disadvantaged area of Dublin, Ireland • Bottom-up approach: community initiative involving 28 community groups, service providers, & local representatives • Evidence of need: Children scored below the norm on cognitive & non-cognitive skills on school entry (Doyle & McNamara, 2011) • Aim:Improve levels of school readiness by assisting parents in developing skills to prepare their children for school • Evaluation Design: RCT & embedded quasi-experimental design

5. Design of Preparing for Life

6. PFL Evaluation Impact Evaluation • Aim: To determine if the programme/service is effective • PFL data collection: Pre-intervention (baseline), 6mths, 12mths, 18mths, 24mths, 3yrs, 4yrs, school entry Implementation Evaluation • Aim: Delve into the blackbox of programme effectiveness & evaluate fidelity to the model • PFL data collection: • Implementation data • Focus groups with participants • Semi-structured interviews with programme staff

7. Recruitment • Eligibility Criteria: • Cohort of pregnant women residing in PFL catchment area between Jan 2008-August 2010 (32 months) • Includes nulliparous and multiparous women • Recruitment: • Maternity hospital at first booking visit (b/w 12-26 weeks) • Within the local community • Population-based recruitment rate, based on all live births during the recruitment phase, was 52%

8. Randomisation Procedure • Individual-level randomisation • Unconditional probability randomisation strategy • Design procedure that preserves the integrity of randomization ie. no opportunity for recruiter to intentionally influence assignment • Randomisation process • After informed consent obtained, participant pressed key on tablet laptop to allocate her assignment condition • Evaluation & Implementation team automatically received an email with the assigned ID number and treatment condition • Thus no opportunity to compromise the randomisation process

9. Baseline Analysis • Determine the effectiveness of the randomisation procedure • Baseline assessment conducted post randomisation, pre intervention • Randomisation worked!

10. Baseline Differences in Selected Socio-Demographics

11. Methodology • Estimate treatment effects at birth, 6 months, 12 months • Permutation based hypothesis testing (Heckman et al. 2010) • Classical hypothesis tests unreliable when the sample size is small and the data are not normally distributed • Permutation test are distribution free, thus suitable in small samples • Based on the assumption of exchangeability between treatment conditions under the null hypothesis • Stepdown procedure (Romano & Wolf, 2005) • Ignoring the multiplicity of tests may lead to the rejection of “too many” null hypotheses • Test multiple hypotheses simultaneously by controlling overall error rates for vectors of hypotheses using the family-wise error rate (FWER) as a criterion • Less conservative & more powerful than other methods as takes account of statistical dependencies between tests

12. Analysis of Maternity Records • Data Collection • Summer 2012 accessed maternity records for PFL participants who attended Rotunda Hospital (85%) & NMH (15%) • 283 parents gave consent to access records - available records for 272

13. PFL Maternity Records – Summary • No programme impact on birth weight or gestational age, but programme altered the birth experiences of the participating families • Significantly fewer mothers in the high treatment group had a c-section compared to those in the low treatment group • Result remains when the incidence of a previous c-section and pariouness are controlled for • Programme particularly effective in preventing high treatment mothers from having a repeat c-section • Result possibly linked to Tip Sheets which specifically focus on preparing the mother for the birth and being aware of the birthing process and possible labour outcomes

14. Analysis of 6 Month Results • Sample: PFLHigh/Low treatment groups: 84/90 • Test for treatment effects across 160 measures & 25 Step-down categories

15. Example of Results: Child development @ 6 Months Notes:1 one-tailed (right-sided) p value from an individual permutation test with 1000 replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000 replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates the variable is not statistically significant.

16. Positive Treatment Effects @ 6 Months • Child Development:none • Child Health:Appropriateness & frequency of eating; Immunizations • Parenting:Parent-child dysfunctional interactions; Baby comparison score; Parental hostile-reactive behaviour; Interaction with baby • Home Environment: HOME sub-scales – variety, childcare, toys & books, physical environment, learning materials; Electrical socket covers, Framingham safety survey • Mother’s Health:Mother less hospitalisation for complications after birth • Social support: Frequently sees grandparent; Discuss programme with partner & others; Knows children same age as baby; Meet friends frequently • Childcare & Service use: Voted in last election • Household & SES:none

17. Analysis of 12 Month Results • Sample: 12 Months: PFLHigh/Low treatment groups: 83/82 • Test for treatment effects across 140 measures & 23 Step-down categories

18. Positive Treatment Effects @ 12 Months Child Development:ASQ Fine Motor skills; BITSEA Social Competence cut-off Child Health:Chest Infections; Immunizations; Appropriateness eating Parenting: none Home Environment: none Mother’s Health: Drank alcohol Social support: Meet friends frequently; Voted in last elections Childcare & Service use: ~ Household & SES:none Results consistent with the home visiting literature

19. Attrition & Dosage • Attrition/disengagement • Attrition may bias evaluation results if non-random • 29% of high treatment & 29% of low treatment dropped out/disengaged between randomisation & 12 months • Few individual characteristics predict attrition – IQ more important • Dosage/engagement • Variation in dosage, may lead of variation in effect • Collected data on number/duration of home visits • On average, high treatment group received 21 home visits of ~1 hr in duration between pregnancy & 12 months • 58% of prescribed visits delivered based on bi-weekly visits • Higher maternal IQ associated with higher engagement

20. Conclusions to date • Many of the results are in hypothesized direction • Few significant effects, yet consistent with literature • Significant findings on birthing experiences, parenting, home environment, child health, social support • Corresponded directly to information on the PFL Tip Sheets • No significant impact on many key domains such as child development, birth weight, breastfeeding, maternal well-being etc. • Attrition & disengagement between 6 & 12 months very low • Engagement below prescribed dosage • Participate satisfaction is relatively high (focus groups)

21. Update on PFL • Oldest PFL child is 4 years and old & youngest is 18 months • 24, 36 & 48 month surveys are currently in the field • 18 month results available very soon • Over 1,600 interviews conducted as part of impact evaluation • Conducted five school readiness surveys with junior infant cohorts in 2008, 2009, 2010, 2011, 2012 • Evaluation will continue until all children start school Programme website:www.preparingforlife.com Evaluation website: http://geary.ucd.ie/preparingforlife/