Impact Evaluation for Social Inclusion

1. Impact Evaluation • for • Social Inclusion • Hélène Giacobino • Director • J-PAL Europe • DG EMPLOI, Brussells,Nov 2011

2. Partisans want to show their program had an impact Further funding, re-election, promotion, depend onit Evaluation is not that easy Participants are often very different from non-participants Often programs are implemented in specific areas, at a specific moment for a specific reason Volunteers are often more motivated or better informed or… But we cannot observe the same person both: exposed and not exposed to the program Why Are Programs Not Evaluated?

3. The Need for Evaluation • We have little hard evidence on key questions • What is the most cost-effective program to reduce unemployment? What is the impact of microfinance? • Evidence is important for maximizing the impact of limited resources • Fight donor fatigue: Build the case for resources to combat poverty • Evidence provides an objective platform for debate • Evaluations can demonstrate that conventional wisdom needs to be rethought

4. The Need for Social Experimentation • Policy needs experimentation… • Too many errors are made all the time; too much time is lost; too much money is wasted • Without experimentation, the stakes are too high. What has been scaled up cannot afford to be wrong. No incentive to learn and progress • Experimentation needs to be serious… • We need to be rigorous in determining success or failure • Otherwise, everyone is free to defend his own pet project

5. J-PAL: Jameel Poverty Action Lab • A network of researchers at universities around the world • Focused on randomized evaluations • Founded in 2003 by Esther Duflo and Abhijit Banerjee, MIT Professors of Economics

6. Build capacity J-PAL Poverty Action Lab Improve the lives of the poor Evaluate social programs Disseminate the results

7. J-PAL today: 5 offices & 64 affiliates

8. J-PAL Evaluations around the World • 308 randomized evaluations in 45 countries • About 180 are completed

9. How to evaluate the impact of an idea? • Implement it on the ground in the form of a real program, and see if it works • To know the impact of a program must be able to answer counterfactual: • How would individual have fared without the program • But can’t observe same individual with and without the program • Need an adequate comparison group • individuals who, except for the fact that they were not beneficiaries of the program, are similar to those who received the program • Common approaches: • Before and after (But many things happen over time?) • Participants vs. Non-participants (But are they different? More motivated? Live in a different region?)

10. Basic set up of a randomized evaluation Target Population Not in evaluation Sample Population Random Assignment Treatment Group Control Group Based on Orr (1999) 10

11. Randomized assignment • This method works because of the law of large numbers • Both groups (treatment and control) have the same characteristics, except for the program • Differences in the outcomes can confidently be attributed to the program

12. How to introduce randomness • Level of randomization: • Person: better to detect very small effects • Group: class, clinics, villages. Sometimes easier to organize • Different ways to randomize • Organize a lottery • Randomly assign multiple different treatments • Randomly encourage some more than others • Randomize order of phase-in of a program • And more!

13. Advantages • Changes the research: • Researchers and partners have to work close together on the ground • The results are • transparent • Easy to understand and to communicate • Difficult to manipulate • Avoid long debates • Help to fight fads • Very efficient to convince donors and policy makers

14. Limits • Not appropriate when: • the impact to measure is a macro impact • the scaling-up of the pilot will modify the impact a lot • the experiment modifies the reality • the beneficiaries are in a context of urgency • It takes a lot of time, sometimes difficult to combine with political needs

15. When to do a randomized evaluation? • As soon as there is an important question and the answer will help fighting poverty • Not too early and not too late… • The program is not too special (can be used in many other contexts) • To be really effective, we need time, talents and .. some money

16. When NOT to do a randomized evaluation • When it is too early, the program still needs to be ameliorate • If the project is too small (not enough participants) • If it was already demonstrate that this program has a positive impact. Use the money for more beneficiaries! • If the program has already started and is not going to be expanded

17. The question of ethic • Resources are very limited any way • This random assignment is usually seen as very fair • Every project has to go through an ethic committee • J-PAL never works on projects if the cost of the evaluation means less beneficiaries. After the evaluation is over, if necessary, the control group will also get the program

18. The French Context

19. The Experimentation Fund • In 2008, creation of the Experimentation Fund for the Youth (Fond d'expérimentation pour la jeunesse) • To encourage innovative youth policies • To experiment and evaluate them • 230 million$ for 2 years, a team of 12 people • Within 18 months funded 350 projects

20. What was new? The French Government • encouraged initiatives and did not only support its own programs • decided to support projects only if they were evaluated • prepared from the very beginning the conditions of scaling up

21. La Mallette des Parents

22. La Mallette des Parents (the parent'stoolkit) Questions: • Is it really possible to improve parents’ involvement ? • Has increased parental involvement any effect on children? • Does the effect on program participants spread out on other families?

23. The program • Children and their parents in first year after primary school in low-income schools • A toolkit: • a booklet with thematic guidelines • 3 meetings with the parents • A DVD in 10 different languages • Cost: 1000€ per school (7€/child)

24. Messages to the parents • All parents can help their children • Work outside of school is extremely important for success • Parents should be involved in their children's homework • Children need to feel that their parents understand how school functions and care that they adhere to the demands of teachers and administration

25. Protocol: Four groups Test Classes Control classes Volonteer Volonteer Compare Non volonteers Non volonteers Compare

26. Parents' involvement • More interactions with school: • Meetings with teachers • Participation in parents' organizations • Helped more their children at home • No more difference in involvement between blue-collar and white-collar families

27. Childrens' behavior • Large improvement, even for classmates whose parents were not volunteers • 34% less likely to be sanctioned for disciplinary reasons • Similar results in reduction of absenteeism • Cognitive achievement: • Limited but significant impact in test scores in French • No impact in Math

28. Policy implication • Simple and inexpensive program • Rigorous evaluation: can convince schools or governments that such action is worth taking • Ongoing generalization in France • New program launched for children in the last class of compulsory school

29. Policy lessons • Directed parent discussion groups are an effective policy tool for increasing parental involvement, even in underprivileged area • Increasing parental involvement and awareness of school structure improved student behavior and positively impacted learning • Even though only a small fraction of parents choose to participate, the benefits of their involvement were felt by all children in the classroom.

30. Other Programs… • Counseling the unemployed • Counseling and job placement for young graduate job seekers • Counseling welfare recipients • Discrimination in hiring and anonymous CVs • Facilitating youth's access to apprenticeships and encouraging them to complete them • Small business training and loans for aspiring entrepreneurs in disadvantaged neighborhoods • Supporting 18-25 year-olds through long-term mentoring plus financial assistance • …

31. Thank you