OVE’s Experience with Impact (Treatment) Evaluations

1. OVE’s Experience with Impact (Treatment) Evaluations Presentation prepared for DAC, 15th November 2006

2. Policy • The general evaluative questions proposed by the IDB’s ex post policy, approved in 2003, are (i) “…the extent to which the development objectives of IDB-financed projects have been attained.” and (ii) “… the efficiency with which those objectives have been attained” (para1.1 ) • Policy left for practice: sampling, methodology, organizational framework, and the forum for the presentation of results. • Note the task is to evaluate already approved and/or closed projects ( average project time is about six years).

3. Implementation decisions • Project Sampling Strategy: Option: random or meta-evaluation. Decision :meta-evaluation. • Method and Project types: (i) process cum naïve or treatment (impact) evaluations. Decision Treatment effect evaluations; and (ii) projects with partial or national coverage. Decision partial coverage models • Organizational. Decisions: (I) separate activity within the office; (ii) evaluations to be carried out both in-house and outsourced.Therefore: (I) hired staff with appropriate expertise; and (ii) created EVALNET, a register of evaluators; • Forum for presenting results. Decision: overall report (sent to the Board) with background-working papers (discussed in ad hoc seminars).

4. Evaluative questions • what were the problems that the program was designed to tackle? • what was the policy response, i.e. the design features of the program? (theory based evaluation) • was the program of a sufficient size given the size of the problem(s)? • were the program’s deliverables provided in a cost efficient (and cost effective manner)? • What was the incidence and was the program well targeted? • what was the impact on welfare outcomes of the program?;and • what were the benefits relative to the cost of the program ?

5. What was the impact on welfare outcomes of the program? • To answer the question OVE normally use three approaches in the same evaluation: • Naïve evaluation • Regression based (cross-section and panel) • Treatment effects

6. Social Investment Fund (naïve evaluations can be misleading • Profile: • Social Investment Fund. Panama • Basic Infrastructure to poor communities • Data: • Distribution of benefits by municipalities from administrative data • Baseline and results of outcome indicators from households surveys 1994-2001 • Technique: • Treatment and comparison group using PSM in double difference. The sample included 75 municipalities. • Potential to work with a sample of more than 250 smaller geographic units but household survey was not representative at that level • Results: • Naïve evaluation: the program failed. Impact evaluation: the program succeeded

7. Labor Training Project (positive effects) • Profile: • Labor Training program – Dominican Republic • Data: • Simple randomization including a follow-up survey done at 10-14 months after graduation from training • 786 treated and 563 controls • Baseline has universe, follow up was a stratified random sample (size determined by standard formulas) • Technique: • Estimated average Intention-to-treat on treated by simple diff of means, verified with weighted diff and regression analysis (no DD b/c faulty baseline) • Results: • Employability, income and health insurance access increased. Program succeeded

8. Public Housing Program • Profile: • Progressive Housing Phase I – Chile • Provision of low cost basic dwellings to poor families • Data: • Household Surveys identified beneficiaries and applicants to the specific housing program • Technique: • Treatment from beneficiaries and comparison from applicants using PSM. Single difference from a sample of 508 Beneficiaries and 476 applicants • Results: • Quality of dwellings improved • Little or not change in other welfare outcome indicators. • Difference between naïve versus impact

9. Benefits Costs, benefits, and internal rate of return • Profile: • Progressive Housing Phase I – Chile • Provision of low cost basic dwellings to poor families • Data: • Household Surveys identified beneficiaries and applicants to the specific housing program • Technique: • The benefits of the program are the additional (necessary) household income required to obtain equivalent dwelling • Results: • IRR: greater than 18% • Benefits: Net present value per solution ~1150 US$

10. Rural Roads (decay of benefits over time) • Profile: • Rural Road – Peru • Construction and upgrade of roads in rural areas • Data: • Specific survey of beneficiaries. Baseline collected after program started. Follow-up survey 3 years after program closed • Technique: • single difference and double difference • Results: • Positive impact on income and assets’ values of rural households. • Decreasing impact for motorized roads not for non-motorized roads.

11. National Transfer Fund (dosage and multi-treatment effects) • Profile: • National Fund for Regional Development • Decentralized investment to finance infrastructure and productive projects • Data: • Administrative data for distribution of benefits by municipalities • Baseline and results of outcome indicators from households surveys 1994-2001. The sample included 343 municipalities. • Technique: • Impact evaluation using PSM in double difference. The municipalities grouped by per capita investment using cluster analysis. • Results: • Positive and increasing impact on poverty incidence (reduction) on per capita investment • Not impact on poverty if investment is intensive in education • Greater impact on welfare composite index in municipalities with diversified investment

12. SCIENCE AND TECHNOLOGY: Research • Profile: • Science and Technology – Chile • Financing for R&D projects • Data: • All projects that between 1988 and 2004 received the financial support of the program and a stratified sample of projects submitted to the program, which were not financed because they ranked below the threshold defined for being admitted to the financing. • 2,936 different research projects (932 financed by the FONDECYT and 1704 not financed) 4,959 publications recoded in the ISI – SCI (1873 by financed researchers and 3806 by not financed researchers). • Technique: • Discontinuity regression design. The selection process drawn by a “threshold” quality value that separates beneficiaries from non-beneficiaries • Results: • Unsuccessful. FONDECYT has no significant positive impact on the scientific production of the financed projects.

13. Profile: Public grants-credits to firms for innovation Data: Administrative data on firms and firm level surveys (OSLO design) Technique: Double difference with propensity score matching Results Generally positive and significant effects on employment, and sales, but little evidence of effects on patents and total productivity . Technology Development Funds

14. EXPERIENCE: Findings Potable Water • Positive effect on health outcome (treatment less than naïve effect) • heterogeneity of results important. a regressive relationship between treatment effect and income, where more educated (and wealthier) households did better than less educated (and poorer) households • Ramification for project design: projects should include or be coordinated with, as a hypothesis to be tested, a health education component together with potable water expansion. Impact on infant mortality

15. Balance • Since 2004 have produced about 23 evaluations • Cost per evaluation was about $60,000 But • Problem of obtaining effective counterparts (in Bank and country) to accompany the evaluation from beginning to end. Started outreach program to obtain formal counterparts in the country, and form ad hoc interested specialist for each thematic study. • Mainstream impact evaluations into other evaluations of the Office • Problem of communicating the findings. Started producing different reports for different audiences for the same evaluations.

16. Before After Still far from the million words of a good picture

17. Regression Approach • Panel data • Cross section data Where y is the outcome of interest, D is the dummy for participation in the program, V control variables