OVE’s Experience with Impact Evaluations

1. OVE’s Experience with Impact Evaluations Paris June, 2005

2. Impact Evaluations • Alternative definitional models: • time elapsed since intervention • Counterfactual comparison • OVE adopted the counterfactual approach, and further limited the initial sample to programs with partial coverage. • Partial coverage allows observation of treatment effects through comparison of treated and untreated groups

3. Policy • The general evaluative question proposed by the IDB’s ex post policy is “…the extent to which the development objectives of IDB-financed projects have been attained.” • This questions is most convincingly answered through treatment effect evaluations

4. Selecting Projects • Random selection is appropriate for accountability-oriented evaluations • Purposive selection of projects of similar design across countries is better for generating learning regarding the model underlying the interventions • Clusters of like projects permit meta-evaluations of models

5. Projects Selected • Chose purposive cluster sampling strategy but some stand-alone projects. A total of 16 projects were selected • Clusters: (i) Neighborhood Improvement projects and (ii) Land Titling Projects. • Stand-alone: Cash Transfer (Argentina), Potable Water (Ecuador); Agricultural Subsidies and Cash Transfer Programs (Mexico’s Procampo and Opportunities programs); Social Investment Fund (Panama) • Stand-alones serve as pilots for future clusters

6. Both Performance Monitoring and Treatment Effect Are Required Treatment Effect includes randomized design; propensity score matching, controlled comparison, discontinuous regressions.

7. Limits to Treatment Effect Evaluations

8. Experience • The required information supposedly generated through standardized performance monitoring is absent in a large majority of IDB projects examined • 10 of the 16 selected projects had inadequate data for treatment effect evaluation • 6 of the 16 could be retrofitted with sufficient data to attempt a treatment effect evaluation • Retrofitting implied significant data collection costs, costs that could have been avoided had adequate performance monitoring been in place over the life of the project.

9. The Bank’s Current Portfolio Of 593 active projects in mid -2004: • 97 (16%) claim existence of information for at least one development outcome, of which • 27 have the information in an electronic form, of which for • 5 the information is held in the Bank, of which • 2 appear to be collecting data for treatment effect evaluation

10. Experience: Limits to Retrofitting • The questions answered are dependent on the information found rather than on the relevance and usefulness of the hypotheses being tested: the tail wagging the dog . • It severely limits the set of control variables’ information thus reduces the veracity of treatment effect findings • retrofitted data may not correspond to the development outcomes declared by the projects. A project can be evaluated using intended and unintended effects, but should at least consider as a minimum the intended ones.

11. Experience Confirms the Value of Treatment Effect Evaluation In just one project (Neighborhood Improvement, Rio-Brazil) comparing naïve and treatment effect the following held regarding naïve and treatment effects: positive/negative, negative/positive; greater/smaller; smaller/greater; and the same. 0.50 Naive 0.40 Treatment 0.30 0.20 0.10 0.00 -0.10 -0.20 Water Sewer Rubbish Illiteracy Income Rent Child Mortality Homicide Rate

12. Before After So pictures need to be interpreted with caution

13. Experience • “…the six treatment effect evaluations undertaken during 2004 do show that the Bank’s interventions have a significant development effect for at least one declared development objective. These findings suggest that the Bank may be currently understating its contribution to development.”

14. EXPERIENCE: Findings Land Titling • “Beneficiaries of Land Regularization projects saw property values for their land increase …. However, for the other purported development effects (greater productivity, increased investment, and greater access to credit), no unambiguous treatment effects were found. • Ramifications for project design: for small and poor producers to benefit from a pro-market regime, titling alone is not sufficient • Transaction costs and market distortions that limit access to credit must be also simultaneously be addressed

15. EXPERIENCE: Findings Potable Water • heterogeneity of results important. a regressive relationship between treatment effect and income, where more educated (and wealthier) households did better than less educated (and poorer) households • Ramification for project design: projects should include or be coordinated with, as a hypothesis to be tested, a health education component together with potable water expansion. Impact on infant mortality 0.1 0 -0.1 -0.2 proportional change -0.3 -0.4 All Sample At least Primary -0.5 Bottom 25% 25%-50% 50%-75% Top 25% Expenditure level

16. EXPERIENCE: Findings from cash transfer and agricultural subsidy programs Issue: Do conditions attached to cash transfers produce more change than the transfers alone • Income effect alone may be substantial, and conditionalities are costly to administer and monitor • In a comparison between two programs in Mexico with and without conditionality the following ramifications for project design were found: • Conditionality (school and clinic attendance) does result in an effect over and above the income effect of the transfer. • Transfers to the mother as opposed to the father matters as the effects are greater when the transfer is to the mother

17. EXPERIENCE: Low Costs • Treatment effect evaluations can be done inexpensively, if attention is paid to data at the time of design and during implementation • Data collection costs can be substantial if retrofitted, but still within reasonable limits. • Costs ranged from $28,000 to $92,000 per evaluation, much lower than the “norm”: small budget high returns

18. Summary • Initial experience with treatment effect impact evaluations provided considerable knowledge relevant for future project design • Costs were moderate, and can be expected to be lower in the future if the performance monitoring system is improved • Data has value to researchers, and cost-sharing in data collection was possible in several cases • Treatment effect evaluation provides the only convincing basis for asserting development effectiveness