An impact evaluation of Ethiopia’s Food Security Program

1. An impact evaluation of Ethiopia’s Food Security Program John Hoddinott, IFPRI (in collaboration with Dan Gilligan, Alemayehu Seyoum and Samson Dejene)

2. The Context of the Food Security Program • Ethiopia has been one of the largest recipients of emergency food aid in Africa for the past decade • emergency appeals approach has been costly: avg cost of $265 mn from 1997-2002, reaching > 5 mn people per year • emergency appeals have had limited effectiveness at protecting productive assets and mitigating drought shocks • In 2005, the Government of Ethiopia revised its strategy of distributing food aid • emergency appeals replaced with a standing safety net in areas suffering from chronic food insecurity • focus of new program was to provide more reliable and timely support to chronically food insecure households in more than 260 woredas (counties) across Ethiopia • increased funding for complementary programs to foster graduation from the safety net Page 2

3. The Food Security Program • Major components of the Food Security Program (FSP) • Productive Safety Net Program (PSNP) • labor intensive public works • use safety net to build productive community assets • transfer payments in cashrather than food in some areas to improve market development through safety net • Other Food Security Programs (OFSP) • Direct Support (DS): unconditional transfers to labor-scare households including elderly and disabled • Other components: subsidized fertilizer, subsidized credit, other inputs or assets • Resettlement to other locations with more productive land • Evaluation study will cover 1 & 2, not 3 resettlement Page 3

4. The impact evaluation: General focus • Measure the impact that the FSP has on the well-being of the chronically food insecure population • Investigate the complementary roles played by the PSNP and the OFSP in achieving positive outcomes for the food insecure • Use the Food Security Bureau’s M & E log frame to guide the choice of process and outcome indicators to be assessed Page 4

5. The impact evaluation: Specifics • Process: • Who participates? • Measures of targeting effectiveness • What do beneficiaries actually receive? • Timeliness • Purchasing power of cash transfers • What public works are of actual use to beneficiaries? • What are beneficiaries’ perceptions of how well the program works? Page 5

6. The impact evaluation: Specifics • Outcomes • Improvements in food security status. Examples (not exhaustive) • Percentage of households reporting no food gap • Improvements in dietary quality • Changes in agricultural production • Increased use of credit, fertilizer, improved seeds • Percentage increase in crop production • Change in production of livestock products • Changes in assets • Increased holdings of livestock and other productive assets • Reduction in distress sales of assets Page 6

7. The impact evaluation: Possible future topics • What are the prospects for significant graduation from the program? • Does the FSP lead to safety net dependency • What is the effect of the FSP on local labor and output markets? • How do these all of these impacts differ by food versus cash transfers? • What role does community participation play in the selection of works projects and their effectiveness? Page 7

8. Evaluation Design: Approach • Empirically examining these issues requires new data • Approach: Process • Close interaction with the Food Security Bureau, CSA and the donor group on scope of evaluation, sample design and questionnaire • Close collaboration with CSA on details of sampling, detailed questionnaire development and implementation • Current status: Training of trainers is ongoing; survey to be implemented in July and August Page 8

9. Evaluation design: Sampling and survey Instruments • Random sample of 68 woredas in the FSP from 4 major regions of Ethiopia; 2 enumeration areas (EAs) per woreda; 25 households per EA (N= 3500 households) • Within EAs, sample stratified on participation in PSNP or Direct Support: 15 beneficiaries and 10 non-beneficiaries • want comparison households to operate in the same markets and have similar distribution of unobservables • Household and community surveys capture measures of program participation, wide variety of outcomes, control variables, transfers • community selection criteria used to determine eligibility • household knowledge of eligibility • household consumption • transfers from all FSP programs and labor supply in PSNP Page 9

10. Evaluation Design: Methods • In some respects, an ideal evaluation design require “before/after” , “with/without” data based on random assignment • In the case of the PSNP: • Program has been purposively placed • Program has been operating for at least one year • Consequently, we are planning on doing the following: • Interview both beneficiaries and non-beneficiaries (which gives “with/without”) • Insert retrospective questions on past asset levels; crop choice; non-ag activity; assets (including livestock); past levels of food insecurity (which gives “before/after”) Page 10

11. Evaluation Design: Methods, cont’d • Use a quasi-experimental approach to identification of FSP impact on household food security and welfare, • Specifically, use the difference-in-differences matching estimator of Heckman, Ichimura and Todd (1997, 1998) • We match using the recall data found in the baseline survey, detailed information on program eligibility and on shocks. This approach has proven to be very effective variables for the matching model in our earlier work on the impact of food aid from the ERHS. • A further survey is planned for 2008 which will permit the assessment of the longer term impact of the program Page 11

12. Summary • The evaluation approach taken here is characterized by: • Participation by stakeholders in design • Collection of process indicators which gives a beneficiary perspective on the functioning of the program • Rigorous impact evaluation Page 12