Monitoring and Evaluation: A Logical Frame

1. Monitoring and Evaluation: A Logical Frame Evaluating the Impact of Projects and Programs Beijing, China, April 10-14, 2006 Shahid Khandker, WBI

2. What is M&E? • Monitoring and evaluation are tools that make it possible to identify and measure the results of projects, programs or policies

3. Why the Concern? • To evaluate and adjust strategies and activities • To report on progress to interested parties, clients, taxpayers and the general public • To identify and share with others best practices and lessons learned • To improve the programming of new interventions and strategies

4. Monitoring • Provides regular information on how things are working • Defn: A continuing function that uses: A) systematic data collection and analysis of specific indicators of progress. B) Provides management with indication of extent of progress towards goals - achievement of deliverables -use of resources. C) Contributes to performance improvement. D) Conducted by business unit.

5. Evaluation • Evaluation can only be done after a certain time and requires more thorough investigations • Conducted by independent evaluators. • Defn: A systematic and objective measurement of the results achieved by a project/program/policy in order to assess its relevance, the efficiency of implementation, its effectiveness, impact and sustainability.

6. Monitoring vs Evaluation • Monitoring assesses progress in implementation of ongoing programs • Evaluation provides a snapshot against some benchmarks at a point in time of programs that may or may not be continuing • Monitoring looks at progress relative to targets and assumes there is causality • Evaluation seeks to prove causality

7. Monitoring and Evaluation - Rationale • Monitoring • Holds implementers accountable for delivery of inputs • Provides basis for corrective action • Provides assessment of continued relevance • Evaluation • Accountability - was money well spent? • Learning - what could we do better next time?

8. … the direct results of which... … must have effects... … to undertake activities... This chain is based on a series of logical relationships (if… then) called the logframe Resources are mobilized… Evaluation Monitoring Monitoring and Evaluation, and the Chain of Results … and an impact on development

9. Health, literacy consumption, life expectancy, poverty amount of services provided, processes completed building, training, delivery budget funds Logframe of a Project or Program … the direct results of which... … to undertake activities... Resources are mobilised … must have effects… … and an impact on development Inputs Outcomes Impact Allocation Outputs

10. What Type of Indicators Are Needed?-MDGs- • Eradicate extreme poverty and hunger • Achieve universal primary education • Promote gender equality and empower women • Reduce child mortality • Improve maternal health • Combat HIV/AIDS, malaria and other diseases • Ensure environmental sustainability • Develop a Global Partnership for Development

11. Impact Impact on living standards – Lower poverty; prevalence of malnutrition 5-10 yrs Access, usage, and satisfaction of beneficiaries - % hh with access to safe water supply; % hh with continuous water supply during the day Outcomes 1-3 yrs Goods and services generated - Newly installed or rehabilitated water points;% wastewater treated Outputs Annual Resources (financial, material and human) provided - Real investment in water supply and sanitation Inputs Annual What Type of Indicators Are Needed? GOAL:Improve the health status of the population through improved water supply

12. From project to program –The new development strategy and its implications for M&E Outcome-based M&E Impact Monitoring the outcome and impact of development across sectors Outcomes Logframe Outputs Inputs Focus on demand-side issues (data for decisions): logframe provides the foundation; largely project/program level: operationally-focused; tend to emphasize M&E at lower end of logframe [inputs/outputs); link to project design/implementation Reflects a greater emphasis on: country-ownership, long-term capacity-building -also provides greater opportunities for civil society involvement and participatory M&E approaches. Link to PRSP

13. Why Monitor? • To ensure best use of scarce resources • To make user feedback integral to poverty reduction efforts • To mitigate the standard pitfalls in service provision to the poor • To enforce a concrete link between policy goals and policy inputs • To incorporate the many dimensions of poverty reduction into our decisions

14. What to Monitor • Identify few indicators, measure them well, and use the results for policy makers • A prioritized list of input, output, outcome and impact indicators for monitoring • Develop a data collection system to institutionalize a monitoring system: budget and administrative data, facility and other survey data.

15. How to Monitor • Manage a monitoring system that integrates a variety of different types of information: MIS, surveys and censuses, and participatory exercises • Draw up a monitoring matrix which identifies: -Data sources for each indicator -Frequency of measurements -The organization responsible for collecting information. • Collect information in a sequence that maximizes the complementaries between different types of data. • Outputs: -Annual progress report -Database for continuous monitoring -Desk studies

16. Purposes of an M&E Manager, Operational staff People in charge Decide 4 main purposes Know, be informed Manage, monitor Researcher Public, beneficiaries, contributors Know, understand and learn lessons

17. Key M&E Products Start Before During Afterwards At the end Program To check the design To improve the implementation For accountability purposes To assess the impact

18. Components of a M&E Strategy • Outcome-based monitoring system • Complemented with systematic impact evaluation • Creating a feedback process • Building capacity for monitoring and evaluation • Promoting participation

19. Outcome-based Monitoring System • Setting goals and targets (including establishing the baseline) – where do we want to go? • Identifying indicators that can be used to measure progress towards goals • Collecting data - what progress is being made? • Providing feedback for decision making – what needs to be changed along the way

20. Monitoring System: Where to Start? • Stock taking: -Who are the actors? What are their activities? What is their capacity? What are their roles and needs? • Review the actions currently undertaken to build or re-enforce capacity. • Review of needs in information • Review of the institutional framework.

21. Monitoring System: What Limitations? A series of issues are of concern: - Roles badly defined - Lack of coordination - Lack of reliability and relevance of information - Difficulty in accessing information - Long delays in production of information - Lack of use of the data by the users

22. Example: Monitoring Poverty Reduction

23. Monitoring poverty reduction requires • Measuring poverty • Understanding poverty reduction strategies • Monitoring outcomes and impacts—goes beyond project-based monitoring • Evaluating m&e system and its capacity

24. Measuring poverty • Poverty is a multi-dimensional concept—income, consumption, health, education, water, housing, sanitation, capability and empowerment • Poverty is also dynamic—vulnerability and coping mechanism • Poverty is also gender, caste, ethnic, and geographical based

25. Poverty reduction strategies • Market matters; state matters too – they can’t be substitutes; a right balance is required. • Involve the poor in development process—poverty reduction requires both growth and targeted measures • Development strategy needs (a) policies to improve the investment climate to raise income and employment growth, and (b) policies to equip and empower people to participate in growth.

26. Better investment climate and better participation • Macroeconomic stability and openness • Good governance and strong institutions • Quality infrastructure • Growth reduces poverty– but not all growth is equally pro-poor • Empowerment through social, human, and infrastructural investment to enable poor to participate in growth

27. Setting up an M&E system for Poverty Monitoring • Defining information needs. Deciding on indicators • What tools do we need? What tools do we have? • Defining institutional functions (including feedback processes) • Defining a 5-10 year work program • Defining resource requirements

28. Measuring outcomes - What tools do we need? What tools do we have? • Tools for measuring the extent and depth of poverty (poverty analysis) • Tools for monitoring changes over time (poverty dynamics) • Tools for comparing differences over space (poverty mapping) • Participatory poverty monitoring (listening to the poor)

29. Measuring poverty and data quality • Tools: - A good household survey (there are various options - HBS, LSMS etc.) • Measuring poverty is a data hungry exercise (means and distributions) • Household Consumption (If you’re going to measure it - do it properly) • Sampling and non-sampling error • Monitoring over time compounds the error problem

30. Monitoring changes over time • Tools: Administrative data/MIS; Institution-based surveys; household surveys • Monitoring leading indicators (Service Delivery) • Key attributes of an ‘indicators survey • Simple to execute • Rapid reporting • Disaggregatable to low levels • Core Welfare Indicators Questionnaire (CWIQ)

31. Monitoring Spatial/regional differences • Main source of information on distributional outcomes - household surveys - permit only limited disaggregation. • Very large data sources (e.g. census) typically collect very limited information on welfare outcomes. • Impute a preferred measure of welfare (e.g. comprehensive real consumption) from household survey into census, using statistical prediction methods

32. What are Poverty Maps?

33. Participatory Poverty monitoring (listening to the poor) • Limitations of statistical information • Qualitative and participative approaches • Complementary not alternative approaches

34. Data to monitor investment climate and empowerment • Firm and institutional surveys • Good governance indicators • Empowerment indicators • Community participation indicators • Data on quality of infrastructure • Both quantitative and quality information required

35. Key messages • Monitoring poverty reduction goals goes beyond selection of indicators and measuring inputs and outputs • A range of different monitoring tools are required • Poverty reduction monitoring goals should include statistical capacity-building as a core element • Must be a national commitment to funding monitoring and statistical systems • Must ensure existence of appropriate incentives for good M&E? • Feedback into budgeting and policy process is essential