Impact Evaluation Conference, Cairo, Egypt April 2, 2009 Nyamwaya Munthali

1. Integration of the most significant change approach in measuring impact for market development projects Impact Evaluation Conference, Cairo, Egypt April 2, 2009 Nyamwaya Munthali Impact evaluation-approaches and methodologies

2. Presentation Outline

3. It’s better to be approximately right than precisely wrong. The Pursuit for Effective Impact Evaluation Tools for Development Programs 1. John Maynard Keynes

4. The Pursuit for Effective Impact Evaluation Tools for Market Development Programmes • Impact evaluation - as the component of a project cycle that seeks to validate to what extent changes occurring to target groups can be attributed to project interventions - is a topical issue in the development sector. • Impact evaluation is a necessary part of project planning and monitoring and for improving implementation strategies regardless of type of organization • For evaluators the challenge now is how to adapt the basic approaches of M&E (qualitative and quantitative) to develop an innovative and cost effective M&E model • For evaluators of market development projects developing models is even more of a challenge due to the complexity of the market itself (supply and demand side)

5. Market development indicators 1. Market intervention indicators -Relating to outputs, measuring the immediate results of project activities e.g sales performance of products, increased stock volumes, size and expanse of retail networks 2. Market development indicators -Relating to outcomes, measuring results over space and time including access to information on products location, price and benefits, impressions of specified products to drive demand, supply side development and adherence to business and marketing plans, suppliers capabilities in product stocking and marketing. Others may includes adherence to product quality, assessment of the users knowledge on how to use technology, their purchasing power and demand. 3. Market sustainability indicators -Relating to impacts, measuring results when the project intervention is withdrawn

6. Beyond market intervention indicators • Market development evaluators need to go beyond measuring market intervention indicators as the success of a market development project depends on sustained product adoption • Market development indicators are difficult to track because they are qualitative • The MEDA experience has shown that the adoption of MSC as an evaluation technique can be an effective tool in assessing market development projects and specifically market development indicators

7. 2 Most Significant Change (MSC); Methodology and Application

8. Defining MSC • Most Significant Change (MSC) is a participatory, qualitative and complimentary approach to impact evaluation. • MSC can be likened to newspaper development. Where stories are collected and prioritised. In this case, however, groups of beneficiaries are asked the MSC question and a panel picks the “best” MSC story. These stories are then compiled into a “catalogues/newspaper” and disseminated to decision makers. There can be many levels of sifting stories depending on the size or goal of MSC enquiry. • The MSC approach was originally developed in 1994 by Rick Davies through his work on a rural development project in Bangladesh with the Christian Commission for Development Bangladesh (CCDB) and has been adopted by a number of organizations in different countries • MEDA has applied MSC to measure market development indicators to be specific issues around sustainable adoption of SITT’s in the value chain. What is Most Significant Change?

9. Steps of MSC (Davies et al 2005)

10. 3 MSC Integration in Market Development Programs; Rationale and Case for MEDA Prosperity through Innovation Zambia Project

11. The MEDA Project - Prosperity Through Innovation

12. Partner Organisation; Irrigation Trainers BUSINESS SUPPORT SERVICES Organization interventions or Member Services Provision of Training on Irrigation Smallholder Farmers Business Transactions on water technologies Water Technology Suppliers and Retailers MEDA Supply side strengthening through capacity building on marketing and cross cutting issues Market Facilitation Market Development Project Framework

13. The Rationale • Lack of Baseline Survey from partners • Market Development program; requiring in-depth understanding of the benefits to program beneficiaries • To supplement training needs assessment tool and client satisfaction survey • To validate sustainable adoption of technologies with the view of accelerating demand. Are clients buying into the technology as appropriate, aim of the project • To check anticipated project impacts/ monitor unintended impacts • Cost

14. MEDA’s MSC Application Model

15. MEDA’s MSC application model

16. 4 Results, Benefits and Lessons Learned in integrating MSC in Market Development Programs

17. Results • Stories were compiled and at the MEDA office the number of stories on a particular domain of change were weighed with those considered MSC. In both scenarios increase in income was highest placed impact. • The application of MSC enabled the MEDA Zambia team to establish that the major impact of the intervention i.e getting the SSIT’s out to the farmers was an increase in gross income for farmers.

18. Results continued • The validation of sustainable adoption for most farmers pointed to benefits in terms of finance accumulated versus livelihood indicators that project monitoring activities were focused on mostly e.g health, education, housing

19. Results continued • In project planning terms meaning that the main information that other farmers should be furnished with to accelerate demand should be the potential for them to increase their business income.

20. Benefits It is not a stand alone technique; • Hit the nail on the head-get to the issue directly • Its never too late too use it • Gives a richer description of the context in which a positive change is occurring, beyond the data from the quantitative case studies • Quality is added to the data collected as the stories are collected in the beneficiaries voice(verbatim) • It is more flexible as it is not based on predefined indicators, you can measure unpredicted outcomes

21. Lessons Learned • MSC can be adapted to measure specific qualitative aspects of a multi faceted project • Not all MSC domains will vary from indicators but can still give a closer look at outcomes • During story telling other demand side players learnt from their peers on best methods to employ the adopted technology • The stories can be furnished with quantitative data for depth • If the MSC is conducted in a target group with participants in the same areas they serve as a verification board to measure the truth in a story.

22. 5 Conclusion

23. Conclusion • Market development indicators are often a challenge to monitor due to the complexity of the market and the qualitative nature they take. • MSC as an evaluation approach can be applied to capture specific qualitative aspects of change for a multi faceted intervention. • For MEDA its application enabled the project validate sustainable adoption of SITT’s as it related to income increase. • There is proof farmers have not only bought the SITT, but have spotted benefits of it and realized its need (accelerating demand) • It was not complicated to apply and a larger number of farmers were reached to gather this specific vital information • MEDA is sure what message should be fed to farmers to accelerate demand as the supply chain is being developed even if it was not what was prioritised initially.