Developing a participatory impact assessment approach and action research culture within a communication for social chan

1. Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal June Lennie, Andrew Skuse, Bikash Koirala, Nirmal Rijal and Michael Wilmore Paper presented at the Perspectives on Impact Evaluation conference, Cairo, Egypt, 2 April 2009

2. Acknowledgement The 1st Chief Investigator of the Assessing Communication for Social Change project is Assoc Prof Jo Tacchi from Queensland University of Technology, Brisbane, Australia

3. Outline of presentation Benefits of a participatory approach to evaluation capacity building (ECB) Potential of ethnographic action research (EAR) for creating a research culture Overview of the Assessing Communication for Social Change (AC4SC) project

4. Outline of presentation (cont.) Context of the AC4SC project Strategies used to develop stronger M&E systems Challenges and issues that emerged Key outcomes and learnings

5. Benefits of a participatory approach to ECB Process is flexible and responsive to change Uses inclusive processes that aim to be empowering Can generate trust, and a shared vision and objectives Can enable better decision making, program improvement and sustainability Can be a cost-effective method

6. Potential of ethnographic action research (EAR) Used to build the research and evaluation capacity of media initiatives Aims to be a rigorous, holistic approach Combines participatory techniques with an ethnographic approach in an ongoing action research framework

7. Potential of EAR (cont.) Uses methods such as participant observation, in-depth interviews, and immersion in research sites Focus is on community capacities to engage and take action to facilitate social change EAR researchers aim to foster a research culture

8. The Assessing Communication for Social Change project: partners and funders

9. Overview of AC4SC Aims to design and trial a participatory methodology for evaluating the impacts of two popular Communication for Social Change (CFSC) radio programs made by Equal Access – Nepal (EAN), and related outreach activities Aims to develop locally-defined indicators of social change Is developing and adapting the EAR approach and setting up systems that integrate an EAR-like approach into the operations of EAN

10. Radio programs the impact assessment is focussed on Saathi Sanga Man Ka Kura (SSMK) (Chatting with my best friend) - youth-oriented; mixes drama, information and listeners feedback. Aims to improve life skills and empower young people Naya Nepal (New Nepal) - aimed at peace-building and reconciliation

11. SSMK listener club members at World AIDS Day rally

12. Evaluation context in Nepal Lack of community of practice in evaluation Lack of M&E understanding at all levels of a program Data manipulated to please donors Lack of understanding by donors of the complexity of measuring social change impacts, and the time and resources required

13. Contextual challenges and issues in Nepal Extreme poverty Communication and travel problems Gender and power issues – low status of women, caste-based culture Sensitivity of issues such as sexuality & reproductive health Political instability – frequent strikes

14. Maoist demonstration in Bharatphur, Central Terai, during AC4SC field visit, September 2007

15. Context in Equal Access – Nepal before AC4SC project Lack of leadership in evaluation Poor feedback systems; use of unrealistic indicators of program effectiveness & impact; uncoordinated approach to data analysis Reliance on ‘success stories’ from listener letters and other feedback Limited training in all aspects of M&E and EAR Unable to demonstrate the usefulness of EAR Staff wanted to improve their skills in a wide range of M&E methods

16. Strategies to develop stronger M&E systems Critically reviewed existing M&E capacities, needs, systems and practices Conducted relationship and team building exercises Developed better data management, reporting and feedback systems

17. Strategies to develop stronger M&E systems (cont.) Held capacity-building workshops with EAN staff and practiced skills with community groups Identified and trained eight community researchers (CRs) Developed basic manual and toolkit for CRs

18. Some key challenges and issues Initial methods and framework seen as too complex - led to confusion; problems dealing with large volumes of qualitative data Resistance from SSMK team to changing its M&E systems; lack of cooperation with M&E team Human resources and implementation problems

19. Key project outcomes Impact assessment methodology streamlined and immediate re outcomes More effective M&E systems Collaborative relationships between M&E and content team improved Content team more aware of importance of M&E and enthusiastic about learning from CR data

20. Some learnings from AC4SC project Building capacities and setting up effective M&E systems takes time and effort but is essential before good quality data can emerge Need to understand organisational culture before collaborative relationships can be improved Can train community members to collect participatory data but they need continuous mentoring & good feedback systems Regular critical reviews needed to identify & address problems & encourage critical approach

21. Some learnings about creating an evaluation culture in development organisations Involve all relevant staff in ECB activities ECB initiatives need to be flexible and open to change Communication and feedback systems need to be effective and timely Benefits of evaluation needs to be actively demonstrated to program staff Keep participatory methodologies simple and practical, esp. in initial phase, but ensure that methods and data are rigorous and trustworthy

22. Thank you! Questions? Comments? For more information contact June Lennie at j.lennie@bigpond.net.au or Bikash Koirala at bikashkoirala@gmail.com For more information about Equal Access go to http://www.equalaccess.org