MONITORING & EVALUATION OF HIV & AIDS PROGRAMS - FEBRUARY 2011

1. MONITORING & EVALUATION OF HIV & AIDS PROGRAMS - FEBRUARY 2011 DEVELOPING M & E PLANS by DR DC TSHIBANGU 6 September 2014

2. Objectives of the Session By the end of the session, participants should be able to: Describe the functionsof an M&E plan Identify the main elements of an M&E plan Describe the process of developing an M&E plan Understand how to go about implementing an M&E plan

3. Overview Definition: What is an M&E Plan? Function: What is the Plan supposed to do? Elements: What comprises an M&E Plan? Standards: What constitutes a good M&E Plan? Developing and implementing an M&E plan Introduction to workshop group work: Developing an M&E plan

4. M&E Plans – Definition Comprehensive document that describes all M&E activities in an M&E system, including: Program objectives, interventions developed to achieve these objectives, & procedures to be implemented to determine whether or not the objectives are met Expected results of the program and how they relate to goals and objectives Data needs, how they will be collected & analyzed Information use, including resources needed to do so How the program will be accountable to stakeholders

5. M&E Plan: Functions States how program will measure achievements Accountability Documents consensus Transparency Responsibility Guides M&E implementation Standardization Coordination Preserves institutional memory An M&E plan is a living document that needs to be adjusted when a program is modified

6. M&E Plan: Elements I. Introduction II. Description of the overall program- including problem statement and framework(s) III. Indicators- including definitions (presented in indicator matrix and/or indicator reference sheets: very detailed!) Data sources and reporting systems (including management/roles and responsibilities) Plans for demonstrating program outcome/impact Plans for dissemination and use of information Analysis of data quality constraints & potential solutions Implementation plan (aka- M&E action plan or road map-should include budget and timeline)

7. Elements of an M&E Plan: Introduction Purpose of the plan Description of how it was developed Stakeholders involved Consensus process

8. Elements of an M&E Plan: Program Description Problem Statement What is the nature of the HIV-related issue being addressed? Conceptual Framework Goal and Objectives What is the ultimate outcome of the program (goal) What are the shorter-term aims (objectives) Program Description: Interventions Geographic scope Target population Duration Logical Framework/Results Framework

9. Elements of an M&E Plan: Indicators Selection of indicators based on: Conceptual and logic frameworks Strategic information needed for decision making at appropriate level (country/state/local) Donor requirements Existing data (really needs to be based in reality) Funding (available and dedicate to M&E) Presented in 2 ways: Indicator Matrix- a table presenting indicators including information on data source, frequency and who is responsible Indicator Reference Sheets- detailed sheets describing each indicator, how to measure it, underlying assumptions & interpretation considerations (may be included as appendices)

10. Elements of an M&E Plan:Data Sources and Reporting Systems Sources of data for indicators Diagram of data collection, processing, analysis, and reporting system Data collection tools Patient records or registers Survey instruments Commodity management forms (e.g., condoms) Others? Management Roles and responsibilities of each group/member of the M&E system Data Flow

11. Elements of an M&E Plan: Plan for demonstrating program outcome/impact A methodology for measuring program impact (the evaluation) Protocols for special studies

12. Elements of an M&E Plan: Dissemination and Use Plan Clearly defined users Databases for information storage Dissemination methods Reports (schedule and audience) Media Speaking events Others?

13. Elements of an M&E Plan: Data Quality Constraints Describe known constraints to data quality and/or system performance and what will be done to address these

14. Elements of an M&E Plan: Implementation Plan Assessment of feasibility to implement plan A detailed work plan for the M&E Plan to include: Each M&E activity (including update of M&E Plan) Timing of each activity Party responsible for each activity Budget necessary for each activity

15. M&E Plan: Standards Utility - serve practical information needs of intended users Feasibility - be realistic, prudent, diplomatic and frugal Propriety - conducted legally, ethically, and with regard to those involved in and affected by the evaluation(s) Accuracy - reveal and convey technically accurate information

16. Developing an M&E Plan: Inputs Authority and mandate to implement the M&E plan Human resources with M&E technical capacity Understanding of the program Commitment to accountability (not only to donors, but to the people whom the interventions are to serve) and program improvement Infrastructure (Technology, space, etc..)

17. Developing an M&E Plan: Process Advocate for the need for M&E Assess strategic information needs Assess existing information systems’ capabilities to address strategic information needs Achieve consensus and commitment among stakeholders on: Indicators and their definitions Reporting structure, roles and responsibilities Establish a common “language” Develop mechanism for M&E plan review Prepare document for final approval for all stakeholders M&E plan should be developed during theinitial stages of program development

18. Developing an M&E Plan: Output A comprehensive document that describes the M&E system and that: Includes all the necessary elements; Has the approval of the governing authority; Has the consensus of primary stakeholders.

19. Implementing the M&E System: Costs How much will it cost? Need to budget: Costs of information systems (costs of data collection, processing, and analyzing) Costs of information dissemination and use Costs of the data quality control system Costs of coordination and capacity building

20. Sample Costs for Selected Data Sources (example from a national system) Source: Rommelmann, et. al. Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania. 2003

21. Developing & Implementing an M&E Plan: Role of the M&E Unit Consensus building – among all stakeholders (donors to program managers to clients) Coordination Data management and manipulation (might includedata entry, data formatting, data analysis and interpretation…) Reporting Data auditing Information dissemination Training and capacity building

22. Consensus Building Tips Make sure everyone understands what you are trying to achieve consensus on Ensure that all stakeholders are involved early in the process Provide stakeholders with the opportunity to provide input and receive feedback Consistently promote the message that M&E provides the means to demonstrate the extent to which a program is achieving its objectives (“It is everyone’s job and to everyone’s benefit’)

23. Developing & Implementing an M&E Plan: Dos and Don’ts Do: Start early Involve stakeholders at all stages in the process Assess current capacity and use what is already available Avoid duplication of data collection and reporting Do not: Collect information that will not be used Underestimate the importance of stakeholder buy-in and ownership at every juncture

24. M&E for Program Improvement- An Iterative Process Indicators Data use Source of data Implementation Data collection methods and tools Human resources and training

25. WHAT IS PLANNING? • Planning is a process concerned with: • Developing and achieving a vision • Understanding some of the wider drivers of change • A process that is flexible in light of these changes • A process that is achievable in light of available resources • A process that is continuously reviewed as part of a learning process • A vision that is, importantly, implementable

26. PRINCIPLES FOR A SUCCESSFUL PLANNING PROCESS Include at least the following four principles: • The problem identified (ensure that the right question is the focus of the planning) • The people involved (ensure that the right people are in the team) • The process used (ensure that this is a learning process that helps key players debate things that matter) • The resources available (it’s essential to match the size of the problem with the resources needed to address it)

27. WHAT IS A PLAN? It is a course of action one intends to follow in order to solve a problem It is a purposeful, detailed outline of actions to allocate resources and coordinate efforts, to achieve desired change (e.g. solve a problem, achieve an objective). It is also regarded as a document that describes a system that links strategic information obtained from various data collection systems to decisions that will improve programs. It ensures that objectives are set to deal with problems and to make best use of available resources 6 September 2014

28. PRACTICALLY WHAT DOES THIS MEAN? • This implies the use of conceptual framework to develop a clear understanding of all the factors likely to positively or negatively influence the problem under investigation • Able to derive few models/scenarios from the conceptual framework at hand

29. CONCEPTUAL FRAMEWORK FOR HIV/AIDS CHALLENGES Individual Characteristics Not Infected Community Characteristics Not Infected Sick Infected Death Infected Programmatic Characteristics

30. THIS CONCEPTUAL FRAMEWORK SHOWS • SOME FACTORS THAT ARE LIKELY TO INFLUENCE THE TRANSMISSION OF HIV • THE OPTIMUM LEVEL OF APPLICATION OF SOME PREVENTION STRATEGIES: 1. Primary Prevention: ABC; VCT; School-based life skill education; PMTCT (infant); BCC. 2. Secondary Prevention: PMTCT (mother); ART 3. Tertiary Prevention: Palliative Treatment/Community- based support and care

31. PURPOSES OF STRATEGIC FRAMEWORK • To provide a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact • To clarify project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends

32. Strategic/Results FrameworkYouth HIV Prevention Program

33. WHO/WHAT DETERMINES THE AREAS OF THE PROGRAM TO BE MEASURED? • The program goals/results indicate the areas in which progress can be expected, hence, the areas in which it can be measured. • How to measure progress in these areas? By selecting appropriate and relevant indicators(country’s goals, objectives, activities, local epidemiology, nature of problem and risk behaviors)

34. WHAT IS THE PLAN MAIN PURPOSE? • To help monitoring and evaluating activities through: • M&E BUY IN • Efficiency – saves time • Effectiveness – doing what we want to do • Data management • Data Quality & Data Use

35. Some References • Adamchak S et al. (2000). A Guide to Monitoring and Evaluating Adolescent Reproductive Health Programs. Focus on Young Adults, Tool Series 5. Washington D.C.: Focus on Young Adults. • Bertrand J et al. (1996). Evaluating Family Planning Programs. The Evaluation Project. • Rommelmann V, P Setel, Y Hemed, H Mponezya, G Angeles, T Boerma (2003). Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania. • Lusthaus, et. al. (1999) Enhancing Organizational Performance: A toolbox for Self-Assessment. • Loos G.P. (1996). Field guide for international health project planners & managers. Janus Publishing Company, London England. • Hadridge P (2004). Strategic approaches to planning health care. Oxford Handbook of Public Health Practice. Oxford University Press.

36. THANK YOU