Indicators and Calculating Coverage indicators

1. Indicators and Calculating Coverage indicators

2. M&E Indicators For Malaria Programs

3. M&E Indicators: Module Objectives At the end of the session, participants will be able to: • Critique indicators • Identify criteria for selection of sound indicators • Understand how indicators are linked to the frameworks covered in the Frameworks Module • Select indicators and complete an Indicator Reference Sheet

4. What is an Indicator? An Indicator is… • a variable • that measures • one aspect of a program/project or health outcome • An appropriate set of indicators includes at least one indicator for each significant aspect of the program or project (i.e. at least one per box in an M&E framework)

5. Anatomy of an Indicator Metric • Proportion of Households with at Least One ITN* • Numerator: Number of households surveyed with at least one ITN. • Denominator: Total number of households surveyed. *An ITN is 1) a factory treated net that does not require any treatment, 2) a pretreated net obtained within the past 12 months, or 3) a net that has been soaked with insecticide within the past 12 months.

6. Common Indicator Metrics • Counts • Number of providers trained • Number of ITNs distributed • Calculations: percentages, rates, ratios • % of facilities with trained provider • Under 5 mortality rate, case fatality rate, annual blood examination rate (ABER) • Index, composite measures • Quality index comprising the sum of scores on six quality outcome indicators • DALY • Wealth index • Thresholds • Presence, absence • Pre-determined level or standard • Cut-off point

7. Characteristics of Good Indicators • Valid:accurate measure of a behavior, practice or task • Reliable: consistently measurable in the same way by different observers • Precise:operationally defined in clear terms • Measurable: quantifiable using available tools and methods • Timely:provides a measurement at time intervals relevant and appropriate in terms of program goals and activities • Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact

8. Characteristics of Good Indicators: Valid Accurate measure of a behavior, practice or task • Indicator measures what it is supposed to measure • Direct measures • Indirect/Proxy measures • Straightforward interpretation: change in value signals a change in focal concept or behavior

9. Validity: Class Activity • Is parasitemia a valid measure of morbidity? • Is fever a valid measure for malaria? • Is parasite testing a valid measure for parasite prevalence? • Is the number of people reached by BCC campaigns a valid measure of malaria knowledge?

10. Characteristics of Good Indicators: Reliable Consistently measurable in the same way by different observers • Types of measurement error • Sampling Error: over-representation of urban populations because access is easier • Non-Sampling Error: survey estimates of bed net use, due to response bias • Subjective Measurement: indicators that ask for personal judgment such as “quality,” “environment” and “progress”

11. Characteristics of Good Indicators: Precise Operationally defined in clear terms Activity: Develop definitions for: • Effective treatment • Population at-risk • Suspected cases of malaria

12. Characteristics of Good Indicators: Measurable Quantifiable using available tools and methods • Are the following indicators measurable? • Number of ITNs distributed • Compliance to antimalarial treatment • Anemia • Parasitemia

13. Characteristics of Good Indicators:Timely Provides a measurement over periods of time of interest with data available for all appropriate intervals • Timeliness Considerations • Reporting schedules • Recall periods • Survey schedules • Length of time over which change can be detected

14. Characteristics of Good Indicators: Programmatically Important Linked to a public health impact or to achieving the objectives needed for impact • Are the following indicators programmatically important? • Example 1: ITN distribution program • Indicator: # ITNs distributed in past quarter • Example 2: Program to increase access to ACTs through community-based health workers • Indicator: Number of ACT sales points with antimalarial drugs

15. Factors to Consider When Selecting Indicators • Link to framework • Programmatic needs/information for decision making • Resources (Time) • External requirements (government, health partner, headquarters) • Data availability • Standardized indicators

16. Operationalizing Indicators Establish exactly how a given concept / behavior will be measured • Precise definition and metric • How the value will be reliably calculated • Anyone using the same data will arrive at exactly the same indicator value • Challenges • Subjective judgment • Local conditions • Unclear yardsticks • Skills of the users

17. Sources of Indicators:Using Pre-Defined Indicators What are some sources for pre-defined indicators? • Past years of the program • Related or similar programs • Lists of global or recommended indicators • Roll Back Malaria. Guidelines for Core Population-based Indicators, 2009 • Global Fund Indicator Guide • Global Fund performance framework

18. Decreases Number of Indicators Increases Indicator Pyramid Global Compare countriesOverview world-wide situation • National/Sub-national • Assess effectiveness of response • Reflect goals/objectives of national/ • sub-national response • District or Facility • Identify progress, problems, and challenges

19. Indicator Matrix

20. Indicator Reference Sheet Compile detailed documentation for each indicator: • Basic information • Description • Plans for data collection • Plans for data analysis, reporting, and review • Data quality issues • Performance data table (baseline and targets)

21. Indicator Reference Sheet: Example

22. Indicator Strengths & Limitations All indicators have limitations, even those commonly used: • Household spraying, net impregnation: Recall bias this can result in considerable ‘heaping’ of dates • Net Use: self-reporting bias, seasonality of survey may affect net use

23. Indicators: How they link to frameworks

24. Logic Model Indicators • OUTPUT • ITNs sold and distributed • IMPACT • Prevalence of malaria • INPUT • Human and financial resources • ITNs • PROCESS • Establish distribution points for ITNs • OUTCOME • Use of ITNs Indicator: Number of ITNs sold and distributed Indicator: Prevalence of malaria parasite infection Indicator: Proportion of household members who slept under an ITN the previous night

25. Results Framework Indicators IR-1: Improved malaria prevention Number of malaria cases IR-1.1: Access to and coverage by ITNs increased Proportion of household members who slept under an ITN the previous night Proportion of women who received IPT during antenatal care visits during their last pregnancy IR-1.2: Improved coverage of IPTs IR-1.3: IRS coverage increased Proportion of households that received spraying through an IRS campaign within the last 12 months

26. Setting Indicator Targets: Useful Information Sources • Past trends • Client expectations • Donor expectations • Expert opinion • Research findings • What has been accomplished elsewhere • International conventions

27. Common Pitfalls in Indicator Selection • Indicators not linked to program activities • Poorly defined indicators • Indicators that do not currently exist and cannot realistically be collected • Process indicators to measure outcomes & impacts • Indicators that are not very sensitive to change • Too many indicators

28. Pitfalls with Selecting Indicators Indicator not linked to program activities • IR: Expanded access to malaria treatment services • Activities: train providers in current clinical protocols • Inappropriate Indicator: % of facilities with adequate conditions to provide care • Better indicators: # of clinicians trained, % of facilities with a trained provider • The program is not aiming to affect facility conditions, only provider skills.

29. Pitfalls with Selecting Indicators Data needed for indicator not available • Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs • Data issue: Information on stock-outs may not be collected daily • Better indicators: % of service delivery points that had a stock out of drugs at some time during the last quarter • If relying on routine data, indicator definition must depend on how data are collected

30. Pitfalls with Selecting Indicators Indicator does not accurately represent desired outcome • IR: Access to effective treatment among children <5 years old with malaria • Inappropriate Indicators: % of children <5 years old who received ACTs; % of people who received ACTs for malaria infection who are children<5 • Better indicator: % of children <5 years old who were diagnosed with malaria in the past 2 weeks who received ACTs • What does it mean if inappropriate indicators increase? Decrease? Do they reflect the desired program effect?

31. Indicator systems -- How much is enough? Rule of thumb • At least one or two indicators per key activity or result (ideally, from different data sources) • At least one indicator for every core activity (e.g., ITN distribution, IRS, training, BCC) • No more than 8-10 indicators per area of significant program focus • Use a mix of data collection strategies/source

32. Choosing the right number of indicators

33. Good indicators: • Provide information useful for program decision-making • Are consistent with international standards and other reporting requirements, as appropriate • Are defined in clear and unambiguous terms • Are non-directional, “independent” • Have values that are: • Easy to interpret and explain • Precise, valid and reliable measures • Comparable across relevant population groups, geography, other program factors, as needed

34. Albert Einstein Not everything that can be counted counts, and not everything that counts can be counted.

35. Ensure that the indicators are linked to the program goals and are able to measure change Ensure that standard indicators are used to the extent possible Consider the cost and feasibility of data collection and analysis Keep the number of indicators to the minimum and include only those needed for program and management decisions or for reporting Summary: Guiding principles to selecting indicators

36. Calculating and Interpreting Coverage Indicators For Malaria Programs

37. Learning Objectives By the end of the session, participants will be able to: • Identify sources of data for calculating coverage indicators • Estimate denominators for routine coverage estimates • Calculate and interpret coverage indicators from routine data • Use online resources for estimating coverage indicators • Assess the quality of relevant data sources • Reconcile coverage estimates from different data sources

38. ITN Coverage Indicators Proportion of households with at least one ITN/LLIN Proportion of population with access to an ITN within their household Proportion of households with at least one ITN for every two people Proportion of the population/children under 5 years old/pregnant women who slept under an ITN/LLIN the previous night

39. IPTp Coverage Indicator Proportion of women who received two, three and four or more doses of intermittent preventive treatment for malaria during their last pregnancy in the last two years

40. Diagnostics and Treatment Coverage Indicator Proportion of children under 5 years old with fever in the last 2 weeks who had a finger or heel stick Proportion receiving ACTs (or other first line treatment), among children under five years old with fever in the last two weeks who received any antimalarial drugs

41. IRS Coverage Indicators Proportion of Households which Received Spraying through an IRS Campaign within the Last 12 Months Coverage of vector control: Proportion of Households with at least one ITN/LLIN and/or sprayed by IRS in the last 12 months

42. Why Coverage Indicators Are Important Understand how effective program is See if one target group is reached more effectively than another Identify underserved areas/regions

43. Estimating Coverage From Routine Data

44. Indicators for Program: Numerators HMIS and routine reports give information on numerators Numerators: number of houses sprayed with IRS, number of LLIN distributed through antenatal care, number of women receiving at least two doses of SP during antenatal care Denominators: ?

45. Example: Importance of denominator • IPTp Provided to • Town A= 200 women • Town B= 400 women • Town C= 600 women • Number of pregnant women: • Town A= 10,000 • Town B= 30,000 • Town C= 60,000 Numerator Denominator Question: What will be the denominator? Response: Number of women that need IPTp who visited ANC clinics in each town Question: Can we say that Town C has the highest coverage? Please justify your response. Answer: No. We need the denominator for each town

46. Indicators for program: Denominators • Population that are targeted by given intervention • District population • Women of childbearing age • Pregnant women visiting ANC • Children under the age of five • Children under 5 years old who had a fever

47. Population registers Censuses Population projections Population growth rate (r) Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) Net migration rate: inmigration - outmigrants per 1000 population Population growth = rate of natural increase + net migration rate How Do We Get Denominators?

48. Estimating population size • Example: • 300,000 people at census • Growth rate = 3% (0.03), • What is the population after 10 years? • 404,958 people • Use the national statistics office project national and sub-national level • Use UN population, World Bank estimates for national level • Use the official figures and only make projections if they are not available • Where: • P(t) is the population size after t years • P(0) is the population size at the last census • r the annual population growth rate

49. Defining Population at Risk • Mid- term population (Mid-year) • Expresses the population at the middle of the year • Person-time • Estimate of the actual time-at-risk in years, months, or days that all persons contributed to the period/under a particular intervention. • Only possible if individuals are follow-up A group of people who share a characteristic that causes each member to be susceptible to a particular event, such as people living in an endemic area who are exposed to malaria

50. Mid-Year Population vs. Person-Years Person time Individual 5 3/12=0.25 5/12=0.42 Individual 4 1 Individual 3 7/12=0.58 Individual 2 1 10/12=0.83 1 Individual 1 12/12=1 Dec 31 Jan 01 Mid year pop: 1+1+1=3 Person year: 0.25+0.43+0.58+0.83+1=3.08