Evaluations

1. Why/when do we have to do them? What does a good one look like? Evaluations L-MEP Liberia - Monitoring & Evaluation Program

2. Understand monitoring vs. evaluation Know some of the evaluation triggers Use the 3 key questions for evaluation design Assess an evaluation L-MEP Liberia - Monitoring & Evaluation Program Evaluation: Objectives

3. L-MEP Liberia - Monitoring & Evaluation Program Let’s start with the word: E*val*u*ate What does it mean? • 1. To ascertain or fix the value or • worth of; • 2. To examine and carefully judge. • Source: The American Heritage Dictionary

4. L-MEP Liberia - Monitoring & Evaluation Program ADS 200.6 Definition. (p. 61) Evaluation is the systematic collection of information about the characteristics and outcomes of Assistance Objectives, projects or activities in order to make judgments, improve effectiveness, and/or inform decisions about current and future programming. See Evaluation Policy in Tab 11

5. L-MEP Liberia - Monitoring & Evaluation Program Relationship: Monitoring to Evaluation Monitoring (or Performance Measurement) Evaluation What happened? Why? Why Not? Continuous Periodic Planned results Planned & Unplanned Results Predetermined Wide variety indicators of measures Usually accepts Usually free to design challenge design

6. L-MEP Liberia - Monitoring & Evaluation Program Evaluation Goes Beyond What Monitoring Can Tell Us Evaluation Goes Beyond What Monitoring Can Tell Us Why were results above/below expectations, and what isfacilitating/impeding activity success? Are the planned results the only results the activity is producing? Are there any side-effects? Did the activity actually “cause” the results we see, or were other factors responsible? Is there any evidence to suggest that results achieved by the activity will be sustained?

7. L-MEP Liberia - Monitoring & Evaluation Program USAID “Triggers” for Evaluation: A key management decision is required, and there is inadequate information;  Performance information indicates an unexpected result (positive or negative) that should be explained (such as gender differential results);  Customer, partner, or other informed feedback suggests that there are implementation problems, unmet needs, or unintended consequences or impacts;

8. L-MEP Liberia - Monitoring & Evaluation Program USAID “Triggers” for Evaluation:  Issues of sustainability, cost-effectiveness, or relevance arise;  The validity of Results Framework hypotheses or critical assumptions is questioned;  Project Reviews have identified key questions that need to be answered or that need consensus.  Extracting lessons is important for the benefit of other on-going or future programs, here or elsewhere.

9. L-MEP Liberia - Monitoring & Evaluation Program What questions need answers and when? Source: ADS, 203 MSI Management Systems International

10. L-MEP Liberia - Monitoring & Evaluation Program What Do We Want to Know – and When? What questions need answers and when? Mid-Term (Formative) Evaluations Process questions Initial results Final (Summative) Evaluations Intended End of Project Results Plans for Sustaining the Activity Ex Post Facto (Impact) Evaluations Long term impact Sustainability

11. Impact Questions Design Questions Cost Questions L-MEP Liberia - Monitoring & Evaluation Program One useful technique for controlling the total number of evaluation questions is clustering.  Put a limit on the total number of questions.  Cluster similar questions together.  Eliminate redundancy.  Put the remaining questions in priority order.  Drop the least important questions in each cluster.

12. L-MEP Liberia - Monitoring & Evaluation Program Evaluation Design The term design refers to the conceptual structure we use to carry out an evaluation. An evaluation’s design or plan responds to: -- The evaluation’s purpose, audience and intended uses. -- The evaluation questions it must address. -- The evidence required to answer those questions.

13. What are the evaluation questions? What data sources can respond to the questions? (In reality out there?) How do I get them to tell the truth with evidence? (Design) L-MEP Liberia - Monitoring & Evaluation Program The 3 key questions for evaluation design

14. Read III. Project Description (p.6-7) Discuss for common understanding Read IV. Purpose of the Evaluation (p.7) What are the evaluation questions? Read 1st paragraph of V. Methodology What are the data sources? (p.8) Paragraph 1 L-MEP Liberia - Monitoring & Evaluation Program Let’s look at the case

15. L-MEP Liberia - Monitoring & Evaluation Program

16. L-MEP Liberia - Monitoring & Evaluation Program Prepare a Detailed Data Collection Worksheet Timing & LOE Data Collection Method Scale of Application Instruments Q’s 1,2, 3,6 Checklist: Strategy, Activities Org. chart, Impact Proposal, Reports, pamphlets ½ day Mon 1pd Document Review ½ day Mon 1 pd Checklist: Doc. Review products & Impact Monitoring Director, 4 Engineers, Trainer 1,2, 3,6 Project Personnel interviews Standard Questionnaire for Qs 1-5 40+ in 6 mines in Oruro & Atocha Oruro 2 Atocha 2 Tu-Th 8pd Miner interviews 1-5 Courseattendance Mine inspection 1 course and 2 mines Atocha Th- Fri 1 pd Observation checklists 2 Standard questionnaire 1 hr. ea. M-F, 0 pd +/- 10 people 5 organizations Support Org. interviews 1,5

17. Findings should be factual representations of the data collected and relationships among them Everybody review the VI. Findings top of p. 10 on CEPROMIN Project Organization Then divide your group into ½ and one sub-group reads Miner interviews Oruro p. 10-12 One sub-group reads Miner interviews Atocha p. 12 - 15 L-MEP Liberia - Monitoring & Evaluation Program Findings

18. These should be evaluator judgments backed up by data in the findings. One sub-group reads VII Conclusions/ Recommendations for Qs 1. and 2. (p.15- 18) The other sub-group reads for Qs 3 – 6. (p.18 – 20) Discuss: Are the questions really answered? Is there sufficient evidence to support the answers? L-MEP Liberia - Monitoring & Evaluation Program Conclusions/Recommendations

19. Decisions Recommendations Interpretations/ C o n c l u s i o n s F i n d i n g s D A T A DATA COLLECION METHODS E V A L U A T I O N D E S I G N EVALUATION QUESTIONS L-MEP Liberia - Monitoring & Evaluation Program Evaluation Pyramid

20. Program Or Project Description Evaluation Methods (optional) Description of Deliverables L-MEP Evaluation Schedule Purpose Of the Evaluation Liberia - Monitoring & Evaluation Program Evaluation Budget Evaluation Questions Key Substantive Sections Main Contractual Sections Key Elements of an Evaluation SOW Key Elements of an Evaluation SOW

21. L-MEP Liberia - Monitoring & Evaluation Program Scope of Work in Work Book

22. The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. YES___ NO ___ Evaluation reports shall address all evaluation questions included in the scope of work. YES___ NO ___ The evaluation report should include the scope of work as an annex. YES___ NO ___ Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. YES___ NO ___ Evaluation findings will assess outcomes and impact on males and females. YES___ NO ___ L-MEP Liberia - Monitoring & Evaluation Program Task XIII: Rate Evaluation Using Policy Checklist

23. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.). YES___ NO ___ Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. YES___ NO ___ Sources of information need to be properly identified and listed in an annex. YES___ NO ___ Recommendations need to be supported by a specific set of findings. YES___ NO ___ Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. YES___ NO ___ L-MEP Liberia - Monitoring & Evaluation Program Task XIII: Rate Evaluation Using Policy Checklist

24. Monitoring and Evaluation Evaluation Triggers Evaluation Questions 3 Key Questions for Evaluation Design Evaluation Pyramid Elements of an Evaluation SOW Evaluation Checklist We do not know what we do not know (even if we think we do.) L-MEP Liberia - Monitoring & Evaluation Program Session 8 Summary