New frontiers E valuation methods Theory of change Project cycle and risk management

1. New frontiersEvaluation methodsTheory of change Project cycle and risk management Jesper Johnsøn, CMI, U4 Bergen, February 4, 2014

3. Frontiers… Currentplateau – (cross) country-level, bribery, fraud • Evaluations ofspecific reforms • Integration with risk management • Proxy indicators

4. Imbalances and the “missing middle” • Imbalances: • So far, resources have beeninvested in standardiseddiagnostic and advocacytools, not learning and evaluation • Diagnostics: follow trends – indicators, data, monitoring • Evaluation: whatworks – question, design and methods, indicators, data, judgement • Missingmiddle Output Outcome Impact Individual Sector Country Level of results Indicator targets

5. When and how to evaluatecorruptioncosts/anti-corruptioneffects? PROBLEM/ DIAGNOSIS Diagnose corruption risk(doescorruption matter, arepatternschanging?) If yes, prepare for evaluation(how and howmuchdoescorruption matter, howeffective is anti-corruption?) • Logic model: Think throughhowtheinitiative is intended to createresults by contributing to behavioral, organisational, political, or societalchange, expressed in a resultchain or theory of change. • Monitor and evaluate:Workonincreasingtheevaluability of the reform by shaping design, establishingindicators, systematicdatacollection, baselines, comparisongroups EVALUATION QUESTION, DESIGN, METHODS INDICATORS DATA ANALYSIS, COMPARISON, JUDGEMENT, CHANGE

6. Controls • Randomisation • Clear objectives • Indicators • Comparisons • Baselines • Trends Overview of types of evaluations • Mixed methods • Triangulation

7. Code ofConductexample, levelofresults

8. Code ofConductexample, building an impact story

9. Diagnostics, evaluation, risk management and theprojectcycle • Framework for holisticuse of different tools • Shows interlinkagesbetweendiagnostics, risk management and evaluationtools

10. Diagnostics, evaluation, risk management and theprojectcycle • http://prezi.com/osddirkyzzpw/?utm_campaign=share&utm_medium=copy&rc=ex0share

11. The Proxy Challenge

12. Challenges weconfront • Linked, but not inseparable: • Measuringchanges in corruptionlevels • Evaluatingwhether anti-corruptioneffortsaresuccessful • Progress onmeasuringbribery and financialfraud, but two overall indicator problems remaining: • How to measureother types ofcorruption(patronage, conflictofinterest, abuseofpower, etc.) • How to present a measureof overall corruptionlevelsin a country, region, sector, or organisationnot biasedtowardsmeasurable types ofcorruption, and can show trajectoriesofchange

13. Whyproxies? • Good, existing “direct” indicators for bribery and financial fraud (need better application of methods) • Can capture “missing middle” – provide needed innovation for some types of corruption, to capture trajectory of performance, and measure behavioural change “Proxy indicators are alternatives to direct indicators that more directly measure the phenomenon under study but that may be hard to operationalize or require overly costly data collection”

14. Criteria Standard criteria for indicatordevelopment (SMART) • Reflect de facto changes and behavioralchanges • Prioritizesensitivity to context over standardization • Measurespecific types ofcorruption • Ensure thatchanges in corruption trends areattributable to reforms

15. “To assemble a body ofpromisingideas…” • Clearly define the proxy indicator (or indicators) you suggest as a good measure of anti-corruption results • Identify the type of corruption measured by the proxy indicator  • Clarify how the proxy indicator reflects changes in corrupt behaviours • Present ideas for how the validity of the proxy indicator can be tested  • Describe any relevant geographic, sectoral, or institutional context • Explain how the indicator can be combined with other proxy and nonproxy (direct) indicators to obtain a better measurement of overall anti-corruption progress • Comment on the strengths and weaknesses of the proxy indicator and the feasibility of its use by aid agencies, governments, civil society, and others for the purposes of monitoring and reporting