1 / 34

Guidance document on ex-ante evaluation

This document provides guidance on ex-ante evaluation, focusing on the approach, differences between ESF/ERDF, program strategy, indicators, monitoring and evaluation, and financial allocation consistency.

mupdegraff
Download Presentation

Guidance document on ex-ante evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Guidance document on ex-ante evaluation 15 March 2012 Joint presentation to the ERDF and ESF Evaluation Partnership Meeting

  2. Ex ante evaluation: our approach What is different? Common Regulation & multi-fund programmes Stronger focus on results / Europe 2020 strategy More detailed Article 48 Links with Partnership Contracts, SEA Our common approach: Stick to Regulation (article 48 and related articles)-as proposed by Commission Focus on ex ante evaluators' role When necessary, give examples of differences between ESF/ERDF

  3. Ex ante evaluation Guidance • Programme strategy Q&A • Indicators, monitoring and evaluation • Consistency of financial allocation • Contribution to Europe 2020 Q&A • Strategic Environmental Assessment • Evaluation Process Q&A

  4. Programme strategy • Consistency of programme objectives • Coherence • Horizontal principles • Linkage between supported actions, expected outputs and results Q&A

  5. Programme strategy – key reference points Art. 24(1) CPR: • Europe 2020 strategy • Common Strategic Framework • Partnership Contract Art. 87(2)(a)(i) CPR: • Country specific recommendations • national and regional needs

  6. Consistency of programme objectives • Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme. • Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.

  7. Step 1: Challenges & needs (1) • National & regional programmes: the evaluators should primarily base their assessment on the NRP, country-specific recommendations, European semester analyses • In addition: • National ERDF sectoral programmes: specific sectoral analysis & territorial priorities • Regional programmes: specific regional situation and needs • Specific challenges of sub-regional or functional areas or of specific target groups • Horizontal principles (Art. 7 & 8 CPR) Justification of specific regional challenges and needs

  8. Step 1: Challenges & needs (2) • The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract. • The evaluators should also base their assessment on the Common Strategic Framework. • They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.

  9. Step 2: Programme objectives thematic objectives investment priorities Specific objectives • While appraising the consistent translation of the identified challenge & needs into programme objectives, the evaluators should focus on: • Their appropriate weight in the investment priorities • Precise formulation of specific objectives • Rationale and evidence for omitting major challenges or needs • The evaluators may recommend complementary analyses

  10. Coherence Internal coherence The evaluators should assess: • Relationship between specific objectives • Coordination mechanisms in multifund programmes External coherence The evaluators should examine: • Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programme • Planned integrated territorial approaches

  11. Horizontal principles • The evaluators should assess the following elements under each principle (Art. 87(3) CPR): • Equality between men an women • Arrangements to integrate the gender perspective at the level of the programme and operations • Contribution to its promotion • Anti-discrimination • Specific actions in preparation and implementation of programme • Sustainable development • Specific actions in the selection of operations Opinion of national equality bodies

  12. Linkage between outputs and results • Clear intervention logic: condition for good programming and evaluations! • Recommendation: use a logical framework for each priority axis or investment priority • The evaluator should examine: - What is the expected change? • How will outputs contribute to intended results? • Will the proposed actions effectively lead to these outputs? • What other factors could influence the expected results? • Would evidence suggest other approaches?

  13. Linkage between outputs and results • Are the planned form of support the most effective? What is the rationale? (grants, prizes, repayable assistance and financial instruments and a combination: Art. 56) • Will the actions effectively meet the needs of specific territories or target groups?

  14. Indicators monitoring and evaluation • Relevance and clarity of proposed indicators • Baseline and target values • Suitability of milestones • Administrative capacity • Data collection • Evaluation • Consistency of financial allocation • Contribution to Europe 2020 Q&A

  15. Indicators, monitoring and evaluation Art. 24 CPR - For each priority axis, indicators to assess progress towards achievement of objectives - Basis for monitoring, evaluation, review of performance • Increased focus on results: role of well designed indicators and evaluations • New focus on territorial integrated approaches: reflected in monitoring and evaluations where relevant • Common indicators compulsory

  16. Indicators, monitoring and evaluation • Relevance of indicators • Output indicators reflect the main operations • Result indicators are responsive to the policy (values evolve with development of operations) • Result indicators reflect the most important intended change • Common indicators - ERDF: used where relevant to operations and specific objectives / ESF: monitored under all investment priorities • Results may be close to the intervention, short or long term: depends on the intervention logic, on the intended change

  17. Indicators, monitoring and evaluation

  18. Indicators, monitoring and evaluation • Clarity of indicators • Unequivoqual title, clear definition • Normative interpretation • Robustness and transparence • Robustness: reliability of data source, representativeness of samples (survey) • Data sources for baselines and target values + definitions are publicly available • See ex ante conditionality (Annex IV CPR)

  19. Indicators, monitoring and evaluation Baseline and target values • Availability of baseline data for result indicators? • latest available data (ERDF and ESF Regulations) • advice on sources and methods to collect data when necessary, or research to establish baselines • Realistic targets? • Compared to actions and form of support, financial allocations • For output indicators, compared to references (previous programmes or sectoral norms) • For result indicators, taking into account other factors

  20. Indicators, monitoring and evaluation • Performance framework: a subset of indicators for each priority axis • Suitability of milestones • Do they capture essential information on the progress of a priority? • Can they be achieved at the review points? • Will data be available for 2017/2019 progress reports?

  21. Administrative capacity • The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in place • Including: Have possible previous bottlenecks at the level of MA and IB been tackled? • The evaluator may advice on how to address any issues

  22. Data collection The ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they: • Respond to all requirements for monitoring • including: is data available on time, in the required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc. • Provide the data needed for evaluations • including: data to i.a. assess effectiveness, efficiency and impact? What are the mechanisms to ensure good quality data? Etc.

  23. Evaluation The ex ante evaluator may advice: • On possible evaluation needs for on-going evaluations • On evaluation methodologies and resulting data needs • On setting up the evaluation plan

  24. Consistency of financial allocation • The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of support • The evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches

  25. Contribution to Europe 2020 • Taking account of the proportionality principle and against the background of national and regional needs • the ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.

  26. Process • Strategic Environmental Assessment • Timing • Interactive and iterative process • Partnership and multi-level governance • Evaluation methods and financing • Final report and publication • An independent process • What requirements for evaluators? Q&A

  27. Strategic Environmental Assessment • A new link with Ex ante: "shall incorporate, where appropriate, the requirements for SEA…" - "Where appropriate": direct impact on environment • "Incorporate": does not mean that the ex ante evaluator has to do SEA • "Requirements": environmental report, consultations, information, monitoring • How to incorporate SEA: still under discussion

  28. Timing • Ex ante evaluation & Partnership contracts PC includes (Art. 14): • Summary analysis of ex ante evaluations justifying selection of the thematic objectives and the indicative allocation of Funds • Summary of main results expected for each thematic objective • Developping ToR … now - PC and OPs adopted Oct.-Dec. 2013 - 6 months negotiations: draft PC & OP April-June 2013 • Programme elaboration 6-8 months: starts Aug.-Oct. 2012 • Calls for tender launched mid 2012

  29. Interactive and iterative process Work in stages Timing and interaction Steering group 29

  30. Partnership and multilevel governance Art. 5 CPR • Strengthening with the objective of: • Increasing legitimacy • Build on wider expertise and knowledge • Ensure collective commitment and shared understanding of expected results • The evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluation • The evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership” 30

  31. Evaluation methods and financing • Mix of methods: - TBE: literature review, workshops, interviews, focus groups, peer reviews • Involve partners in programme design • Cost: - number of evaluation questions • methods • Extra tasks (quantification of baselines, complementary analyses)

  32. Final report and publication • Submitted with the OP to the Commission - Main evaluation methods used - Changes and improvements to programme - Final assessement of OP - Strategic Environmental Assessment • Executive summary (+ in English) • Made available to citizens

  33. An independent process Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation" No hierarchical link with the entity responsible for implementation: internal expert may contribute Balance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence

  34. What requirements for evaluators? • Knowledge of EU and national documents • Deep knowledge of context • Situation of areas or target groups with specific needs • Situation of economic sectors - Able to judge quality of socio-economic analysis • Experience in evaluations • Data needs for future impact evaluations • Relevance of indicators within the intervention logic (thematic evaluations)

More Related