Program and policy evaluation

1. Program and policy evaluation PPAS4200 February 1st, 2012

2. Program evaluation “Systematic collection of information about the activities, characteristics, and outcomes of programs to make judgements about program, improve program effectiveness, and or inform decisions about future programming.” M. Patton. 1997. Utilization-focused evaluation. Thousand Oaks: Sage, p. 23.

3. Program evaluation vs Policy evaluation • Program evaluation: assesses activities that provide services • Policy evaluation: assesses policies, programs, products, individuals (their performance)

4. Characteristics of evaluation • Value focus • Fact-value interdependence • Future, present and past orientation • Value duality Source: Dunn, W. 2008. Public Policy Analysis. An Introduction. Upper Saddle River (NJ): Pearson/Prentice Hall, p. 353.

5. The planning-evaluation cycle Source: http://www.socialresearchmethods.net/kb/pecycle.htm

6. Functions of evaluation • Providing reliable and valid information about policy performance • Contributing to clarification and critique of values underlying goals and objectives • May contribute to the application of other policy-analytic methods (such as problem structuring) Source: Dunn, W. 2008. Public Policy Analysis. An Introduction. Upper Saddle River (NJ): Pearson/Prentice Hall, p. 353-4.

7. Traditional stepsof policy evaluation • Specification (what is being measured) • Measurement • Analysis • Recommendation

8. Three traditionaltypes of evaluation • Impact evaluation • Process evaluation • Efficiency evaluation • Cost/benefit analysis

9. Approach to program evaluationDeveloped by the City of OttawaSource: http://ottawa.ca/en/city_hall/funding/toolkit/

10. Four functions of monitoringAdapted from www.policyonline.org – chap. 6 • Compliance of actions With legislative procedures, regulatory agencies, professional bodies, etc. • Resources and services reaching target groups Ex.: are revenues intended for local communities reaching them?

11. Four functions of monitoringAdapted from www.policyonline.org – chap. 6 • Accounting Measuring economic and social changes Ex.: EU education and training policies • Explanation Why outcomes may differ

12. The professionalisationof policy evaluation – ProsAdapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Evaluation 16 (4): 349-369. Credibility and legitimacy of the field Increasing training offerings Enhancing and improving the status and prestige of evaluation

13. The professionalisationof policy evaluation – ProsAdapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Evaluation 16 (4): 349-369 Facilitating the selection of evaluators and improving the quality of evaluations Protecting the public Avoiding problematic/unethical behavior

14. The professionalisationof policy evaluation – ConsAdapted from: Jacob, S., and Y. Boisvert. 2010. “To Be or Not to Be a Profession: Pros, Cons and Challenges for Evaluation.” Evaluation 16 (4): 349-369. Homogenizing evaluation and restricting diversity Reducing training offerings Restricting or blocking access to the profession