Program Evaluation Overview

1. Program Evaluation Overview E. D. Bell

2. Introduction • Kapp & Anderson (2010) argue that the program evaluation process has eight steps that are fluid, dynamic, and interdependent, i.e., they are not linear – they influence and shape one another throughout the evaluation process.

3. The Eight Steps • Identify the evaluation question • Conceptualize the evaluation question • Operationalize the evaluation question • Pick and define the method • Select the sample • Collect the data • Analyze and manage the data • Utilize the information to improve program performance

4. Fundamental Notions Kapp & Anderson (2010) make this point The steps of the evaluation process need to be framed with an appreciation of two fundamental notions: (1). The use of information for service improvement is primary, and (2). The context for evaluation is an agency where evaluation must compete with many enterprises for attention. (p. 29)

5. Agency-Based Steps in Evaluation • Given these notions, the agency needs to implement the following steps concurrently with the implementation of the program evaluation: 1. Define the program 2. Support users 3. Implement and report design of the evaluation (Kapp & Anderson, 2010)

6. Ethics and Program Evaluation “The process of conducting research and the evaluation of programs and the outcomes associated with these activities have the potential to be helpful or harmful to individuals, groups, organizations, and the broader society” (Kapp & Anderson, 2010, p. 39).

7. Golden Rule Principle So, if you were a participant in a study, what would you want to know? If you were a program director, how would you want the evaluators to conduct themselves? If the evaluation of your program found a number of faults, how would you want these to be reported? (Kapp & Anderson, 2010, p. 59)

8. Evaluation Ethical Guidelines • Guiding Principles for Evaluators • Systematic inquiry • Competence • Integrity/Honesty • Respect for people • Responsibilities for general public welfare

9. Cultural Competency and Program Evaluation “The process of program evaluation does not take place in a social and cultural vacuum devoid of gender, race, ethnicity, and other consideration” (Kapp & Anderson, 2010, p. 107).

10. Defining Cultural Competency Kapp & Anderson (2010) describe four attributes: 1. Awareness and acceptance – that generates respectful treatment and no prejudgment of abilities, performance, and competency. 2. Self-awareness – that helps you understand how your culture shapes what you see and provides the ability to examine, control, and change assumptions, false beliefs, and stereotypes

11. Defining Cultural Competency (continued) • Dynamics of differences – historical and cultural differences influence how you see and define reality, e.g., see Emmett Till. This story that I saw in Jet magazine as a young person still influences how I see and understand reality. Click on the link “The Picture of Emmett Till”. • Knowledge of individual cultures – of the stakeholders who are involved with the program being evaluated.

12. Defining Cultural Competency (continued) 5. Adaption of skills – The evaluator must be able to determine if her or his methods are culturally appropriate and respectful of the persons and programs that are being evaluated.

13. Designing a Culturally Responsive Evaluation Plan • Review Kapp & Anderson (2010) pp. 111 – 131.

14. Define the Program “With most evaluation projects, one of the first things to address is the way the program is supposed to operate” ( Kapp & Anderson, 2010, p. 133). • Develop the logic model of how the program is supposed to work (Program Development and Evaluation, University of Wisconsin – Extension for resource tools)

15. Developing the Initial Logic Model • Interview Stakeholders • Board of Directors • Clients • Community Partners • Funders • Management • Staff (Kapp & Anderson, 2010).

16. Developing the Initial Logic Model (continued) • Use Logic Model Questionnaire (Kapp & Anderson, 2010, p. 139) • Given the first draft of the logic model for the agency what areas of the model require additional information, i.e., what are the potential evaluation questions. • Select one of the areas as the evaluation question for your evaluation plan