Tools for Creating a Culture of Assessment The CIPP Model and Utilization-Focused Evaluation

1. Tools for Creating a Culture of AssessmentThe CIPP Model and Utilization-Focused Evaluation Yvonne Belanger, Duke University Library Assessment Conference September 25-27, 2006 Charlottesville, VA

2. Key Questions for Libraries • How can we build a culture of evaluation, so that many people contribute to evaluation? • How can we provide a context for evaluation strategies and results? • How can we conduct evaluation that helps with decision making?

3. Culture of Assessment “…organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders.” Lakos et al, 1998-2002

4. Barriers to a Culture of Assessment • Lack of evaluative thinking (at all levels) • Lack of engagement in evaluation • Pseudoevaluations (Stufflebeam, 1999) • Promote a positive or negative view of a program, irrespective of its actual merit and worth

5. Building evaluative thinking: CIPP Model Stufflebeam’s CIPP Model - Context, Input, Process and Product evaluation • Focus: decision-making • Purpose: facilitate rational and continuing decision-making, particularly for programs and services with long-term goals

6. Details of the CIPP Model • CIPP • Context: Environment & Needs • Input: Strategies & Resources • Process: Monitoring implementation • Product: Outcomes - both quality and significance More information at www.wmich.edu/evalctr/pubs

7. CIPP approach recognizes… “All politics are local” – offers a tailored evaluation approach designed to answer locally interesting & useful questions, emphasis is on credibility and usefulness rather than generalizability to other places, times, audiences

8. CIPP View of Institutionalized Evaluation Stufflebeam, OPEN, 2003

9. Advantages of the CIPP Model • Adapts well to carrying out evaluations on any scale (projects, programs, organizations) • An organizing framework, not a lockstep linear process • Sensitive to needs of decision makers • Systems approach – encourages a systems view of projects and programs

10. Building evaluative thinking and engagement: Utilization-focused evaluation approach Taking a utilization-focused approach means asking… • Why is this evaluation being undertaken? • What decisions need to be made with the results? • Who will be most affected by those decisions? • How can we engage those people in the entire evaluation process?

11. Utilization-focused evaluation • Premise – by engaging stakeholders in the entire evaluation process from design to implementation of recommendations • Evaluation addresses questions of greatest importance to those in a position to directly make use of its findings • Reduces the cultural barriers that can inhibit use of results by increasing transparency, empowering stakeholders

12. Another advantage of the Utilization-focused approach • “Process Use” benefits • First described by Patton - ‘ways in which being engaged in the processes of evaluation can be useful quite apart from the findings that may emerge from these processes’ • Four types of Process Use 1. Enhancing shared understandings, especially about results; 2. Supporting and reinforcing the object of the evaluation through intervention-oriented evaluation; 3. Increasing participants’ engagement, sense of ownership 4. Organizational development Patton 1997, pp. 88-91

13. Process Use & Culture of Assessment • Increased capacity to make use of evaluation findings • Know how to use evaluation information – producing better evaluation users in the organization who can effectively “weigh evidence, consider contradictions and inconsistencies, articulate values, and examine assumptions” Patton, 2004, “On Evaluation Use: Evaluative Thinking and Process Use”

14. Example Evaluation of the Duke iPod experiment & Duke Digital Initiative…

15. Summary • Foster a culture of assessment by: • Adopting frameworks that support decision-making • Engaging staff as stakeholders in the entire process of evaluation from design to implementation of recommendations • Leverage the opportunity of Process Use to develop staff and make them more saavy evaluation consumers

16. Final Thoughts… “…evaluation's most important purpose is not to prove, but to improve.” Daniel Stufflebeam (CIPP Model) “Research is aimed at truth. Evaluation is aimed at action.” Michael Quinn Patton (Utilization-focused Evaluation)

17. Thank You! Yvonne BelangerHead, Program Evaluation Academic Technology & Instructional Services Perkins Library Duke University yvonne.belanger@duke.edu

18. References • Stufflebeam, D. (1999). Foundational models for 21st century program evaluation. • Stufflebeam, D. (2003). The CIPP Model for Evaluation: An update, a review of the model’s development, a checklist to guide implementation. Paper read at Oregon Program Evaluators Network Conference, at Portland, OR. http://www.wmich.edu/evalctr/pubs/CIPP-ModelOregon10-03.pdf • Patton, M. Q. (2004). "On evaluation use: Evaluative thinking and process use." The Evaluation Exchange IX(4). • Patton, M. Q. 1997. Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.