Volunteer Michigan: An Introduction to Evaluation By Robert W. Kahle, Ph.D.

1. Volunteer Michigan: An Introduction to Evaluation By Robert W. Kahle, Ph.D. Kahle Research Solutions www.KahleResearch.com March 22, 2011

2. Background & Introduction • Bob Kahle, Kahle Research Solutions • Sociologist with a Ph.D. from Wayne State • Designer and analyst of the Mentor Michigan Census now in its eighth wave. • Involved with multiple program level evaluations, including Volunteer Michigan, Prisoner Reentry Initiative in Detroit and Mentoring Programs. • Integrate evaluation analysis with strategic planning facilitation so that new strategic plans are developed with metrics to measure plan achievement. Kahle Research Solutions Inc.

3. Learning Objectives • Be able to define evaluation and distinguish • Formative (Needs Assessments, Process Evaluation) • Summative (Outcome, Impact, Cost Benefit) • Understand an overview of four broad evaluation strategies • Understand prerequisites for conducting evaluation • The logic model and theory of change • Data collection capacity • Program maturity, capacity and commitment • Be able to describe benefits and challenges of evaluation • Understand what you miss if you do not evaluate Kahle Research Solutions Inc.

4. Evaluation Defined • Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of social intervention programs (Rossi and Freeman, 1989). Kahle Research Solutions Inc.

5. Evaluation Defined • Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of social intervention programs (Rossi and Freeman, 1989). • Evaluation is the the systematic acquisition and assessment of information to provide useful feedback to some object (Trochim, 2006). Kahle Research Solutions Inc.

6. Evaluation Defined • Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of social intervention programs (Rossi and Freeman, 1989). • Evaluation is the the systematic acquisition and assessment of information to provide useful feedback to some object (Trochim, 2006). • Volunteer program evaluation is the ongoing process of collecting and analyzing data to provide the organization with adequate knowledge on which to base programmatic and organizational decisions (HandsOn Network). Kahle Research Solutions Inc.

7. Process Compared to Outcome Evaluation • Process Evaluation seeks to answer questions like: • What is the program experience for participants? • Were they satisfied with their experience? • Did the program recruit appropriate volunteers? How Many? Who were they (demographics)? • Were volunteers screened, trained and supervised? • What changes need to be made to the program to improve the experience for participants? • To more effectively address the specific problems? • How does the program actually work (compared to how it was designed?) • How can it be improved? Kahle Research Solutions Inc.

8. Process Compared to Outcome Evaluation • Outcome Evaluation seeks to answer questions like: • Did the program have the desired effect on the participants and the problem you seek to address? • Does the program work as intended? • What changed as a result of these volunteer efforts? • Can we observe and document the change? • Did we change what we intended to change? • If so, by how much? • Were there unintended changes or consequences? Kahle Research Solutions Inc.

9. Process Evaluation Questions Your Turn • What are some of the key process evaluation questions specific to your Volunteer Michigan efforts? Please type one or two process questions into the chat box on your screen. Kahle Research Solutions Inc.

10. Outcome Evaluation Questions Your Turn • What are some key outcome evaluation questions specific to your Volunteer Michigan initiative? Please type one or two outcome evaluation questions into the chat box on your screen. Kahle Research Solutions Inc.

11. Why should you care about evaluation? • Funders and other key constituents (board members, partners, community leaders) deserve (require) to know if and how the program works. • Good process information can help you improve your program and overall organization resulting in better service to target audience. • Lessons learned can be shared with others to help them strengthen their programs and organizations. Kahle Research Solutions Inc.

12. Why is Evaluation Important? Kahle Research Solutions Inc.

13. Choose not to Evaluate and… …accept that the status quo is good enough. …be forced to rely on anecdotes and opinions, rather than data and facts, when promoting, marketing and raising funds your program. …risk being perceived as “old-school”, unsophisticated. …be forced to rely on gut instinct to make decisions and plans. …complete the program or project and be neither better prepared nor smarter for the next one. Kahle Research Solutions Inc. 12

14. Key Constituents Your Turn • Other than MCSC and VCM, who are your key constituents? • What do your key constituentswant to know about your Volunteer MI efforts? Why? Please type your responses into the chat box on your screen. Kahle Research Solutions Inc.

15. Four General Strategies (Source: Trochim, 2006) Kahle Research Solutions Inc.

16. Four General Strategies Emphases: Impartiality, accuracy, objectivity, validity. Methods: Experimental designs, Randomized Control Trials (RCT), econometric, cost-benefit. (Source: Trochim, 2006) Kahle Research Solutions Inc.

17. Four General Strategies Emphases: Comprehensiveness, continuous process improvement, improved mgt. decision making, leadership. Methods: PERT = Program Evaluation Review Technique. Logical Framework UTOS = Units, Treatments, Observations, Settings (Source: Trochim, 2006) Kahle Research Solutions Inc.

18. Four General Strategies Emphases: Understand context, subjective human experience, culture, environment. Methods: Personal Interviews, Observation, Focus Groups, Grounded Theory (Source: Trochim, 2006) Kahle Research Solutions Inc.

19. Four General Strategies Emphases: Understand and enhance participant experience. Methods: Personal Interviews, focus groups, customer/participant satisfaction surveys, analyses of repeat participation, loyalty to program/organization (Source: Trochim, 2006) Kahle Research Solutions Inc.

20. Which strategy is best for your Volunteer Michigan Initiative? Your Turn Please type your response into the chat box on your screen. Kahle Research Solutions Inc.

21. Prerequisites to Conducting Evaluation Kahle Research Solutions Inc.

22. Five Steps to Building an Evaluation Foundation Kahle Research Solutions Inc.

23. Develop an Evaluation Team • Volunteers • Paid staff • Person managing volunteer efforts • Outside evaluation experts • Organizational decision makers • Community members • Board members • Clients or those served by your organization • Students and interns Kahle Research Solutions Inc.

24. Other Ways to Build Organizational Capacity for Evaluation • Partner with your local college or university • Recruit skills-based volunteers • Network with other nonprofits • Continue to learn (read, research, take additional training, etc.) Kahle Research Solutions Inc.

25. Define Your Purpose • Why are you conducting an evaluation? • What decisions do you want or need to make upon completion of the evaluation? • Which strategy might be best for your organization? • The purpose determines • Audience • Methods • Data to be collected Note: This is the most important part of the process! Kahle Research Solutions Inc.

26. Align with Organizational Plan and Allocate Resources • Connect evaluation efforts with organizational planning. • Link objectives from strategic plan to key metrics • Always put evaluation efforts in context. • The level of evaluation and the resources allocated to the project need to be in sync. • Ballpark estimate: evaluation should be about 10-15% of overall program resource allocation. Kahle Research Solutions Inc.

27. Identify Stakeholders and Audiences • A stakeholder is anyone who has a vested interest in the success (or failure) of your program, service, organization, corporation. “Those groups without whose support the organization would cease to exist”(as defined by the Boston University’s Work and Family Research Network.) Kahle Research Solutions Inc.

28. Ready for Evaluation? • Is program history and model documented and recorded? • Are resources in-place? • Have all stakeholders been identified? • Are program staff in-place (job descriptions, roles identified)? • Are program goals and objectives clearly specified? (Logic model) • Does the organization have the capacity to collect relevant data? How is the data going to be collected? • Are program leaders willing to listen, change as a result of evaluation research? Kahle Research Solutions Inc.

29. How ready are you for Evaluation? Your Turn • We are on it! • Feeling good about evaluation, but still a way to go! • Have started, but everything else is more important! • When will this session be over? I have real work to do. In the chat box on your screen, please type the letter associated with the phrase above that best describes your readiness for evaluation. Kahle Research Solutions Inc.

30. What is a Logic Model? • An explicit statement of: • Resources • Actions • Expected Outputs • Expected Outcomes • Short term • Longer term Kahle Research Solutions Inc.

31. Logic Model • A diagram demonstrating the materials and activities that will produce the desired programmatic results. • There area at least five elements of a logic model: • Inputs • Processes or activities • Outputs • Outcomes • Impact • Logic models help you follow a logic chain so that if you add these inputs and engage in these activities you will get these outputs and outcomes Kahle Research Solutions Inc.

32. Definitions • Inputs - Resources used to meet your volunteer program goals. Examples: time, money, and materials. • Activities - Actions that the program engages in to meet the needs of those they serve or their constituents. • Outputs - The “bean counting” results of your efforts. These are what you get after you put all your inputs and activities into the program. Examples: Number of community members served, dollars spent, number of volunteers, etc. They answer the “what” question. • Outcomes - Short, intermediate and long-term meaningful changes that occur during, or as a result of, your program. Outcomes answer the question “so what.?” They are changes to knowledge, skills, behaviors, values and conditions. Kahle Research Solutions Inc.

33. Outcomes-Based Evaluation Kahle Research Solutions Inc.

34. Theory of Change • Similar to Logic Modeling but: • More emphasis on making assumptions explicit • Starting with the end in mind • Depicting causal pathways and all necessary ingredients to achieve END RESULTS Kahle Research Solutions Inc.

35. Source: Anderson, A. The Community Builders Approach to a Theory of Change. The Aspen Institute. Kahle Research Solutions Inc.

36. Challenges and Barriers To Evaluation • Lack of money and staff resources • Concern that spending resources on evaluation diverts resources from service to community • Fear that the evaluation will be “negative” and force change • Fear of complexity of evaluation, data collection, stats • Lack of existing data collection capacity • Inability to start because tasks seem overwhelming and scary Kahle Research Solutions Inc.

37. Kahle Research Solutions Inc.

38. Resources to Learn More • www.cdc.gov/healthyouth/evaluation/resources.htm • American Evaluation Association (www.eval.org) • Kellogg Foundation (Logic Models) (www.WKKF.org) • Theory of Change (www.AckKnowledge.org) and (www.theoryofchange.org) • Local Colleges, Universities and Community Colleges • Search Institute (www.search-institute.org) • Aspen Institute (www.aspeninstitute.org) Kahle Research Solutions Inc.