CalSERVES: AmeriCorps Programs of Napa County Office of Education

1. CalSERVES: AmeriCorps Programs ofNapa County Office of Education

2. CalSERVES AmeriCorps Programs • After School: Local Direct Service, 34 FT/60 HT • Collective Impacts: Local Direct Service, 20 FT/36 HT Members • PREP: Statewide Capacity Building, 40 FT Members • VIP: Statewide Capacity Building, 100 FT Members

3. What We Did • Develop Evaluation Goals and Questions • Employ an Evaluation Firm that Understands AmeriCorps • Determine the Method of Evaluation • Determine Strategy for Comparison Group • Conduct Analysis • Report Findings • Discuss Program Changes and Next Steps for Research

4. Evaluation questions • Does participation in the CalSERVES VIP program increase organizational capacity to utilize volunteers compared to organizations that did not take part in the program? • Does participation in the CalSERVES VIP program improve organizational capacity concerning volunteer recruitment, training, and retention compared to organizations that did not take part in the program? • Does participation in the CalSERVES VIP programimprove organizational capacity to create and sustain successful volunteer programs compared to organizations that did not take part in the program?

5. VCA Instrument • To measure the effect of program participation on organizational and volunteer capacity over time, data were collected using the Volunteer Capacity Assessment (VCA), a survey that includes three sections with items focused on: organizational capacity, volunteer recruitment, and elements of a successful volunteer program. • The VCA is designed to assess how well a non-profit or educational organization is prepared for recruiting, training, and utilizing volunteers to achieve its mission and goals. The instrument asks organizations to report how well their practices align with the best practices listed on the survey.

6. Propensity score matching • List of Characteristics Used in Propensity Score Matching

7. What the Results showed Average Pre/Mid VCA Score and Average Post VCA Scores, Matched Comparison

8. Conclusion • “Overall, the partner sites that participated in the CalSERVES VIP program reported strong, positive changes over time on the items included in the VCA instrument. Organizations that did not participate in the program reported much smaller (and sometimes negative) changes on the VCA items. Because the matching procedure diminishes the likelihood that factors other than participation in the program influenced the change in reported scores over time, the evidence strongly suggests that the CalSERVES VIP program positively impacts organizations’ capacity concerning volunteer programs.”

9. Challenges • Finding the right evaluation firm. • Willingness of comparison sites to participate. • Resources to fund the evaluation. • Experimental design requirement to reach highest level of scoring on funding request.

10. Successes • Kept costs reasonable using comparison group design. • Able to develop and implement a comparison group design that allowed for the development of some real understanding of impact. • Able to use data to inform the program and determine next steps.

11. Next on our research agenda • Examine the type of organizations (both in size and scope of work) that benefit the most from the CalSERVES VIP program. • Explore the specific types of services that are most beneficial to these organizations.