1 / 50

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs. If you have not already done so, please connect to the audio portion of this Webinar: 1. Dial 1-877-668-4490 2. Enter the access code 622 071 435 #

zeno
Download Presentation

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio portion of this Webinar: 1. Dial 1-877-668-4490 2. Enter the access code 622 071 435 # 3. Enter your unique participant ID number displayed on your screen OR choose the option to let the program call you back!!!

  2. Housekeeping • Phones are muted • Ask questions by sending a chat message to “All Participants” using the chat panel on the right side of your screen

  3. Who’s on the Webinar 175 Attendees Representing Programs and Organizations from 44* States

  4. Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs Outcomes You will: • Learn about Performance Measurement for Veteran and Military Family Programs (VMFP) • Discover methods to measure outputs and outcomes with available tools • Learn about VMFP data sources to help measure outputs and outcomes • Explore best practice examples in measuring outcomes from Equal Justice Works

  5. Hello and Welcome from CNCS! • Koby Langley • Corporation for National and Community Service • Webinar hosts: Education Northwest

  6. The Serve America Act’s Influence on CNCS VMF Performance Measures Employment *Service to VMF *Creating Service/ Volunteer Opportunities for VMF Employment Disaster Preparedness Youth Mentoring Access to Benefits Education Health Transportation Education and Certification Wellness and Other Support Services * Overarching activities of programs in SAA *Community Coordination Specific activities Mentioned in SAA

  7. Session Presenters Dr. Chris Spera, CNCS Director of Research and Evaluation Kathryn Gravely, Program Manager, Federal Programs and Strategic Initiatives, Equal Justice Works Kerry O’Brien, Director of Federal Programs and Strategic Initiatives Equal Justice Works

  8. Key National Service SAA Issues Areas by Program

  9. Wellness/Other Supports and Services by Program (n=59)

  10. Performance Measurement for Veteran and Military Family Programs • What is performance measurement? • A systematic process of tracking outputs and outcomes based on the goals of your program, as articulated in your program’s logic model • What is a logic model? • A logical chain of connectionsshowing what the program is to accomplish

  11. Why is Performance Measurement important for CNCS? • Assess the individual and collective results of programs and continue to enhance program effectiveness • Expand CNCS’s ability to account for the combined contributions of all organizations making use of our resources

  12. Why is Performance Measurement important for VMF Programs? • Assesses the individual and collective results of VMF programs by tracking outputs and outcomes related to improving • Military family strength and resilience • Quality of life for veterans, i.e.,: • Transition to civilian life • Education attainment • Job/vocational skills • Behavioral health

  13. Performance Measurement for VMFP: Collecting Data on Outputs and Outcomes • Tools to measure outputs: • Intake forms, attendance and tracking sheets to keep count • Tools to measure outcomes: • Pre, mid-term and post program surveys designed to measure increases in specific areas over time, as a result of participation in the program, focus groups, interviews • Where to find instruments, tools and resources for performance measurement: • http://www.nationalservice.gov/resources/npm/home • search by focus area

  14. Performance Measurement for Veteran and Military Family Programs • Important point: Performance measurement is NOT evaluation • “Unlike performance measures, evaluations estimate the impacts of programs by comparing the outcomes for individuals receiving a service or participating in a program to the outcomes for similar individuals not receiving a service or not participating in a program.” • Code of Federal Regulations for CNCS http://www.gpo.gov/fdsys/pkg/CFR-2010-title45-vol4/pdf/CFR-2010-title45-vol4-part2522-subpartE-subjectgroup-id1784.pdf

  15. Performance Measurement for Veteran and Military Family Programs • Performance measures track outputs and outcomes based on the goals of your program’s logic model • A logic model shows the chain of connections illustrating plans to meet community needs; the theory of change Need: access to employment resources INPUTS OUTCOMES OUTPUTS Program investments Activities Participation Medium Long-term Short What we invest: People, time Service provided: # of in-take & referrals Service recipients: # of returning veterans receiving referrals What results/changes: Veterans placed in employment prep programs; then employed

  16. Outputs for Veteran and Military Family Programs • Defining outputs • Amount of service provided (e.g., people served, products created, or programs developed) through the service intervention of your program • Common outputs for VMF programs • # of veterans or active Service members and families served • # of veterans or active Service members engaged in service provision through CNCS programs • Others?

  17. Outputs for Veteran and Military Family Programs – Program Examples • Veterans Helping Veterans Now • # of veterans referred to service providers that can meet their needs • Metro Community College, Omaha, Nebraska • # of people who received outreach from VISTAs regarding support needed by veterans returning to college • CA Conservation Corps • # of veterans employed as crew members

  18. Outcomes for Veteran and Military Family Programs • Defining outcomes • Reflect the changes in attitudes/beliefs, knowledge/skills, behavior, or conditions of individuals that occur in the organizations, communities, or environment as the result of the services provided • Common outcomes for VMF programs • Increase in educational attainment • Increase in vocational skills and credentials (e.g., professional licensure) • Increase in employment outcomes • Others?

  19. Outcomes for Veteran and Military Family Programs– Program Examples • Veterans Helping Veterans Now • # of veterans receiving services as result of referral who then gain employment (for example) due to services received • Metro Community College, Omaha, Nebraska • # of veterans succeeding in college due to support of informed college staff and faculty • CA Conservation Corps • # of veterans who completed the program and gained key skills for future employment

  20. Sources of Data Collection • Self-reported data through national service members and volunteers (e.g., # of veterans served) • Self-reported data from veterans through surveys (e.g., percentage of veterans served that obtain employment post-service) • Public use and administrative data sources (e.g., data from local VA organizations) • Observation – outside observers making ratings, counting behaviors

  21. Self-Report Data from Service Members • Key Question: Who are you serving? • Self-reported data through national service members and volunteers • Example: # of veterans served • Example: # of military families served • Important to know penetration rate (i.e., percent of veterans served in relation to total size of veteran population in your community) • VA data source, called VETPop, has publicly available data source on veteran population by zip code

  22. Self-Report Data from Beneficiaries • Key Question: What outcomes are you achieving? • Potential Outcomes: • Increase in educational attainment • Increase in vocational skills and credentials (e.g., professional licensure) • Increase in employment outcomes • Data Sources: • Self-report survey data from veterans/military • Public use data sources

  23. Public Use Data Sources of Interest • Demographic Data on Population • Veterans Population (VETPoP) • DMDC (DOD) – demographics report • Public Use Survey Data • Health (Wellness) • Survey of Army Families (SAF) • Air Force Community Assessment Survey • National Survey of Veterans

  24. Public Use Data Source of Interest • Public Use Survey Data (Cont.) • Health (Behavioral Health) • Survey of DoD Health Related Behaviors • MHAT Screenings • Sexual assault survey • Employment • VBA Vocational Rehab Survey • Homelessness • VA/DoL homeless counts

  25. Administrative Data Sources of Interest • Administrative Data • Homelessness • DoL VETS data on HVRP data • Education • Data on GI bill usage (VA) • Vocational Rehabilitation data • Employment • Transition Assistance Program data • Local data sources from DVOPs, LVERs, etc.

  26. Tools and Resources for Performance Measurement http://www.nationalservice.gov/resources/npm/home

  27. Equal Justice Works Kathryn Gravely Program Manager Kerry O’Brien Director of Federal Programs

  28. Equal Justice Works Overview • Mission • Equal Justice Works is a 501(c)(3) nonprofit organization dedicated to creating a just society by mobilizing the next generation of lawyers committed to equal justice  • AmeriCorps Program • 40 full-time lawyer members in 10 states. Most members serve for two years • 360 minimum-time law student members in 48 states. Most members serve 300 hours in the summer • Placement sites are legal aid organizations and courts working with low-income residents, including veterans

  29. Equal Justice Works: Veterans Theory of Change Summary Need: Veterans have complex legal problems that keep them homeless and in poverty Intervention: Lawyers and law students, in conjunction with veterans service infrastructure, resolve legal problems Evidence: 2012 Evaluation, Studies show that those with lawyers fare better than those without Outcome: Lives of veterans are improved, increased income, access to health care, better housing

  30. Equal Justice Works: Veterans Theory of Change • Need • Low-income and homeless veterans face complex legal problems related to disability benefits, fines, warrants, housing, employment, debt, and family issues • Lawyers are uniquely equipped to resolve such problems • Data Documenting Need • 2011 V.A. CHALENG report; Research articles on veterans & need for lawyers; HUD, HHA and Veterans Benefits Administration statistics; PTSD and TBI studies (e.g. 3 of top ten unmet needs of homeless vets are legal needs – CHALENG)

  31. Equal Justice Works: Veterans Theory of Change • Intervention • Lawyers and law students will resolve the legal problems of low-income and homeless veterans, leading to improved lives: higher incomes, better health, safe and secure housing and family stability • Intended Outcome • Improve the lives of veterans • Evidence-basis for Intervention • Impact of 2010-2012 Veterans Pilot Project • Studies that lawyers make a difference

  32. Equal Justice Works: Outputs vs. Outcomes • Overarching Goal • Through legal assistance, improve the lives of veterans • Outputs: Amount of Service Provided • Example: Assist 30 veterans with benefits claims • Outcomes: Reflect the Changes that Occur • Example: 20 veterans win benefit cases, providing access to health care and $360,000 in additional yearly income

  33. Equal Justice Works: Outcomes

  34. Equal Justice Works: Data Collection • What we could say in 2010 • 1,500 veterans served • Compelling stories about individual cases • What we could not say in 2010 • Outcomes of cases • Impact on veterans served • Specific information on veterans served (e.g., demographics, legal issues addressed, etc.) • Which sites were performing well/needed assistance to meet goals

  35. Equal Justice Works: Data Collection • Today, we can say: • 1,500 veterans served • Members completed 226 thorough case evaluations • Filed 91 disability benefits petitions or appeals • Thus far have won 29 of these cases, securing $642,668 in back benefits and $430,933 in yearly benefits for their veteran clients • 114 benefits cases are still pending • Successfully able to remove a barrier to employment for 127 veterans served • Demographic information specific to veterans served (e.g., era of service, age, gender) • Number of veterans served for each particular legal issues, both on macro and micro levels

  36. Equal Justice Works: Data Collection • Today, we can say: • Fines or other penalties waived or reduced for 78 veterans served, totaling $36,771 • Assisted 19 veterans in getting their criminal matters dismissed or expunged • Assisted 118 veterans in repairing their credit • Of the 1,416 veterans served, 740 were homeless or at risk of becoming homeless, and 739 were veterans with a disability • 77% were male and 23% were female—meaning that Fellows served a high number of female veterans, as 10% of veterans in the U.S. are female

  37. Equal Justice Works: Data Collection • Today, we can say: • 20% of those receiving services served in the Vietnam era, and 23% served in Iraq and Afghanistan • The remaining veteran clients served in all other eras of service, including pre-Vietnam, the Gulf Wars and during peacetime • The majority of veterans served were either between 26 and 35 years of age or 55 and older (216 and 420 served, respectively)

  38. Equal Justice Works: Data Collection How?

  39. Equal Justice Works: Data Collection • 2011 • Conducted interviews and focus groups about outcomes • 2012 • Implemented new optional measures early 2012 • Brought supervisors together in person – October • Outside evaluation completed – December • 2013 • Data resulted in stronger application for funding – Feb • Implemented refined measures & made them mandatory in March 2013 • Report submitted May 1 substantially better

  40. Equal Justice Works: Data Collection • Collection Format: • Was: Custom-built web application • Cost an hourly charge every time data needs changed • Was getting expensive • Output was Excel spreadsheet • Now: Google Drive - Forms • Program manager and coordinator designed, refined • Output is Excel spreadsheet • Drive.google.com – Create “Forms”

  41. Equal Justice Works: Data Collection in Google Forms

  42. Equal Justice Works: Great Stories • Equal Justice Works AmeriCorps Legal Fellow Rochelle Richardson • Homeless Persons Representation Project in Baltimore, MD • Many years experience as an employee of the VA • Reexamine the case of Mr. A, a homeless veteran with a service-connected mental disability whose claim for veteran's disability benefits had already been denied • Rochelle obtained records that had not previously been reviewed and petitioned to reopen the claim and submit further evidence • The petition was successful & the claim was granted in March 2013 • The client will receive a retroactive one-time payment of $50,000 and $3,000 a month in benefits

  43. Equal Justice Works: Lessons Learned • Define intended outcomes early • Align reporting with what funders require • Have performance measures and data collection tools ready at the beginning of program year • Have a “steering committee” of on the ground experts to consult • Keep refining and improving measures to more accurately reflect program impact

  44. Equal Justice Works: Next Steps • Figure out how to track and capture long-term outcomes • Example: Did increased income result in better housing, better health, better outcomes for children of veterans served? • Comparison Evaluation • Compare veteran outcomes in our program with national statistics or other programs • Big Vision • Randomized control trial

  45. Performance Measurement for Veteran and Military Family Programs: Re-Cap • Performance Measurement: • Tracks outputs and outcomes based on the goals of your program to meet community needs • Outputs: • Amount of service provided (e.g., veterans) through the service intervention of your program • Outcomes: • Reflect the changes that occur as the result of the services provided (e.g., veterans employed)

  46. Questions??? Dr. Chris Spera CNCS Kathryn Gravely Equal Justice Works Kerry O’Brien Equal Justice Works

  47. To Continue this Discussion…and Others Veterans and Military Families Knowledge Network www.nationalserviceresources.org/veterans http://bit.ly/VetsNet

  48. Join Us for the Next Webinar in this Series June 12, 2013 • Making a Difference: What We Know about Using an Evidence-based Approach • Register: http://tinyurl.com/c8gp8ll

  49. Thank you! Please complete the post-webinar survey

More Related