Getting To Outcomes and Teen Pregnancy Prevention: Past, Present, and Future

1. Getting To Outcomes and Teen Pregnancy Prevention: Past, Present, and Future Matthew Chinman, PhD, RAND Corp.; Joie Acosta, PhD, RAND Corp HTN Conference October 13, 2011

2. Agenda • Why is Getting To Outcomes (GTO) needed in teen pregnancy prevention? • What is GTO? • How do we know GTOis helpful? • Current products and future directions

3. Teen Sex Is a Significant Problem • Rates of teen pregnancy in the U.S. are the highest among industrialized nations and increasing1 • Sexually active teens are at high risk for contracting sexually transmitted infections (STIs)2 • Early child-bearing puts adolescents and their children at risk for additional negative consequences3 • Kost K, Henshaw S, Carlin L. U.S. Teenage Pregnancies, Births and Abortions: National and State Trends and Trends by Race and Ethnicity 2010. • Kaestle E, Halpern CT, Miller WC, Ford CA. Young age at first sexual intercourse and sexually transmitted infections in adolescents and young adults. American Journal of Epidemiology. 2005;161:774-780. • Levine JA, Pollack H, Comfort ME. Academic and behavioral outcomes among the children of young mothers. Journal of Marriage and Family. 2001;63:355-369.

4. Research Supports Prevention Interventions • Mathematica Policy Research review • http://www.hhs.gov/ash/oah/prevention/research/index.html • “A program had to be supported by at least one high- or moderate-rated impact study showing a positive, statistically significant impact on at least one priority outcome (sexual activity, contraceptive use, STIs, or pregnancy or births)” • 28 programs met criteria • BUT…evidence-based programs for reducing teen pregnancy are not in wide use and are often implemented with low fidelity* *Cassell C, Santelli J, Gilbert BC, Dalmat M, Mezoff J, Schauer M. Mobilizing communities: An overview of the Community Coalition Partnership Programs for the Prevention of Teen Pregnancy. Journal of Adolescent Health. 2005;37:S3-S10

5. There Is A Gap Between Research & Practice • Prevention delivered in schools, CBOs, community coalitions • Communities face challenges reaching outcomes: • “Off the shelf” programs can be difficult to implement • Funding is limited • Implementation requires specialized knowledge and skills (“capacity”) • Practitioners are seeking to accommodate national accountability movement • There have been few efforts to narrow the gap

6. Agenda • Why is Getting To Outcomes (GTO) needed in teen pregnancy prevention? • What is GTO? • How do we know GTOis helpful? • Current products and future directions

7. Getting To Outcomes Is Designed to Build Capacity for High Quality Prevention • GTO is a model: poses ten steps that must be addressed in order to obtain positive results AND • GTO is an intervention: provides practitioners with the guidance necessary to complete those steps with quality

8. GTO as a Model Modify the program or best practices to fit your needs. 4 3 Find existing programs and best practices worth copying. 2 Identify goals, target population, and desired outcomes. 5 Assess capacity (staff, financing, etc.) to implement the program. Choose which problem(s) to focus on. 6 Make a plan for getting started: who, what, when, where, and how. 1 DELIVERING PROGRAMS Evaluate planning and implementation. How did it go? 7 Consider how to keep the program going if it is successful. 10 Evaluate program’s success in achieving desired results. Make a plan for Continuous Quality Improvement. 8 9 Steps 1-6 PLANNING Steps 7-10 EVALUATING AND IMPROVING

9. GTO as an Intervention • Goal of GTO is to build capacity in practitioners to conduct high-quality prevention • Includes three components • Manuals of text and tools • Training • Technical Assistance (TA) • Based on Empowerment Evaluation theory and consistent with social cognitive theories of behavioral change

10. How does GTO work? Increases program staff: -GTO knowledge -GTO attitudes Improves outcomes for youth program participants • Increases program staff GTO behaviors = High-quality prevention • GTO intervention

11. Program Operations As Usual Capacity Program Characteristics Performance Program Staff Characteristics Community- wide youth outcomes Outcomes for youth program participants

12. Program Operations With GTO Capacity Program Characteristics GTO Intervention Performance Program Staff Characteristics Community- wide youth outcomes Outcomes for youth program participants

13. GTO’s 10 Steps Are Documented inGTO Manuals • Chapters for each of the 10 accountability steps include: • Definitions • Justification • Strategies & techniques • Examples from real prevention programs • Checklist • Glossary of terms • Appendices that contain tools • Model program descriptions

14. Manual of Text & Tools

15. GTO Training • For staff who deliver programs and their organization leaders • Often lasts a full-day or longer, often with follow-up • Involves learning about how to apply 10 steps in their programs • Walk through manual • Learn about various tools to accomplish 10 steps • Try to tailor it to local program needs

16. Technical Assistance (TA) • Who provides TA? Varies. From PhD to BA. From full to part time. • Who receives TA? Usually program coordinators When? Varies. Weekly to 2X/month via meetings, phone, email. Ongoing over 1 to 2 yrs. • What is done? • Deepen understanding of GTO • Diagnose programs, determine priorities • Apply the GTO process to programs • e.g.: TA would help programs to develop or revise logic models, goals/objectives, existing programming, evaluation processes

17. An Example: Promoting Science-Based Approaches to Teen Pregnancy Prevention Using Getting to Outcomes

18. Step 1: Needs and Resources • Identifies important behaviors and determinants for science-based pregnancy prevention Teaches how to conduct a needs/resources assessment using -existing data -new data

19. Step 2: Goals and Outcomes • Supports: • -development of SMART goals and outcomes • -creation of a logic model

20. Step 3: Best Practices • Walks users through a checklist for identifying and adapting best practices

21. Step 4: Fit • Helps program to conduct a fit assessment, • including emphasis on cultural fit

22. Step 5: Capacity • Supports a program capacity assessment to identify strengths and areas for improvement

23. Step 6: Plan • Supports the development of a work plan and accompanying report

24. Step 7: Process Evaluation • Helps users identify existing process data and areas to augment, includes sample data collection instruments

25. Step 8: Outcome Evaluation • Helps users design and implement an evaluation, including sample survey questions

26. Step 9: Continuous Quality Improvement • Provides support for using data to improve • program quality

27. Step 10: Sustainability • Provides a template to help programs plan for sustainability

28. Agenda • Context for Getting To Outcomes (GTO) • What is GTO? • How do we know GTOis helpful? • Current products and future directions

29. GTO Study # 1: GTO in Drug Prevention • Quasi-experimental design • Across 2 AOD prevention coalitions, included 6 GTO programs and 4 comparison programs • All programs were different, some evidence based • Approach • Implement GTO for 2 years (Manuals, Annual training, bi-weekly TA from .5 PhD w/ modest oversight) • Compare GTO and comparison groups from baseline to 2 year point on • Capacity of individual coalition members • Program performance of whole programs

30. GTO Diffused Throughout the Organizations • Established a monthly report using the GTO framework • Met quarterly to discuss the progress of GTO implementation • Formed workgroups to address progress on GTO steps • Initiated a continuous quality improvement program

31. Example of GTO: Teen Court • Jury of peers diverts 1st offenders (7th – 11th graders) from Juv Just Sys while holding them accountable • 90 days “Sentences” could be peer groups, jury duty, community service, AOD education

32. Example of GTO: Teen Court How GTO helped: • Established pre-post evaluation (surveys, database) • Survey development (Pre-Post surveys of AOD use/attitudes & individual and family risk factors (better AOD use & knowledge, decision-making, school importance) • Data entry and database construction • Data collection procedures (Track completion of sentence, ~80% completed)

33. Example of GTO: Teen Court How GTO helped: • Utilized CQI to improve the evaluation each time • Promoted better communication between program staff and Executive Director • Data used • to meet grant reporting requirements • for sustainability including continuation and expansion (e.g., work w/ grant writer) • to highlight ongoing training, staffing & technical assistance needs

34. Also GTO diffused into programs…Example Program Evaluation:Teen Court (CA) *=significant change * * * Frequency of decision-making skill use --- --- ---

35. Individual Capacity Assessed Via Survey of Coalition Members • Measured the knowledge, attitudes and skill regarding the activities that GTO targets at three time points (baseline, Yr 1 and Yr2) • Compared responses using two different approaches • Intent-to-treat (GTO vs comparison programs) • Participation Index (Staff reported GTO use vs no use)

36. Programs Performance Assessed Via Interview • Interview key program staff of all programs • Interviewer rates whole program on 14 dimensions corresponding to the tasks prescribed by GTO • Each dimension is on a 7–point scales from “high performance” to “low performance”

37. GTO “dose” Looked at the amount of GTO delivered in two ways: • Program level: TA staff tracked—by GTO step—how many TA hours they gave to each GTO program • Individual level: Tallied how much each coalition staff member participated in GTO via coalition survey items (T/F) • I received technical assistance on Getting To Outcomes • I have participated in training on Getting To Outcomes • I have read most of the Getting To Outcomes materials • I have made plans to use Getting To Outcomes • I have talked in details with others in ____ about how Getting To Outcomes can improve my programs • I have secured, or tried to secure, resources to use Getting To Outcomes

38. GTO Evaluation Showed Positive Results • Individual level • Capacity of individual coalition members to do prevention increased with more GTO participation • Program level • GTO programs improved performance more than comparison programs

39. GTO Programs Showed Greater Improvement Than Comparison Programs High Performance Low Performance GTO Programs Comparison Programs

40. TA Contributed to Improvement Correlation (r)=.55, p=.09, n=10

41. Study #1 Summary: What Did We Learn? • GTO improves capacity to do prevention and the performance of tasks thought to be important • Those (individuals & programs) with greater exposure to GTO demonstrated more gains • Evaluation activities were emphasized in TA and increased significantly over time • GTO takes time and resources to implement

42. GTO Study # 2: GTO applied to Assets • Randomized Controlled Trial • 12 PYD coalitions in Maine, each with up to 5 programs • Use AGTO: GTO applied to Developmental Assets • All programs were different, few evidence based • Approach • Implement GTO for 2 years • Compare GTO and comparison groups from baseline to 2 year point on • Capacity of individual coalition members • Program performance of whole programs

43. Multi-tiered AGTO infrastructure Large team of collaborators TA Supervisors Two full-time TA providers Community coalitions and programs

44. GTO Study # 2 uses the same data elements as Study #1 • Individual capacity – coalition survey • Program Performance – Interview • AGTO “dose” – T/F survey items/TA hours

45. AGTO Study Design Year 2 Year 3 Year 4 Cohort 1 -6 coalitions -30 programs -174 members Cohort 2 -6 coalitions -30 programs -174 members AGTO AGTO AGTO Baseline Mid Post Follow-up Coalition Survey Coalition Interview

46. Where are we now? Year 2 Year 3 Year 4 Cohort 1 -6 coalitions -30 programs -174 members Cohort 2 -6 coalitions -30 programs -174 members AGTO AGTO AGTO Baseline Mid Post Follow-up Coalition Survey Coalition Interview

47. Preliminary Results from Year 1: Outcome Evaluation • Individual staff prevention capacity • Participation was associated with significant increases in knowledge & targeted by AGTO • Program Performance • AGTO programs showed some improvement in how well they developed goals and carried out evaluation; started new evaluations • TA efforts make a difference overall (more TA hours=more change), BUT that can differ across prevention tasks • i.e. hours devoted to evaluation improvement nets more than hours devoted to best practices improvement

48. Program Performance % change from BL to 1 Yr.

49. Outcome Evaluation Details More TA, more improvement Correlation (r)=.64, p=.12, n=7

50. Outcome Evaluation Details New Process Evaluations • 79% (n= 22) of coalitions and programs are conducting process evaluations • Of those 82% (n=18) have been started since participating in the AGTO project New Outcome Evaluations • 65% (n=18) of coalitions and programs are conducting outcome evaluations • Of those 44% (n=8) have been started since participating in the AGTO project