Frameworks for program planning and evaluation: Problem Analysis and Logic Models

1. Frameworks for program planning and evaluation:Problem AnalysisandLogic Models Geraldine Oliva M.D. , MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez, MA Paula Fleisher, MA June 2013

2. Today’s Agenda • Why Evaluation?! • Using a socio-ecological problem analysis framework to identify “causal pathways” • Developing strategies to address a problem based on upstream causes and risks • Developing a logic model for program planning and evaluation

3. Why Evaluation?! Root word: value Whose values? Funders, clients, organization, etc. It helps us tell our story It helps us understand what we are doing well and what we can improve…

4. “Traditional” Program Planning Cycle Convene Public Health/ Community Coalition Evaluate/ Measure Performance Plan & Implement Programs Assess Community /MCAH Resources & Strengths/Capacity Develop Objectives Analyze Problem & Select Interventions Assess & Prioritize Health Status/Problems

5. New Paradigms for Understanding Persistent Disparities • Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities • Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities • Re-emergence of “life course model”, importance of prevention and intervention along age spectrum

6. Socio-Ecological Models

7. Lifecourse Model

8. Lifecourse Model as a Cycle Mother overweight/diabetic Poverty/fast food/food habits in young woman Infant born LGA/Abnormal GT Adolescent with increased # fat cells/ abnormal GT Working Mom-Infant Fed Formula School with no PE, unsafe parks latch key child watches TV / gets more overweight Family Culture - high fat, high sugar diet- child overweight UCSF Family Health Outcomes Project

9. How do we Incorporate these New Paradigms into a Community/Public Health Planning Framework ?

10. African-American, Latino Children Drink More Sugary Fruit Juice Than Their White Peers Share this story: Share Print ByJuliana Bunimon May 30, 2013 Email While there has been a steep decline in kids’ consumption of sugar-sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco. The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California. “The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend,” saidAmy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children’s Hospital and San Francisco General Hospital and Trauma Center. “But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda.” …Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in 2011. Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.

11. Seeing solutions with “Ecological Lenses” • At the individual level: Nutrition information (for parent or child) • At the family level: Nutrition education for parent and child • At the community level: a media campaign targeting families • At the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean water • At the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WIC • OR • Eliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools

12. Rationale for Doing a Formal Problem Analysis? • In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another • Using a multilevel socio-ecological framework ensures that upstream factors are included • Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation

13. Steps in a Problem Analysis 1. Examine epidemiologic data 2. Examine literature and consult experts (community and science) 3. Determine extent to which these factors are active in the community

14. Steps in a Problem Analysis 4. Determine relative contribution of each identified factor 5. Identify the interrelationships among factors – causal pathways 6. Determine the most effective points in the causal pathways for intervention

15. A Generic Framework for Health Problem Analysis Social/Economic/Policy Level SES Safety Education Culture Environment Economy Health Care Policies Family/Community/Institutions Level School/Workplace Family/Household Community Health Care/Providers Individual Level Health Status/ Medical Conditions Psychological Factors Genetic/Biological Cognitive Factors Health Behaviors Identified Problem

17. Marketing by fast food industry SES/Policy Unsafe neighborhood/streets Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Lack of funding for education Family/community No opportunities for exercise in school or recreation areas Family lack of health Insurance Health services lack of knowledge of nutrition High costs for treatment Families limited income Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Unavailability of healthy food choices in local stores , schools and restaurants Too much screen time (TV computer) Individual Unhealthy diet Insufficiency of physical activity PROBLEM: Childhood Obesity

18. Linkages Definition: The association between precursors and problem • Review the literature • Consult experts • Community • Science • Consult stakeholders • Analyze your data

19. Marketing by fast food industry SES/Policy Unsafe neighborhood/streets Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Lack of funding for education Family/community No opportunities for exercise in school or recreation areas Family lack of health Insurance Health services lack of knowledge of nutrition High costs for treatment Families limited income Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Unavailability of healthy food choices in local stores , schools and restaurants Too much screen time (TV computer) Individual Unhealthy diet Insufficient physical activity PROBLEM: Childhood Obesity

20. No opportunities for exercise in school or recreation areas Diagramming Causal Pathways to Identify Strategic Interventions Unsafe neighborhood streets Intervention Overweight Inadequate physical activity Too much screen time (TV computer)

22. Problem Analysis – Homey Clients Social Determinants  crime/gun policies/  prevention $ Economic policy Poverty / no jobs or bias in hiring Racism/Destruction of indigenous culture  $ for schools/low standards/legacy of colonialism Family/Community/Local Institutions Police racial profiling Community-lack of social capital/ cohesion/pride/trust Family disintegration Parents don’t care/use drug/don’t supervise/not educated/poor/lackcultural pride  Jobs/lack of training opportunities Barriers to accessing services: Language/Cultural/ $ Transportation/Legal Schools push kids out/racism in staff/poor quality/lack of cultural affinity Individuals Alcohol/drug use Unsafe sex/ bad food/no exercise Mental illness/PTSD/  self esteem/  resiliency/  trust/ sense of belonging FAS/ADD/ADHD/other drug exposure Learning difficulties/brain trauma/school failure/no college prep PROBLEM: Gang Membership/Incarceration

23. HOMEY- Kapuli Causal Pathway: Program Intervention Points Kids: Mental health problems School failure Low self esteem lack resiliency No sense of true cultural identity No sense of trust or belonging Communities: Racism in schools Racism by police Lack of cultural identity in family/community family disintegration Gangs/Jail Society: Racism Lack of respect for indigenous cultures School failure Self- Destructive Behaviors Kapuli Interventions

24. INPUTS OUTPUTS OUTCOMES–IMPACT Staff Cameron Space Core Members Funds Activities Participation Short Medium Long-Term Positive response during recap activity Active participation in group Successful school/CBO outreach Come back Join political campaigns Attend ceremonies Express value of social action Increase self esteem/resiliency/capacity for action Kids return to school Kids get jobs Kids stay out of gangs Kids stay out of jail Street outreach Educate about history of social justice movements for people of color/indigenous 10 strategies for change Teach about racism/colonialism Schools CBO’s Kids from streets ASSUMPTIONS: Learning about history of social justice movements will give them tools for change Learning about impact of racism/colonialism will give kids an undestanding of their own history Episodes of tragedy due to racism in their community Media/movie about examples of racism/harassment by police ENVIRONMENTAL FACTORS Adapted from MOMEY-Kapuli Logic Model

25. Another Example: Obesity prevention/ intervention in the Bayview Hunters Point of SF • Designed in 2011, in partnership between: • UCSF • San Francisco Department of Public health • BVHP residents • other stakeholders • See handouts for problem analysis and logic models • As of 2013, focus is on: • Reducing consumption of sugar sweetened beverages • Activating community park space

26. Benefits • Clearly documents the decision making process for the group and for others • Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public

27. Logic Model - Defined Logic models are a common tool used by evaluators and program planners Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes. U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

28. Logic Model – Terms U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25. • Relationship • Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes. • Intended • Logic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed . Over time, evaluation, research, and experience will help us learn what works, and the model will evolve.

29. Logic Model - Components • Inputs • Resources that go into the program and are required to make it happen • Activities • events or actions done by the program and its staff • Outputs • Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants). • Outcomes • The changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.

30. Basic Program Logic Model U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

31. Let’s build a logic model! Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest) What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does What resources do you need (inputs)?

32. Q and A What was clarified for you; any “aha!” moments? What still concerns or challenges you?

33. Contact Information Community Engagement and Health Policy Program, CTSI/UCSF 415-206-4048 rousej@fcm.ucsf.edu paula.fleisher@ucsf.edu roberto.vargas@ucsf.edu www.sfhip.org