“There’s nothing so practical as good theory” Evaluating ‘Wicked’ Problems

1. “There’s nothing so practical as good theory”Evaluating ‘Wicked’ Problems Bob Gardner July 2011

2. Clear Starting Points • English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that “there’s nothing so practical as good theory” • Pawson wasn’t arguing for abstract theory, but for ensuring we are always clear about • the assumptions we are making that underpin our work – whether planning a specific service initiative or developing a broad multi-sectoral strategy • the pathways and factors that we assume will lead from the planning through service delivery to the hoped-for impact • how we think all of this will vary depending upon the organizational, social or policy context • there are many approaches and as many terms – theory of change, program theory, framework for change • but the basic idea is to be really clear about starting points and premises when planning any initiative • and it’s this set of assumptions, pathways and objectives in our theory of change that we evaluate

3. The Problem to Solve = Systemic Disparities in Ontario • there is a clear gradient in health in which people with lower income, education or other indicators of social inequality and exclusion tend to have poorer health • + major differences between women and men • the gap between the health of the best off and most disadvantaged can be huge – and damaging • in addition, there are systemic disparities in access to and quality of care within the healthcare system

4. ‘Wicked’ Policy Problems • health inequities and their underlying social determinants of health are classic ‘wicked’ policy problems: • shaped by many inter-related and inter-dependent factors • in constantly changing social, economic, community and policy environments • action has to be taken at multiple levels -- by many levels of government, service providers, other stakeholders and communities • solutions are not always clear and policy agreement can be difficult to achieve • effects take years to show up – far beyond any electoral cycle • have to be able to understand and navigate this complexity to develop solutions • need comprehensive strategy to tackle the roots of health inequality in the wider social determinants of health • from high-level national social and policy change to reduce inequality through community-based innovation, cross-sectoral collaboration and mobilization

5. …Need Theory • for complex social problems – from health inequities, through poverty reduction, to neighbourhood renewal and comprehensive community initiatives -- we need to be able to: • identify the connections and causal pathways between multiple factors • articulate the mechanisms that we assume drive change in these factors and the problem as a whole • within that, identify the crucial leverage points that will drive significant change in the problem • develop strategies for moving these leverage points • specify the short, intermediate and long-term outcomes expected and the preconditions for achieving them. that is our theory of change

6. … And Here’s How how to develop theory of change: • basic principle is to pull group working on the issue together to answer those questions • to clearly articulate our starting points – what we think are the key causal pathways, mechanisms for change and crucial leverage points • these are our assumptions and goals for how we think program, community initiative or policy direction will work • based upon best available evidence, learning and experience and some wrinkles • can’t have all the answers and can’t be too prescriptive – don’t wait for ‘perfect’ theory • be as participatory as possible – especially if community-based evaluation then tests these assumptions about change mechanisms and outcomes → contributes to further learning and development of the initiative

7. Theories of Change for ‘Wicked’ Problem I: Health Inequities • complex multi-level strategies to tackle health gaps: • international frameworks such as WHO Commission on Social Determents • European Union and other international initiatives • many individual countries • key features in all: • focus on inclusive labour market, anti-discrimination, childcare, affordable housing, social security and other macro policies • targets and deliverables for relevant departments • equitable access to quality health care is just one part of this broader package • emphasized partnerships with community service providers and organizations – in both policy development and service delivery • Theory of change = all about reducing structured inequality : • health inequities arise out of wider social determinants of health • won’t solve through health reform alone • underlying structures of social and economic inequality need to be addressed by new policies • policies need to be aligned with the incentives and processes that drive government • key levers will vary – e.g. could be reducing prevalence and impact of precarious employment • has to involve collaboration and coordination across governments and with many community and non-government stakeholders

8. Theories of Change for ‘Wicked’ Problem II: Community Level Action • Comprehensive community initiatives: • broad partnerships of local residents, community organizations, governments, business, labour and other stakeholders • coming together to address deep-rooted local problems – poverty, neighbourhood deterioration, health inequities, racism • collaborative cross-sectoral efforts – not just for better coordinated service provision but for community development and capacity building • Vibrant Communities – 14 communities across the country working to build individual and community capacities to reduce poverty • Theory of change: • poverty, social exclusion, etc are not individual problems but are deeply rooted in fundamental social and economic inequality → even the best programs cannot get at these underlying structures • underlying structures of inequality need to be addressed by new policy directions • building community capacities is another direction to address structural inequalities: • drilling down – Vibrant Communities works to build up individual skills and other assets as building block for community change • broad collaborations are key to both enhancing community capacities and implementing policy changes locally • broad coalitions also can highlight the deep-seated nature of problems → build community and public awareness and focus political advocacy

9. Evaluating Complex Policy and Community Initiatives • far too complex to pick apart all the causal relations and patterns of influence: • very difficult to attribute particular changes to particular components of the overall initiative • will never meet RCT gold standard of proof – that approach can’t capture complexity • but that doesn't mean initiative is ineffective • impact can take many years to show up • and that doesn’t mean nothing is happening • traditional evaluation of one program in isolation or of a particular population among many will not capture this complexity • so we evaluate our theory of change • Tamarack talks of framework for change – to highlight the breadth of activities and stakeholders involved in comprehensive initiatives • we identified levers in our strategy – did they prove to be important in practice? • looking for indications that the change mechanisms unfold as we expected, that the direction of causal influence and impact is as we expected ,etc • looking for evidence that outcomes anticipated are being achieved, and the change mechanisms we anticipated have been important

10. Build Use and Users Into Evaluation • crucial to identify users of evaluation • what they need • the context in which they will interpret and use the results of evaluation • policy makers need to: • solve social problems as they see them • determine where to invest scarce resources for maximum impact • avoid mixing up accountability requirements with evaluation • get beyond the assumption that randomly controlled trials are the ‘gold standard’ of evidence • community leaders and members are interested in: • the same – how to solve social problems as they see them – and live them • how to make case for reform/spending to policy makers • how to ensure programs and services meet the needs of community members – and how to continually improve service delivery • both need to be able to determine – and demonstrate – what types of initiatives work, for whom and in what contexts

11. Case Study: Health Equity Strategy Into Action • goal is to ensure equitable access to high quality healthcare regardless of social position • can do this through a multi-pronged strategy: • building health equity into all health care planning and delivery • doesn’t mean all programs are all about equity • but all take equity into account in planning their services and outreach • aligning equity with system drivers and priorities • embedding equity in provider organizations’ deliverables, incentives and performance management • targeting some resources or programs specifically to addressing disadvantaged populations or key access barriers • looking for investments and interventions that will have the highest impact on reducing health disparities or enhancing the opportunities for good health of the most vulnerable • while thinking up-stream to health promotion and addressing the underlying determinants of health

12. Equity-Focused Planning • a critical component of all these strategic directions is good planning • addressing health inequities requires a solid understanding of: • key barriers to equitable access to high quality care • the specific needs of health-disadvantaged populations • gaps in available services for these populations • great deal of emphasis on tools such as Health Equity Impact Assessment • but can’t see particular tool as ‘magic bullet • to develop effective planning, we need: • clear strategy • a coherent approach • a repertoire of effective tools and techniques • support for planners and practitioners to effectively use them • good actionable information • and drilling down: what is our ‘theory’ of how equity-focused planning works?

13. takingaccount of social constraints & conditions not just individual programs but coordination, partnerships & collaboration

14. enhanced access to health promotion for most disadvantaged up-stream heath conditions & opportunities improve fastest for those in greatest need

15. Complexities At All Stages • processes and constraints are complex, and outcomes uncertain and unpredictable, at each of these junctures • and all of this varies by context: • particular communities or neighbourhoods – with their different health challenges and needs • particular population health and service landscape in specific areas • community resilience, connectedness, organizing and traditions • further specified by health condition or concern (e.g. the determinants and dynamics of mental health are different) • we need to know what works best at each of these junctures (and cumulatively), what programs and frameworks work for particular populations, and what works in varying contexts • → need to build evaluation in from the start to learn