SYSTEMATIC REVIEWS 1 ESRC Methods Festival, Oxford 9.25-9.45 am 19 th July 2006

1. SYSTEMATIC REVIEWS 1ESRC Methods Festival, Oxford9.25-9.45 am 19th July 2006 Introduction to research synthesis and the ESRC Methods for Research Synthesis Node of the National Centre for Research Methods David Gough EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

2. Today’s sessions • Morning: Systematic Reviews 1 • 4 approaches to systematic reviews • Afternoon: Systematic Reviews 2 • Implications of different approaches for stages of a review

3. Social Science Research Unit 1. Five streams of work: • Childhood Studies • Evaluation of Social Interventions • Sexual Health, Reproduction and Social Exclusion • Evidence for Policy and Practice Information and Co-ordinating Centre • Perspectives, Participation and Research

4. EPPI-Centre vision and mission statement Developing and promoting participatory and user-friendly systematic reviews addressing important questions in different domains of policy, practice and research in the public interest

5. Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre Conducting reviews since 1993 In health promotion, education, transport. Supporting 25+ groups and 70+ reviews in criminology, employment,speech and language, social care Methods and tools to Support all types of review Question and primary studies EPPI-Reviewer software Methodological work, e.g. Methods for Research Synthesis Project ESRC National Centre for Research Methods Formal links with Cochrane and Campbell Collaborations Short courses and Masters course in evidence for public policy and practice On-line libraries of research evidence

6. ESRC NCRM Methods for Research Synthesis node • A question-driven framework • Methods for integrating a breadth of approaches • Contribution from primary studies • Linking synthesis with e-social science • Training and capacity building • Engagement with users in relation to understanding and demand for SRS

7. Reviews: Building on what we know What do we want to know? What do we know already? How do we know it? What more do we want to know? How can we know it? Not just a linear process: many overlapping research questions and questions, concepts and methods will change

8. How do we do this? Methods of study • Primary study: explicit rigorous methods within accepted rules of evidence for that discipline • Secondary study: exactly the same!

9. Key decision-making stages in SRS Form review team Formulate review question and develop protocol Define studies to be considered (inclusion criteria ) Search for and screen studies (search strategy) Describe studies (systematic map of research) Assess study quality (and relevance) Synthesise findings (answering review question) Communicate and engage

10. USERS OF RESEARCH: PERSPECTIVES AND PARTICIPATION QUESTIONS What do we want to know? (i) Research (ii) User of service/ public (iii) Practitioner TYPES OF KNOWLEDGE (iv) Policy community (v) Organisational Interpretation and application Effects? RESEARCHQUESTIONS How/processes? Nature/extent/frequency? Perspectives/ concepts? Communication Action research What has been done? RESEARCH STUDIES AND METHODS Survey Case study Experimental Multimethod What do we know? How do we know it? What don’t we know? What more do we want to know? How could we know it? Effects? How/processes? RESEARCH EVIDENCE Perspectives/ concepts? Nature/extent/frequency?

11. Some reasons why research does not make a difference • Research only one of many factors • Instrumental use / avoidance of research • Questions about the quality of research or the perspective being ‘sold’ • Access to research • Interpretation and use and relation to other knowledge and contexts • Relevance of the research

12. Approaches to reviewing and types of data and synthesis Nature of data and analysis in the review predominantly: • A priori specified method of review or iterative approach (for e.g review question, bases for assessment of quality of studies, and/or framework for synthesis emerging during the review) • Numerical or narrative data • ‘Empirical’ or ‘conceptual’ data • Relatively homogeneous or heterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative (meta empirical) or interpretative (meta conceptual) synthesis of data

13. Key decision-making stages in SRS Form review team Formulate review question and develop protocol Define studies to be considered (inclusion criteria ) Search for and screen studies (search strategy) Describe studies (systematic map of research) Assess study quality (and relevance) Synthesise findings (answering review question) Communicate and engage

14. Synthesis • ‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a theory or system’ (OED) • Not just a report of the findings of the individual studies in a review • Involves a transformation of the data from primary studies in some way

15. Approaches to reviewing and types of data and synthesis Nature of data and analysis in the review predominantly: • A priori specified method of review or iterative approach (for e.g review question, bases for assessment of quality of studies, and/or framework for synthesis emerging during the review) • Numerical or narrative data • ‘Empirical’ or ‘conceptual’ data • Relatively homogeneous or heterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative (meta empirical) or interpretative (meta conceptual) synthesis of data

16. Approaches to reviewing :examples Statistical meta analysis1 • A priori or iterative procedures of review • Numerical or narrative data • ‘Empirical’ or ‘conceptual’ data • Relatively homogeneous or heterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative or interpretative synthesis of data 1For e.g. Many statistical meta analysis reviews addressing ‘what works’ questions

17. Meta analysis example: Does sex education improve the use of contraception amongst young people? From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. British Medical Journal 231: 1426-1434

18. Approaches to reviewing:examples Narrative reviews of quantitative studies1 • A priori or iterative procedures of review • Numerical or narrative data • Empirical’ or ‘conceptual’ data • Relatively homogeneousorheterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative or interpretative synthesis of data 1For e.g. Many narrative reviews addressing ‘what works’ questions

19. Approaches to reviewing:examples Thematic reviews of views studies and some meta ethnography • A priori or iterative procedures of review • Numerical or narrative data • ‘Empirical’ or ‘conceptual’ data • Relatively homogeneousorheterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative or interpretative synthesis of data

20. Approaches to reviewing:examples Some meta ethnography, critical interpretative synthesis, realist synthesis • A priori or iterative procedures of review • Numerical or narrative data • ‘Empirical’ or‘conceptual’ data • Relatively homogeneous or heterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative or interpretative synthesis of data

21. Approaches to reviewing:examples Some Bayesian meta analysis • A priori or iterative procedures of review • Numerical or narrative data • ‘Empirical’ or ‘conceptual’ data • Relatively homogeneousorheterogeneous data • Numerical or narrative analysis of data in synthesis • Integrative or interpretative synthesis of data

22. Mixed methods SCOPING AND MAPPING (of 272 reports of 193 studies) Review question e.g. What is known about the barriers to, and facilitators of, fruit and veg intake amongst children aged 4 to 10 years? ‘Views’ studies (N=8) 1. Application of inclusion criteria 2. Quality assessment 3. Data extraction 4. Thematic synthesis Trials (N=33) 1. Application of inclusion criteria 2. Quality assessment 3. Data extraction 4. Synthesis using statistical meta-analysis Trials and ‘views’ Mixed methods synthesis

23. Extensiveness of reviews Concerns related issues of (i) time and staff resources and (ii) breadth and depth of evidence (i) Quick overview • Non systematic scoping reviews • Systematic but limited maps or syntheses • Are Rapid Evidence Assessments 2 or 3?

24. (ii) Breadth of evidence and evidence sources in reviews a. Broad or narrow range of evidence types. b. Broad or narrow evidence sources. c. Focus on secondary or primary sources: Inclusion of previous reviews, primary data or both In addition: reviews may have sub-questions focusing on different evidence (or different questions / methods on the same evidence)

25. Other aspects of extensiveness of systematic reviews Each stage of a review can vary in its relative breadth, depth, exhaustiveness, and analysis: • Question: breadth broad or narrow • Searching: full exhaustive or purposive or scoping • Map: analytic or descriptive • Synthesis breadth: broad or narrow • Extent of evidence: adequate unless specified minimal • Synthesis type: synthetic unless specified descriptive

26. Feed into decision making Feed into decision making Communication of review findings Communication of review findings

27. Thank you SSRU website: http://www.ioe.ac.uk/ssru/ SSRU's EPPI website: http://eppi.ioe.ac.uk NRCM MRS website http://www.ncrm.ac.uk/nodes/mrs/people.php Email: d.gough@ioe.ac.uk