1 / 16

Government Social Research Unit

Government Social Research Unit. What Government Needs From Social Research: Systematic Reviews. Philip Davies PhD Government Social Research Unit HM Treasury London SW1A 2HQ. www.gsr.gov.uk. Outline. Evidence-Based Policy Making Sheer amount and flow of information/research

lizina
Download Presentation

Government Social Research Unit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Government Social Research Unit What Government Needs From Social Research: Systematic Reviews Philip Davies PhD Government Social Research Unit HM Treasury London SW1A 2HQ www.gsr.gov.uk

  2. Outline • Evidence-Based Policy Making • Sheer amount and flow of information/research • Limitations of single studies • Need for the balance of evidence • Variable quality of research outputs • Need to separate the wheat from the chaff • Different Types of Review/Evidence Assessments

  3. Evidence-Based Policy Making “Evidence-based government helps people make well-informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (Davies, P.T., 1999)

  4. Sheer Amount and Flow of Information/Research • Large number of journals and texts • Plus other sources/media for research outputs • Publish or perish ethos  increases stock and flow • “The limitations of human information processing abilities exceeds the capabilities of the unaided human mind” (Eddy, Hasselbad and Shachter, 1992) • Researchers need help to find, sort and appraise research

  5. Limitations of Single Studies • Single studies can misrepresent the balance of research evidence • Illuminate only one part of a policy issue • Sample-specific • Time-specific • Context-specific • Often of poor methodological quality • Consequently, biased

  6. Systematic Reviews “attempt to discover the consistencies and account for the variability in similar-appearing studies” “seeking generalisations also involves seeking the limits and modifiers of generalisations” “identify the contextual-specificity of available research and evidence” (Cooper and Hedges, 1994:4).

  7. Variable Quality and Relevance of Research Outputs • Not all research is of sufficient quality • Unclear objectives • Poor research design • Methodological weaknesses • Inadequate data reporting • Selective use of data • Unsupported conclusions • Not all research addresses policy questions • Uncertainty and inconclusiveness of scientific research • Need to separate the wheat from the chaff

  8. Separating the Wheat from the Chaff • Critical appraisal criteria will different for each study type • Transparency of the grounds for inclusion/exclusion is essential

  9. Different Types of Review • Narrative (Traditional Literature) Reviews • Vote Counting Reviews • Systematic Reviews (non-aggregative) • Statistical Meta-Analyses (aggregative) • Meta-Ethnography • Rapid Evidence Assessments

  10. Rapid Evidence Assessments - What Are They? • Systematic reviews of existing evidence • Timed to meet the needs of policy makers/practitioners • Strategically using the ‘three arms’ of systematic searching • Electronic sources • Print sources • Grey Literature • Critical appraisal of studies identified • Summary of findings, with caveats and qualifications • Tentative findings Interim Evidence Assessments

  11. Rapid Evidence Assessments - Advantages • Provides sounder evidence than selective literature reviews • Better than opinion-based policy • Provides a challenge function to received wisdoms • Challenges and strengthens a policy’s theory of change • Provides more precise estimates of likely outcomes/effects • Provides valid and reliable evidence on implementation • Transparent strengths and weaknesses of evidence-base

  12. Rapid Evidence Assessments - Limitations • Usually not comprehensive use of existing literature • Introduces selection and publication biases • Can misrepresent the totality of evidence • Can lead to Type I and Type II errors • Need to be continued to produce full-blown systematic reviews • Hence Interim Evidence Assessments to be preferred

  13. Policymakers’ Views of Academic Research For Policymakers Research Evidence Is Too: • Long • Verbose • Detailed • Dense • Impenetrable • Jargonesque • Methodological • Untimely • Non-relevant/irrelevant

  14. Formatting and Presentational Issues for S-Rs Some Solutions • Identify the key message(s) from research • Identify limitations of evidence • Communicate clearly and concisely • Use appropriate format (eg 1:3:25) • Don’t patronise/over-simplify/dumb-down • Don’t ‘blind with science’ • Persistence and Opportunism

  15. Sources of Systematic Review Evidence • Centre for Evidence-Informed Policy and Practice (EPPI) http://eppi.ioe.ac.uk/ • NHS Centre for Reviews and Dissemination (CRD) http://www.york.ac.uk/inst/crd • National Institute of Clinical Excellence (NICE) • www.nice.org.uk/ • Social Care Institute of Excellence (SCIE) www.scie.org.uk • Campbell Collaboration http://campbellcollaboration.org • Cochrane Collaboration www.cochrane.org • ESRC Centre for EBPP and the Evidence Network http://www.evidencenetwork.org

  16. Contact philip.davies@hm-treasury.x.gsi.gov.uk Government Social Research Unit HM Treasury 1 Horse Guards Road London SW1A HQ England Tel: +44 (0)20 7270 5156 www.policyhub.gov.uk

More Related