1 / 21

Extensions to traditional statistical meta-analysis James Thomas

Extensions to traditional statistical meta-analysis James Thomas Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10 th July 2014. EPPI-Centre Social Science Research Unit Institute of Education University of London

briar
Download Presentation

Extensions to traditional statistical meta-analysis James Thomas

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Extensions to traditional statistical meta-analysis James Thomas Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St Catherine’s College, Oxford, 10th July 2014 EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC1H 0NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe.ac.uk Web eppi.ioe.ac.uk/ The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London

  2. Outline • What is meta-analysis, and why are ‘extensions’ needed? • What is complexity? And why is it a challenge in systematic reviews? • Extensions to traditional meta-analysis when complexity is encountered • Acknowledgement: presentation draws on: O’Mara-Eves A, Thomas J (2013) Methods for configurational synthesis: extensions to traditional meta-analysis for addressing intervention complexity and contextual variation in reviews. 21st Cochrane Colloquium: Quebec 19-23 September

  3. The common stages of a systematic review; focus here on synthesis Form review team (involve ‘users’) Formulate review question, conceptual framework and inclusion criteria (develop ‘protocol’) Search for and identify relevant studies Describe studies Assess study quality (and relevance) Synthesise findings Communicate and engage Map Synthesis

  4. Traditional meta-analytic models • Typically are used to address three key research questions: • What is the overall estimate of the size of the effect and its precision? • Is there heterogeneity across the study effects? • What (if any) variables explain differences across the study effects (if heterogeneity is present)?

  5. Aggregative approaches in research Aggregative reviews predominately add up (aggregate) findings of primary studies to answer a review question… … to indicate the direction or size of effect

  6. RCT forest plot: Does children’s participation in structured arts activities improve their cognitive learning outcomes? Newman M, Bird K, Tripney J, Kalra N, Kwan I, Bangpan M, Vigurs C (2010) Understanding the impact of engagement in culture and sport: A systematic review of the learning impacts for young people. London: Department for Culture, Media and Sport. http://culture.gov.uk/images/research/CASE-systematic-review-July10.pdf

  7. Are we limiting ourselves? • Current popular meta-analytic methods are limited to: • Questions of differences between two groups or correlations or variables • One causal / relational proposition at a time • One outcome at a time

  8. Whose questions are we addressing? • Policymakers and practitioners usually do not ask a single narrow aggregative question • They begin with a particular problem and ask “what is the best way to achieve outcome X?” • They also ask “does it vary according to…?” and “What does X mean to Y?” • And they ask these questions relating to complex problems -

  9. A complex intervention • Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome.” • Craig Pet al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337 Some would say the above is merely complicated…

  10. Complicated and complex • Truly complex interventions are best conceptualised as dynamic processes • Virtuous circles • Feedback loops • Non-linear step changes in responses / outcomes • Multiple ‘routes’ to effectiveness • Rogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008;14(1):29-48

  11. Complex reviews of social research: • Start from a given ‘problem’ • often a known outcome and population • Aim to identify a range of possible ‘solutions’ • Rarely aim to come to a single answer • Acknowledge that there are rarely replications of interventions in social research • Use heterogeneity to better understand factors that influence the impact of interventions • Contain detailed and complex conceptual frameworks (programme theories etc) • Soto configure findings (as well as aggregate)

  12. Configurative approaches in research • Configurative reviews predominately arrange (configure) the findings of primary studies to answer the review question…. • … to offer a meaningful picture of what research is telling us

  13. Other tools in our toolbox • Sub-group analysis • Meta-regression • Network meta-analysis • Multivariate meta-analysis • Path analysis • Factor analysis • Qualitative comparative analysis • …?

  14. Subgroup analysis DiCenso et al (2002) BMJ;324:1426

  15. Meta-regression Catalá-Lópezet al. BMC Psychiatry 2012, 12:168

  16. Network meta-analysis • Facilitates an estimate of the relative effectiveness of interventions – even when they have not been directly compared with one another in a trial Thorlund and Mills Systematic Reviews 2012, 1:41

  17. Problems with ‘traditional’ approaches • The above approaches work well in some situations, BUT • There are rarely replications of complex interventions (is it possible to have a genuine replication?); leads to a lack of data • Even when an analysis has many studies, interventions, contexts etc. all differ • Lots of unexplained heterogeneity • Symmetrical nature of correlational analysis

  18. Complexity and correlational analysis • Correlation is symmetric • When testing for a connection between cause and effect, also tests equally for absence of cause and absence of effect • Correlation therefore cannot detect multiple causal pathways* • E.g. Asserting that ‘interventions which are delivered by peers tend to be effective’ should not require that those not delivered by peers are not effective * There are usually too few studies to use interaction variables

  19. Another approach: qualitative comparative analysis (QCA) • Originally developed by Charles Ragin in political science and historical sociology • Not a correlational approach • Useful for small numbers of studies • Focus can be on combinations of intervention components • More inductive mode of analysis than above Suggests that intensity & quality are sufficient to gain effective outcome Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67

  20. Summary • Meta-analysis can address a single focused question, requiring a straightforwardly aggregative answer • “Real world” questions tend to require configuration AND aggregation • Extensions to meta-analysis are able to configure and aggregate study findings successfully • But few current methods can cope with genuinely complex situations

  21. Thank you for your attention Websites EPPI-Centre Website http://eppi.ioe.ac.uk Twitter @James_M_Thomas @EPPICentre Email j.Thomas@ioe.ac.uk EPPI-Centre Social Science Research Unit Institute of Education University of London 18 Woburn Square London WC1H 0NR Tel +44 (0)20 7612 6397 Fax +44 (0)20 7612 6400 Email eppi@ioe.ac.uk Web eppi.ioe.ac.uk/ The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London

More Related