1 / 30

Systematic Literature Searching: Preparing a Campbell Review Protocol

Systematic Literature Searching: Preparing a Campbell Review Protocol. Philip Curry Leyla De Amicis Robbie Gilligan. Overview. Background to review and introduction to systematic reviewing (Philip)

kaminskic
Download Presentation

Systematic Literature Searching: Preparing a Campbell Review Protocol

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Systematic Literature Searching: Preparing a Campbell Review Protocol Philip Curry Leyla De Amicis Robbie Gilligan

  2. Overview • Background to review and introduction to systematic reviewing (Philip) • Protocol on the ‘Effects of cooperative learning on inter ethnic relations in school settings’ (Leyla)

  3. Origin of Review • The Children, Youth and Community relations project • Intensive qualitative work in seven primary schools in Dublin North inner city • Problematic issues identified early on: • Racism • Name-calling • Bullying • General lack of social interaction between migrant and local children

  4. Such behaviours generally occur in situations which are by their very nature beyond adult supervision. • What can be done? Ad hoc knowledge=>Review=>Systematic review • IRCHSS RDI grant

  5. Systematic Review • Family of techniques used to bring together as much evidence as possible on a topic and synthesize it. • Often based on quantitative analysis [‘meta-analysis’] of experimental interventions but not necessarily so. • Goal is to limit bias in the identification, evaluation and synthesis of the body of relevant studies.

  6. Biases • Outcome bias • Publication bias • Multiple publication bias • Database bias • Availability bias • Familiarity bias • Cost bias • Language bias • Confirmation bias • Citation bias

  7. Each line represents a different report on the same piece of research…

  8. Systematic versus traditional or narrative review • More systematic approach to identifying evidence • More transparent and explicit procedures • Allow us to rigorously examine differences due to study methods, types of participants, settings, etc.

  9. Systematic versus traditional or narrative review (Cont.) • Where data is quantitative focus on the magnitude of effects rather than statistical significance • Statistical significance just indicates a non-zero effect, could be big, moderate or very small • Two results that are statistical significant may in fact represent very different effects • Especially to be avoided is the practice of ‘counting’ significant results

  10. The Campbell Collaboration (C2) • An international voluntary research network founded in 2000 • In the social sciences meta-analysis had grown dramatically since the 70’s not only in usage but also in method • C2 can be thought of as embodying best practice in this area in a very concrete fashion • Sister to the Cochrane (C1)

  11. Produce systematic reviews on the effects of interventions in the areas of crime and justice, social welfare and education. • Two other groups within C2 work on: • Developing the methodology of systematic reviewing • Engaging organizations with responsibility for translating research into practice and policy [Users group].

  12. Stages of a review • Title registration Potential reviewers prepare a short (1 to 2 page) document outlining definition of the intervention and issue to be reviewed, status of existing reviews and plan for methodology. C2 comment on this and return for revision. Published on C2 website.

  13. 2. Protocol Detailed document outlining procedures : • Discussion of intervention and existing reviews on the topic. Why does the review need to be done? • Criteria for inclusion/exclusion of studies • Search strategy • Details of study coding categories • Statistical procedures and conventions Protocol is send to C2 and reviewed by 2 reviewers

  14. 3. Undertaking the review • Two reviewers and coders follow the protocol to locate and code the studies identified. Third reviewer acts as arbiter. • Meta-analysis or synthesis – C2 provide support.

  15. 4. Publishing the review Review is sent to C2 for review. Published by their own and related websites. Authors usually publish separately as well.

  16. Typical results: CCTV and crime • Modest positive • effects • Analysis then shows that CCTV most effective: • - Car parks • - Car crime • - The UK

  17. 5.a. Protocol: Starting from the title Effects of cooperative learning strategies (NOTintervention programs) on inter ethnic relations in school settings • More manageable amount of material • More reasonable for our timeframe • More homogeneous material • Cooperative learning has a solid theoretical- and empirical-based tradition • Micro-level analysis (more useful for practical applications)

  18. 5.b. Protocol: Background Previous reviews are: • no-systematic (e.g., Cooper & Slavin, 2004) • no-quantitative (e.g., Paluck & Green, 2008) • no-update (e.g., Johnson & Johnson, 1989) • too specific (e.g., Bratt, under review) • too generic (e.g., Pettigrew & Tropp, 2006) • conflicting results (e.g., Cooper & Slavin, 2004)

  19. 5.c. Protocol: Criteria for inclusion/exclusion of studies • Cooperative learning defined as school based instructional practices applied to 11 approaches (e.g.,Team – Games -Tournament). • Small group work • Tasks and activities carefully planned in advance • Interdependence among team members

  20. 5.c. Protocol: Criteria for inclusion/exclusion of studies • All school age groups (4 to 18 years of age) • Primary outcome variable: children’s behaviour/attitudes/ emotions/beliefs toward at least one ethnic, religious, linguistic out-group • Excluded other stigmatised groups (e.g, disabled children) • Secondary outcome variables: school achievement performance and self-esteem

  21. 5.c. Protocol: Criteria for inclusion/exclusion of studies • Research design: randomised field experiments and quasi-experiments. Including pre-test and control group/s • Studies after 1965 • Studies conducted anywhere in the world and reported in any language

  22. 5.d. Protocol: Search strategy • A variety of methods and sources to identify studies • Developing procedures to reiterate for systematically minimizing bias and maximizing the amount of relevant material

  23. 5.d. Protocol: Search strategy • Keyword search on several online electronic databases (e.g., 15 among which C2-SPECTR, PsychInfo, Chinese-ERIC, Web of Knowledge, Dissertation Abstracts, ISI Citation Indexes) • Search references from previous reviews and pertinent material • Hand-search journal (7 among which Educational Psychology Review, Journal of Social Issues, Journal of Community & Applied Social Psychology)

  24. 5.d. Protocol: Search strategy • Key-words found in relevant studies and thesauri • Multiple terms on intervention, context, target population as topic and subject in titles and abstracts • Different combinations of specific key- words (e.g., cooperat* AND racial AND school*) • Authors (e.g., Cohen*) and specific intervention programs used as key-terms

  25. 5.d. Protocol: Search strategy • Examples for intervention terms: collaborat*/ small-group-teaching/ Learning Together • Examples for context: integrat*/ rac*/ minorit*/ tension/ prosocial/ bias/ prejudice/ cultur*/conflict* • Examples of target populations: child*/ grade*/ pupil/ preschool*/ adolescen*

  26. 5.d. Protocol: Search strategy • Key internet search (Google scholar, Yahoo) • Contact leading scholars in this area

  27. 5.d. Protocol: Search strategy • Government institutions/ministries- Departments of Education/Youth Affairs/Integration for OECD countries - Government reports online (www.gpoaccess.gov) • Professional specialised agencies (e.g., American Evaluation Association, Society for Research on Child Development) - Conference proceedings • Non-profit organisations for human rights, children and immigration issues (e.g., UNICEF, SAVE THE CHILDREN)

  28. 5.d. Protocol: Search strategy Limits • Key words: old-fashioned/disagreement of labels among researchers • Reference lists: non accurate/omissions/not provided • Hand-search: cost bias • Internet sources: language • Scholars: availability/bias • Government resources/Non-profit organisations: time-consuming/quality/ language

  29. 5.e. Protocol: Details of study coding strategies • Reference information (e.g., title, journal) • Intervention characteristics (e.g., type, inter ethnic context) • Methodology characteristics (e.g., design, control group) • Statistics information (e.g., effect size) 5. f. Protocol: Statistical procedures and conventions • Analyses to control for bias (e.g., publication bias) • Analyses for testing research questions (ANOVAs and regressions)

  30. Conclusion • Systematic reviewing is extremely, extremely labour intensive and time consuming. • Not to be undertaken lightly or without resources. • There are general lessons about the importance of being systematic in how we evaluate evidence and conduct literature searches. http://www.campbellcollaboration.org/ http://www.tcd.ie/immigration/community/index.php

More Related