Catrin Eames Bangor University c.eames@bangor.ac.uk

1. The Importance of Treatment Fidelity in Delivering Parent ProgrammesPresentation for the Incredible Years Wales ConferenceLlandrindod Wells, 7th February 2008 Catrin Eames Bangor University c.eames@bangor.ac.uk Supervisory committee: Dr Judy Hutchings, Dr Carl Hughes, Dr David Daley Funded in collaboration by ESF/Objective 1 fund and Incredible Years Wales

2. Overview of presentation • The importance of treatment fidelity • History of treatment fidelity research • Current Ph.D. research

3. What is fidelity? • Fidelity may be referred to as the degree to which intervention delivery adheres to the original intervention protocol (IOM, 2001).

4. Fidelity research terminology • Adherence • Treatment delivery • Treatment differentiation • Treatment exposure • Treatment integrity • Participant responsiveness

5. Effective delivery • Adequate evidence base • Methods used • Means of assessing fidelity

6. History of treatment fidelity research • Poor methodologies • Surge of treatment manuals • Manuals alone do not guarantee fidelity • Few studies reported any fidelity issues • Of those that did, no reference to the development of valid fidelity measures

7. The importance of fidelity • It is necessary to implement all components of a programme fully in order to preserve the behaviour change mechanisms that made the original programme model a successful one (Arthur & Blitz, 2000; Mihalic et al., 2002) • Can offer a template of service delivery • Further resources/supervision/training • Ensure adherence to delivery of intervention models • Programme drift most common reason for poor treatment outcomes (Mills & Ragan, 2000; Mowbray et al, 2003)

8. Current fidelity measures • Typically, manuals or checklist format • Measure programme content • Threats to validity

9. Benefits of observation • Precise account of behaviour as it unfolds • Bias reduced with independent observation • Continuous sampling, quantitative

10. Rationale for current research • IY : Internationally delivered and evaluated • Over 10,000 trained in the BASIC • Numbers of independent replications increasing • Quantitative observational measure of process skills • Ensure measure is reliable and valid • Relate leader fidelity scores to changes in parent and child behaviour from the Sure Start Study

11. Leader Observation Tool (LOT) • Objective frequency count • Continuous time sampling at 10-minute intervals throughout the 2-hour session • Both leaders coded • 18 behaviour categories forming five skills subgroups: • Listening • Empathy • Physical Encouragement • Positive Behaviour • Other/Negative

13. Data set • 12 groups, two leaders per group, 2 hour sessions • Course split up into : • Play and relationship building • Praise and reward • Effective limit setting • Handling misbehaviour • Four sessions selected for coding for each group • Of these, 30% randomly selected for second coding • Coders blind to behaviour change scores to avoid bias

14. LOT reliability and validity • Internal reliability at both p<.05 and p<.01 • 84% average inter-rater agreement • Intra-class correlations of each category: mean of .92 and .95 for code-recode and inter-rater reliability respectively • Concurrent validity with parent and leader reports of the group intervention process • LOT identified variability between groups, highlighting the heterogeneity in process skills and treatment fidelity

15. LOT scores and behaviour change • Leaders attributed total score per subgroup • Parent data = 86 completers • Each parent attributed a leader fidelity exposure score • Predict that probability of parent behaviour change greater when attending a group delivered with high fidelity • Greater change in parent behaviour in turn predict greater change in child behaviour

16. Measures Leader  Parent  Child • Leader: LOT • Parent: Parenting Scale, Positive Parenting • Child: ECBI, Child positives, Compliance

17. Results

18. Therefore…. • In order to change child behaviour, need to change parenting behaviour • Leader treatment fidelity of process skills significantly predict both observed change in parenting behaviour as well as parent self reported behaviour • In turn, predict the change in child behaviour

19. Conclusion • Need to report fidelity criteria in treatment outcome studies, especially considering the number of manual based interventions that are replicated in the community today • Poor implementation fidelity is the most common factor for poor treatment outcome (Mills & Ragan, 2000) • Valid fidelity measurement can predict treatment outcome, as well as document treatment differentiation in multi-centre research

20. Thank you for listening For further information please contact: Catrin Eames School of Psychology College of Health and Behavioural Sciences Bangor University Gwynedd LL57 2DG 01248 38 2651 c.eames@bangor.ac.uk