Ensuring Effective Treatment and Supervision

1. Ensuring Effective Treatment and Supervision Operationalizing Fidelity in Real World Settings Kimberly Gentry Sperber, Ph.D.

2. Workshop Objectives • Review literature on fidelity • Identify opportunities for assessing fidelity • Identify resources for monitoring and improving fidelity • Address barriers to monitoring fidelity • Begin to think through implementation plan

3. Objective 1 What Does Science Say About Fidelity?

4. Latessa, Cullen, and Gendreau (2002) • Article notes 4 common failures of correctional programs: • Failure to use research in designing programs • Failure to follow appropriate assessment and classification practices • Failure to use effective treatment models • Failure to evaluate what we do

5. The Role of Fidelity • Lowenkamp and Latessa (2005) • Examined data from 38 residential correctional programs for adults • Looked at relationship between program fidelity and program effectiveness. • Program fidelity was assessed using the CPAI. • Found significant correlation between fidelity and effectiveness • CPAI scores correlated to reincarceration

6. Lowenkamp and Latessa FindingsContinued • Differences in recidivism rates based on CPAI scores: • Scores of 0-49% demonstrated 1.7% reduction compared to comparison group. • Scores of 50-59% demonstrated 8.1% reduction. • Scores of 60-69% demonstrated 22% reduction.

7. CPAI Data Continued • Holsinger (1999) • Examined data from Adolescent Community Correctional Facilities in Ohio • Looked at relationship between program fidelity and program effectiveness. • Program fidelity was assessed using the CPAI. • Outcome measures examined included any court contact, felony or misdemeanor, felony, personal offense, and commitment to a secure facility

8. CPAI Data Continued • Total composite score significantly correlated with all outcome measures. • Each individual domain of the CPAI also significantly correlated with all of the outcomes • Program Implementation • Client Assessment • Program Characteristics • Staff Quality • Evaluation

9. More Fidelity Research • Landenberger and Lipsey (2005) • Brand of CBT didn’t matter but quality of implementation did. • Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers. • Sexton (2001) • Direct linear relationship between staff competence and recidivism reductions. • Schoenwald et al. (2003) • Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients. • Henggeler et al. (2002) • Supervisors’ expertise in the model predicted therapist adherence to the model.

10. More Fidelity Research Cont’d. • Schoenwald and Chapman (2007) • A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatment • A 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.

11. Washington State Example(Barnoski, 2004) • For each program (FFT and ART), an equivalent comparison/control group was created • Felony recidivism rates were calculated for each of three groups, for each of the programs • Youth who received services from therapists deemed ‘competent’ • Youth who received services from therapists deemed ‘not competent’ • Youth who did not receive any services (control group)

12. Functional Family Therapy Results: % New Felony Results calculated using multivariate models in order to control for potential differences between groups

13. Washington State Study Continued • When FFT was delivered competently, the program reduced felony recidivism by 38% • When considering how much the program costs, substantial savings in ‘avoided crime’ were observed – particularly for the competent therapists • When ART was competently delivered, felony recidivism was reduced by 24% • Also resulted in substantial savings

14. Project Greenlight • Short-term prison-based reentry program in New York • CBT Skills Training • Employment Services • Housing Services • Drug Education and Awareness • Family Counseling • Practical Skills Training • Community-Based Networks • Familiarity With Parole • Individualized Release Plans

15. Project Greenlight Benefits • Participants received more service referrals • Participants reported more contacts with community services after release • Participants demonstrated significantly more familiarity with parole conditions • Participants were more positive about parole

16. But Did It Work?

17. What Went Wrong? • Violation of the risk principle • Ceased use of risk assessment instrument when staff deemed process too cumbersome • Violation of the need principle • All offenders received same services whether needed or not • Violation of the fidelity principle • Staff modified delivery of the CBT curriculum (shortened the duration, increased frequency, increased class size) • Differential staff competence • Certain case managers produced worse outcomes

18. 2010 UC Halfway House/CBCF Study in Ohio:Adherence to CBT in Groups and Changes in Recidivism

19. Fidelity and the Validity of the LSI-R • Flores, Lowenkamp, Holsinger, & Latessa, (2006) • Higher correlations among scores produced by trained staff and future incarceration • Lower correlations among scores produced by untrained staff and future incarceration • Higher correlations among agencies using the LSI-R for at least 3 years and future incarceration • What are the implications of these findings?

20. What Do We Know About Fidelity? • Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability). • Poor fidelity can lead to null effects or even iatrogenic effects. • Fidelity cannot be assumed • Fidelity can be measured and monitored.

21. Relationship Between Evaluation and Treatment Effect (based on UC Halfway House and CBCF study)

22. NPC Research on Drug Courts

23. Objective 2 Opportunities for Assessing Fidelity

24. Opportunities to Monitor Fidelity • Training • Assessments • Treatment groups • Individual sessions • Case Management • Milieu • Documentation Review • Program Assessments

25. Ensuring Training Transfer • Use of knowledge-based pre/post-tests • Use of knowledge-based proficiency tests • Use of skill-based rating upon completion of training • Mechanism for use of data • Staff must meet certain criteria or score to be deemed competent. • Failure to meet criteria results in consequent training, supervision, etc.

26. Assessments • Desktop Reviews • Accurate scores • Reviews of overrides • Integration with service plan/dosage • Observations • Use of standardized audit sheet • Assess interviewing skills • Assess accuracy of item ratings

27. Treatment Groups • Observation-based ratings of adherence to treatment model. • CBT: • Frequency of role-plays • Structure of role-plays • Appropriateness of role-plays • Use of behavioral reinforcers • Effective use of authority and disapproval • Teaches the thought-behavior chain • Teaches structured skill building • Follows curriculum

28. Individual Sessions • Observation-based ratings of adherence to treatment model. • CBT: • Teaches thought-behavior chain • Teaches problem-solving • Teaches structured skill building • Conducts role-plays • Appropriate use of thinking reports/homework • Graduated practice • Appropriate use of reinforcers

29. Case Management • Observation-based ratings of adherence to treatment model. • CBT: • Teaches thought-behavior chain • Teaches problem-solving • Teaches structured skill building • Appropriate use of reinforcers • Helps client to integrate skills learned into real world environment (e.g., employment)

30. Milieu • Observation-based ratings of competence in core correctional practices • Focus is more on effective use of authority and disapproval and appropriate use of reinforcers and sanctions. • Standardized list of behavioral indicators • Structure for observing and rating staff interacting with clients in milieu • Can also review incident data for trends

31. Documentation Review:Why Do It? • Clinical Implications • Documentation is not separate from service delivery. • Did the client receive the services he/she needed? • Operational Implications • Good documentation should drive decision-making. • Means of communication • Risk Management Implications • If it isn’t documented, it didn’t happen. • Permanent record of what occurred in the facility • Source of Staff Training • Reflection of the provider and organization’s competency: • EBP • Outcome of care

32. Program Assessments • Correctional Program Checklist (CPC) • Correctional Program Assessment Inventory (CPAI) • ICCA Treatment Survey

33. Sample Measures • Percentage of groups containing role-plays • Percentage of successful completers receiving appropriate dosage based on risk/needs assessment • Percentage of staff achieving 4:1 ratio • Percentage of groups observed where staff modeled the skill prior to having clients engage in role-play • Percentage of role-plays containing practice of the correctives • Percentage of role-plays that required observers to identify skill steps and report back to the group

34. Objective 3 Resources Required to Monitor and Improve Fidelity

35. Observation-Based Ratings • Creation of audit sheets • Schedule for conducting the reviews • Staff qualified to conduct and rate the observations • Time for staff to conduct observations • Mechanism to record and use the data • Supervision and individual staff development • QI and training initiatives

36. Documentation Review • Staff to conduct the review • Schedule for review rotation • Audit sheet • Time to conduct the review • Mechanism for recording and using the data • Action planning

37. Objective 4 Barriers to Monitoring Fidelity

38. Common Barriers • Strength of conceptual understanding of the EBP to be measured • Resources • Setting priorities • Understanding/skill sets required for measurement • Conflicting philosophies (helper vs. evaluator) • Time!

39. Potential Strategies • Start small • For example, desk top review of assessments versus observation-based ratings • Use technology to increase efficiencies • For example, videotape interactions for observation-based ratings • Take the time to build expertise • Train on model • Train on evaluation methodology • Insure understanding of purpose (e.g., QI versus punishment)

40. Objective 5 Planning for Implementation

41. Considerations • What services do you say you deliver? • What does your contract say? • What do referral sources expect? • List all programming components • What is the model? • What curricula are in use? • Identify which component of service delivery is most important to monitor at current time • Make a selection for measurement • Identify the specific component of the model to evaluate (e.g., CBT groups in a CBT model)

42. Considerations • Investigate possible methods for measuring the component you have selected • Look for existing tools • Guidelines for measurement tools: • Scale should adequately sample the critical ingredients of the EBP • Need to be able to differentiate between programs/staff that follow the model versus those that don’t • Scale should be sensitive enough to detect progress over time • Need to identify who will be observing, measuring, and documenting • Need to identify how results will be calculated • For example, if a percentage, what is numerator and what is denominator • Define frequency of observations and who will be observed

43. Considerations • Decide how data will be coded, stored, aggregated, reported • Determine responsible parties for all parts of the process • Determine how data are to be used (e.g., are there expectations that program staff will document improvement plans) • Agree on common definitions for key terminology contained in measurement tools (e.g., modeling) • Determine training needs for assigned evaluators

44. Considerations • Create training package for all staff involved • To address with all staff: • Defining fidelity, outlining importance, details of fidelity initiative, who’s impacted and how, expectations for all staff • To address with evaluators: • Clarify role of evaluation and how this might differ from their assigned operational role in the program • Define all terminology contained in measurement tools/process • Agree on what constitutes evidence that staff are utilizing the model correctly • Specify how they are to document results and any responsibilities they have for data coding, storage, etc.

45. Conclusions • Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model. • There is evidence that fidelity directly affects client outcomes. • There is evidence that internal evaluation processes directly affect client outcomes. • Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s. • Requires a formal infrastructure to routinely monitor fidelity performance.

46. Questions and Answers Contact Information: kimberly.sperber@talberthouse.org