Building & Reporting District & Program-Level Evaluation of PBIS

1. Building & Reporting District & Program-Level Evaluation of PBIS Lead Presenter: Jake Olsen, PhD Exemplars: AIMEE CLARK & Lynn Nord Key Words: Applied Evaluation, Assessment, Training3 National PBIS Leadership Forum – October 4-5, 2018 – Chicago, IL

2. Session Objectives • Gain a clear understanding of the process and practices involved in PBIS program evaluation • Learn about evaluation tools and how to use these tools in your local context • Gain insight into the benefits and challenges of district level PBIS program evaluation from the perspective of practitioners

3. Maximizing Your Session Participation When Working In Your Team • Consider 4 questions: • Where are we in our implementation? • What do I hope to learn? • What did I learn? • What will I do with what I learned?

4. Where are you in the implementation process?Adapted from Fixsen & Blase, 2005

5. Leadership Team Action Planning Worksheets: Steps • Self-Assessment: Accomplishments & Priorities Leadership Team Action Planning Worksheet • Session Assignments & Notes: High Priorities Team Member Note-Taking Worksheet • Action Planning: Enhancements & Improvements Leadership Team Action Planning Worksheet

6. Please Provide Feedback Your feedback is important to us! Please take a few moments at the end of the session to complete an evaluation form for this session. Forms are available: • In our mobile application by clicking the link in the session description. • Online underneath the posted presentations at www.pbis.org/presentations/chicago_forum_17 • Via paper form from your session facilitator

7. Purpose of PBIS Program Evaluation • Determine impact of systems and practices • Document implementation process • Are goals being met? • Are appropriate activities being used? • Is process being implemented as intended? • Is process successful? (Algozzine et al., 2010)

8. Evaluation Process and Practices (Algozzine et al., 2010)

9. Core Features of Program Evaluation (Algozzine, 2015)

10. Evaluation Tools • Tiered Fidelity Inventory • Measures extent to which school personnel are implementing core features of Tiers I, II, and III of PBIS • Intended for evaluation every 3rd or 4th team meeting • Based on existing evaluation tools (SET, BoQ, TIC, SAS, BAT, MATT) • Efficient, valid, reliable • Also has walkthrough component, observation form, action planning form

11. Reporting Evaluation Results • Considerations • Breadth and depth determined by: • Needs of audience • Length of PBIS implementation • Complexity of results data/information (Algozzine et al., 2010)

12. Reporting Evaluation Results • Considerations • Audience • Most appropriate timing • Effective communication/writing styles • Appearance of reports • Accuracy, balance, fairness • Nature of information (positive, negative, neutral) (Algozzine et al., 2010)

13. Reporting Evaluation Results (Fitzpatrick et al., 2011)

14. Reporting Evaluation Results Written Report Recommended Content • Executive summary • Purpose, timeframe, summary • Vision, mission, foundations of SWPBIS program • Current implementation efforts • Extent of implementation • Outcomes • Capacity • Implications and next steps • Supporting materials (Algozzine et al., 2010)

15. Reporting Evaluation Results • Oral Reporting Recommendations • Determine the story you want to tell • Decide who should tell the story • Consider varying the medium used in order to engage audience • Use visuals to accompany oral reporting • Involve the audience • Develop and adhere to an agenda

16. Exemplar Hemet Unified School district Hemet, ca

19. Plan Identify purpose, team, and timeline

20. Purpose • Decrease office discipline referrals • Reduce suspensions and expulsions • Increase school attendance • Increase positive perceptions among district students and staff regarding school climate • Implement all three tiers of the PBIS framework district wide • corresponds with HUSD’s goals from the

21. Team • Student Support Services Director • Grant Project Director • Two District PBIS Coaches • District Intervention Counselor • District MFT • District MH Clinical Coordinator

22. Timeline

23. Timeline

25. Define Identify questions to be answered

26. Evaluation Questions • How can HUSD reduce discipline referrals, suspensions, and expulsions? • How can the district improve systems to promote environments that are conducive to learning? • How can HUSD improve school climate, safety, and connectedness to improve academic achievement?

27. Do Select measures and gather data to answer questions

28. Tiered Fidelity Inventory (TFI)

30. Data Gathering TFI Walk-Through Schedule February 5-16, 2018

33. Data Gathering • Self-Assessment Survey (SAS) • Self-assessment tool to determine to what extent PBIS practices and systems are in place within a school • Four systems: School-wide, Non-Classroom settings, Classroom, Individual Student • Goal: 80% In Place School-wide

34. Document Analyze data and answer questions

38. Self Assessment Survey (SAS)

39. Answering Evaluation Questions • Through the process of implementing PBIS district-wide: • Worked with the student information system Aeries to create a PBIS Dashboard • Reduced ODRs, suspensions, and expulsions. • Identified a process to drill down and look at student groups to better understand the issue of disproportionality • Improved systems of proactive supports for students through prevention and education

40. Answering Evaluation Questions • How can HUSD reduce discipline referrals, suspensions, and expulsions? • Reduced ODRs, suspensions, and expulsions by implementing PBIS with fidelity at Tiers 1, 2, 3 • How can the district improve systems to promote environments that are conducive to learning? • Worked with the student information system Aeries to create a PBIS Dashboard • Identified a process to drill down and look at student groups to better understand the issue of disproportionality • Improved systems of proactive supports for students through prevention and education • How can HUSD improve school climate, safety, and connectedness to improve academic achievement?

41. Program Evaluation Reporting Audience, benefits and challenges

42. Audience

46. Benefits of Evaluation and Reporting District Level Student Outcomes • Accountability • Concise mechanisms for sharing project outcomes to garner political support • Progress monitoring and identifying needs for refinement

47. Challenges of Evaluation and Reporting Time Change in district leadership and direction Administrative involvement and support Consistent use of evaluation tools Inconsistencies between state and federal reporting requirements Absence of data correlating academic and behavior outcomes Competing with other evaluation priorities

48. Q & A • General questions • Guiding questions • What tools do you use to evaluate implementation? • How do you report evaluation results? • How do you address challenges to evaluation and reporting?

49. Resources • PBIS Program Evaluation Examples https://www.pbis.org/evaluation/evaluation-examples • Evaluation within PBIS Implementation Practice Brief http://www.pbis.org/presentations/chicago-forum-16 • Evaluation Blueprint for SWPBS http://www.pbis.org/blueprintguidestools/blueprint/evaluation-blueprint • PBIS APPS-Tiered Fidelity inventory https://www.pbisapps.org/Applications/Pages/PBIS-Assessment-Surveys.aspx#tfi • DCA Resource http://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-DCA-Brief-05-22-2017.pdf

50. Contact Information • Jake Olsen, California State University Long Beach jacob.olsen@csulb.edu • Aimee Clark, Hemet Unified School District • aclark@hemetusd.org • Lynn Nord, Hemet Unified School District • lnord@hemetusd.org