Performance Goals and Indicators for PTIs: Using Data to Guide Improvement Efforts

1. Performance Goals and Indicators for PTIs: Using Data to Guide Improvement Efforts Based on Materials Prepared by the OSEP TA&D Performance Measurement Work Group Matrix, West Region PTI Meeting July 25-27, 2003, Portland, OR

2. My Self-Imposed Charge • MATRIX wanted a talk about PGs&Is… • First title: “Performance Goals & Indicators for Parents” • My Final Title: “Performance Goals and Indicators for PTIs: Using Data to Guide Improvement Efforts” • Theme: If PTIs partner with SEAs, LAs, & others in supporting improved results for children with disabilities, what might that mean? What would PTIs need to know?

3. Five Key Points • PTI Dual Role: helping ensure rightsandresultsforindividualchildren; helping ensuresystemsperformaccountablyforallchildren. • Shared Accountability: PTIs share responsibility for system accountability. • Consistent Language: Common language will help avoid confusion among partners. • Evidence-Based Decision Making: Data aren’t decisions, but good decisions require good data. • Improvement Resources: Communities of Practice, the TA&D network, SEAs and LAs.

4. The Role: Congress finds… “Systemic change benefiting all students, including children with disabilities, requires the involvement of States, local educational agencies, parents, individuals with disabilities and their families, teachers and other service providers, and other interested individuals and organizations, to develop and implement comprehensive strategies that improve educational results for children with disabilities.” (in both HB1350 and S. 1248)

5. FOCUSED MONITORING ‘‘The primary focus of Federal and State monitoring activities … shall be on improving educational results and functional outcomes for all children with disabilities, while ensuring compliance with program requirements, with a particular emphasis on those requirements that are most closely related to improving educational results for children with disabilities.” S. 1248, SEC. 616

6. ED/OSEP State Local Eligibility/ Resources Eligibility/ Resources Eligibility/ Resources Policies/ Procedures Policies/ Procedures Policies/ Procedures Support Improved Systems Support Improved Programs Support Improved Instruction Evaluation Evaluation Evaluation Partnership for Improvement: Aligned Functions Improved Outcomes for Children and Families Support from TA&D Network…

7. Federal Statutes Related to the Use of Data in Improvement Efforts • Government Performance and Results Act of 1993 (GPRA) • No Child Left Behind Act of 2001 (NCLB) • Individuals with Disabilities Education Act of 1997 (IDEA)

8. IDEA 97: 34 CFR 300.137Performance goals and indicators The State must have on file with the Secretary information to demonstrate that the State- (a) Has established goalsfor the performance of children with disabilities in the State that - (1) Will promote the purposes of this part, as stated in §300.1; and (2) Are consistent, to the maximum extent appropriate, with other goals and standards for all children established by the State; (b) Has established performance indicators that the State will use to assess progress toward achieving those goals that, at a minimum, addressthe performance of children with disabilities on assessments, drop-out rates, and graduation…

9. IDEA’s Purpose Paraphrased = Improved Results • From 34 CFR 300.1 - Overall Purpose [Parts B, C, & D]: • Prepare children for employment & independent living • Protect the rights of children & parents • Assist state, local, intermediate & federal agencies provide services • Ensure effectiveness of efforts to educate children with disabilities From USC 1431(a) [“Congressional Findings” for Part C]: • enhance development & minimize delay; reduce costs by minimizing need for special ed.; minimize institutionalization; maximize potential for independent living; enhance family capacity; enhance providers’ capacities to meet the needs…of infants & toddlers with disabilities. • From 34 CFR 303.1 [Part C’s Purpose]: • Statewide systems of early intervention for infants and toddlers and their families, coordinated payment from all sources, & special focus on historically underrepresented populations

10. Accountability (IDEA & NCLB) • Performance Reports (Part B & C) • SEA Public Reporting (NCLB) • External Context/Influence: • OSEP Monitoring • Court Orders, consent decrees Report Content Indicator Measures Data Needs General Supervision (CIFMS) Self-Assessing: Results & Compliance Improvement Planning: Priorities, Indicators & Measures Focus New Direction Progress Evaluation • Improvement Plan Implementation: • SIGS, GSEG, B/C Discretionary $ • Other State/Agency Investments Aligning IDEA State Accountability, Reporting, and Improvement • Data Sources • IDEA & B/C Data Reporting • NCLB, Health&SS, NCES) Performance Goals and Indicators Bold-Underlined Text = Federal Involvement

11. Leveraging Improvement - Bringing Action to Bear To Support Results Where the Problems Exist ED/OSEP State Local Levers to help support change where needed • Making policy; setting regulation • Making & enforcing compliance determinations • Administering discretionary grants • Collaborating with other agencies • Making evidence-based decisions • Marketing the message • Displaying information to support improvement • Disseminating research and promising practice • Brokering technical assistance • Providing targeted training & technical assistance Improved Outcomes for Children and Families

12. Consistent Terms and Definitions • State Goals (or “goals”) • Performance Indicators (or “indicators”) • Indicators measures (or “measures”) • Targets (desired level of indicator performance) • Benchmarks (intermediate target levels) • Data verification and review • Evidence-based decision making

13. State Goalsare the conditions we want for the population of children with disabilities. • State goals should be broad enough to match the purposes and rationale of IDEA: • prepare children for life, work and independent living (Part B) • ensure rights and effective programs (Parts B & C) • meet Part C’s purpose (improve/coordinate services) & rationale (improve results) • Example goal statements: • All students with disabilities are successfully prepared for the workplace and living independently. • All infants at risk of having a disability will be identified and referred for appropriate services.

14. Performance Indicators help quantify the achievement of a goal. For the goal: “Students with disabilities are successfully prepared for the workplace and living independently,” there may be multiple performance indicators. Indicators may include a specific target, e.g.: • % of students with disabilities successfully completing school will be comparable to students without disabilities. • All students 14 years of age and older will have transition plans included in their IEPs. • The %-age of students with work experience prior to leaving high school will ≥ 90%. • The %-age of students employed or in post-secondary education 6 months after high school will ≥ 75%.

15. Indicator Measures are the specific metrics for the performance indicator. Indicator measures for the performance indicator, “% of students with disabilities successfully completing school will increase to a level comparable to students without disabilities ” might include: • Measures of graduation (diploma) rate (variety of calculations) • Age and status at exit (e.g., ages of receipt of modified diploma, certificate of attendance, etc.).

16. Targetis the desired level of performance to be reached • A target is usually the criterion (long-term) level of desired performance. • Often, the target level is a part of the performance indicator (e.g., % of students with disabilities completing school will be comparable to that of students without disabilities). • The same target level may imply different amounts of change for different groups.

17. Benchmarkis an interim target • Benchmark and target indicator measures should be the same. • Performance levels on the way to the target, often with expected achievement dates. • Different subpopulations (e.g., NCLB subgroups) may have different benchmarks at different points in time.

18. Use of Terms - A Part B Example DATA

19. Use of Terms - A Part C Example DATA

20. Data verification & reviewmeans ensuring that valid & accurate data. • Evidence-based decision-making can’t be better than the quality of the data. • Westat is working on materials to assist states in improving their data verification processes. • You may have to spend as much time cleaning data as gathering it. • Using data will lead to improving it.

21. Evidence-Based Decision Makingis using information to guide decisions • Quantitative and qualitative data/ information to guide decisions regarding systemic improvement. • Ideally, data is: • systematically collected and analyzed, • includes multiple measures, and • is consistent with the experience of people familiar with the system. Review

22. Evidence-Based Decision Making for Improvement - Main Ideas • Manage the message • Hypothesis testing and planning improvements • Extant data sources • Data development agenda • Use variability to identify problems • Multiple data perspectives

23. 1 2 3 Manage the Public Message • Public reporting & motivation. • Don’t just publish a report - explain. • Write the press release. • Get buy-off from leaders, others. • Resource commitments - for data collection & analysis.

24. Hypothesis Testing and Improvement Planning • Hypothesis testing helps explore causal relationships. • Understanding causal relationships is critical to designing improvements in a system. • Improvement interventions intend actions that will cause performance to improve in desired areas • “Intervention and Measurement Systems” display how a set of suggested problems and associated improvements fit with improved performance.

25. A Process for Developing an “Intervention and Measurement System” • Develop specific hypotheses (e.g., from a logic model or understanding of causal relationships of what schools/localities do and your state’s goals and performance indicators) • Conduct Problem Analyses (disaggregate indicator measures) to test the hypotheses • Identify What Works (research-based practices in areas of function identified in the logic model) • Select and Implement Improvement Strategies (instructional/service and program interventions) • Monitor progress (using measures sensitive to the changes you expect)

26. Multiple Perspectivesin Analyses • Listen to others carefully. • Experience and multiple measures line up. • Triangulation of data. • Fair comparisons (e.g., comparable individuals or cohort data?) • Change/improvement over time

27. Limits of Measures, Indicators, & Goals • Do no harm. • Measurement can never match reality. • Find the best indicators and measures we can. • Confirm with multiple sources. • Avoid analysis paralysis.

28. Resources • OSEP TA&D Performance Measurement Work Group (Bob Brick, ALLIANCE, is a member). • Communities of Practice - www.tacommunities.org • If you want to work with your staff, others on performance measurement (alignment of IDEA management, accountability and reporting, terms, main ideas) check out the video streaming available after 8/1/03 at: http://interact.uoregon.edu/wrrc/perf.html • If this kind of training package is something PTIs would use, we may be able to find other efficient & accessible dissemination methods (e.g., DVDs).