1 / 25

REA Brown Bag – SAT10 Growth Methods and Reporting

REA Brown Bag – SAT10 Growth Methods and Reporting. Tom Watkins, Ph.D Director of Research, Evaluation and Assessment, SPPS. Goals for Brown Bag (2 or more of the following):.

luigi
Download Presentation

REA Brown Bag – SAT10 Growth Methods and Reporting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. REA Brown Bag – SAT10 Growth Methods and Reporting Tom Watkins, Ph.D Director of Research, Evaluation and Assessment, SPPS

  2. Goals for Brown Bag (2 or more of the following): 1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS. 2. To describe some strengths and weaknesses of various approaches for reporting achievement growth on the SAT10 in SPPS. 3. To examine the degree of correlation between different growth measures at the student score level 4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level

  3. Goals for Brown Bag 1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS. 4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level within school, and at the group level within the district

  4. Part 1: Strengths and Weaknesses of Approaches for Calculating Achievement Growth • Illustrations and Basis for SAT10 Growth • Scaled Score Improvement (SSI – OLS Reg) • Percent Gaining (NCE change 0 or more) • Index Rate Growth (SAT10 to MCA) • Heirarchical Linear Modeling (HLM) • General Psychometric Considerations • General Practical Considerations • SPPS Considerations

  5. Which Indicators Matter to SPPS Elementary Schools?

  6. The method for calculating growth for student groups from MAT7 to SAT10 (similar to what has been used since): • For each grade level and subject area, run a simple linear regression of SAT10 scaled scores on MAT7 scaled scores. • Save the standardized residuals (unstandardized residuals were used for the MAT7 SSI). • For groups with N's of 10-39, a mean standardized residual at or below -.20 is "low growth", and residuals at or above +.20 is "high growth". • For groups with N's of 40-79, a mean standardized residual at or below -.15 is "low growth", and residuals at or above +.15 is "high growth". • For groups with N's of 80+, a mean standardized residual at or below -.10 is "low growth", and residuals at or above +.10 is "high growth".

  7. Basis of Index Growth from SAT10 to MCA

  8. Basis for MLM Approach (HLM one type) (Steenbergen, 2002) • Uses “Nested” Data - Distinguishing Student-Level from Teacher, Grade, School and District Levels • Data Intensive • Theory Intensive • Requires More Assumptions

  9. General Practical Considerations in Growth and Gain Models

  10. General Psychometric Considerations in Growth and Gain Models

  11. Additional SPPS Growth Method Considerations • Reflects local input (Administrator’s Academy, Accountability Framework) • Live or dead growth norm • Local, national or state growth comparisons • Provides estimates of growth of student groups (ethnic, FRL, ELL, Spec Ed) • Multiple measures/methods preferred • As simple as appropriate, but no simpler. • Acknowledges real student growth patterns • Descriptive vs. VAM approach (Davison, Thum) • Consider link to reported growth history, and likely future • Consider link to formative assessment system

  12. Additional SPPS Considerations in Growth and Gain Models

  13. Part 2: In what ways do different growth measures produce the same results? Growth categories (average, hi or low): • at the grade level within the school, • at the group level within the district

  14. Some Relevant Items in CRESST’s Standards for Educational Accountability Systems (Baker, et al., 2002) • (A1) Accountability expectations should be made public and understandable to all participants in the system. • (A2) Accountability systems should employ different types of data from multiple sources. • (A4) Accountability systems should include the performance of all students, including subgroups that historically have been difficult to assess. • (B9) The validity of a measure that has been administered as a part of an accountability system should be documented for the various purposes of the system. • (B10) If tests are to help improve system performance, there should be information provided to document that test results are modifiable by quality instruction and effort. To what degree do these standards apply to School Improvement Systems?

  15. Buried Treasure – Implications (Celio & Harvey, 2005) • Less may be more • Parsimony and power should be respected • Status data necessary but not sufficient • Potential to change board-supt relations • Cost-benefit analysis • Professional development and tech assistance will be required • State leaders have a significant role to play

  16. Buried Treasure – Indicators (Celio & Harvey, 2005) • Achievement (reading and math) • Elimination of the achievement gap • Student attraction (school ability to attract students) • Student engagement with the school • Student retention/completion • Teacher Attraction and retention • Funding Equity

  17. Four Quadrants of School Improvement Indicators (Dale Carlson, 2002)

  18. So, Where Are We in SPPS? • Put growth models in perspective (only part of one piece of the puzzle) • Transition from SAT10 to MCA-II growth • Need to dedicate resources to solid cost-benefit analyses, using multiple outcomes • Have to prioritize and commit to indicators so solid professional development and technical assistance is possible.

  19. Partial List of References Baker, E. L., Linn, R. L., Herman, J. L., & Koretz, D. (Winter, 2002). Standards for Educational Accountability Systems. Policy Brief 5: Center for Research on Evaluation, Standards and Student Testing. Celio.M.B & Harvey, J. (2005). Buried treasure: Developing a management guide from mountains of school data. http://www.crpe.org/pubs.shtml#leadership Olson, L. (2005, July 13). Education Department convenes working group on ‘growth’ models. Education Week. Available at http://www.edweek.org/ew/articles/2005/07/13/42growth.h24.html . O’Malley, K., Vansickle, T. & Housson, S. (2005, June). Measuring growth: Where policy makers and psychometricians meet. Paper presented at the annual national conference on Large-Scale Assessment. San Antonio, TX. Thum, Y.M (2005, April). Designing school productivity indicators: Connecting goals, data & models. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

More Related