1 / 84

Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann

Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource Allocations Presented February 13, 2013. Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann York College. Background: Linking Data Sets.

radwan
Download Presentation

Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource AllocationsPresented February 13, 2013 Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann York College 26th Annual Management Information Systems [MIS] Conference

  2. Background: Linking Data Sets Part C Early Intervention Services data (Birth – 3) Part B Special Education Services data (4, 5 – 21) General Education Services data K Children assessed for K Readiness WSS-K, scaled then scores at 3 levels Readiness (Developing, Approaching, Fully Ready) Maryland State Assessment (MSA), scaled then scored at 3 levels (Advance, Proficient, Basic) • Math and Reading assessments administered annually in grades 3 through 8 2013 MIS Conference

  3. 2013 MIS Conference

  4. Benefits of Using the Maryland IDEA Scorecard • Data at the State, District, and School levels that drills down to the student level • Data that allows users to identify students in need of targeted interventions in the alert categories (attendance, academics, suspension, and mobility) • Data to create an action plan to monitor student progress within targeted interventions • Reporting functions to support monitoring of progress towards targets of interventions 2013 MIS Conference

  5. How to use these data at the state level? • Descriptive • Who do we serve? • Longitudinal • What is the educational placement of children served in EIS by K and Grade 3? • Is Fall K WSS is a successful predictor of later standardized test performance? • Comparative • Is there a difference in standardized achievement performance within educational services (Gen Ed and Sp Ed) for children served in EIS? 2013 MIS Conference

  6. Tracking Plan 2013 MIS Conference

  7. Method • PARTICIPANTS • 42% tracking match • Missing data • Outcomes • WSS-K (2006-07) • RSAA and MSAA (Spring 2011) • Procedure • Three studies presented 2013 MIS Conference

  8. Missing Data • N = 5,328 participants received EIS • 42.1% (n = 2,245) matched • 57.9% (n = 3,083) unmatched (missing data) • Is there a significant difference between matched and unmatched participants on characteristics: gender, Part C eligibility category, Part C MA, Race, age of entry, age of exit, and/or months in EIS? 2013 MIS Conference

  9. Gender Results of the Chi-Square statistic assuming equal probabilities indicated no significant difference between Unmatched and Matched participants for gender [X2 (1, N= 5,328) = 0.93, p> .05]. Inspection of cell counts and percentages indicated that the distribution for gender was similar for both groups. 2013 MIS Conference

  10. Part C Eligibility Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Eligibility [X2 (2, N = 5,317) = 47.23, p< .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible due to a Condition with a High Probability than Matched participants. Conversely, a greater proportion of Matched participants were eligible based on 25% Delay than Unmatched participants. 2013 MIS Conference

  11. Part C Eligibility cont. 2013 MIS Conference

  12. Part C Medical Assistance Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Medical Assistance [X2 (1, N = 5,278) = 17.49, p = .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible for Medical Assistance while receiving EIS. 2013 MIS Conference

  13. Race Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Race [X2 (5, N = 5,328) = 16.76, p< .05]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were Black or African American while a smaller proportion of participants in the Unmatched group were Hispanic than in the Matched group. 2013 MIS Conference

  14. Entry/Exit Age in Months 2013 MIS Conference

  15. Summary – Missing Data • Gender – the two groups are similar • Eligibility – significantly greater proportion of Unmatched participants in High Probability Category • Medical Assistance – significantly greater proportion of Unmatched participants received MA • Race – significantly greater proportion of Black or African American participants in Unmatched group with significantly greater proportion of Hispanic participants in Matched group • Age – Matched participants were significantly younger at age of entry than Unmatched participants. Matched participants received EIS longer than Unmatched participants. Age of exit was similar for both groups. 2013 MIS Conference

  16. Outcome Instruments • Work Sampling System-Kindergarten (WSS-K) Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development) • Personal and Social Development • Language and Literacy • Mathematical Thinking • Scientific Thinking • Social Studies • The Arts • Physical Development • Reading State Accountability Assessment (RSAA) • scaled then scored at 3 levels (Basic, Proficient, Advanced) • Math State Accountability Assessment (MSAA) • scaled then scored at 3 levels (Basic, Proficient, Advanced) 2013 MIS Conference

  17. Procedure • Student data tracking system links records of students in General Ed, Special Ed (Part B) and EIS (Part C) • Identify children in 3rd Grade 2010-11 school year with birth dates between Sept. 1, 2001 and Aug. 31, 2002 • At the state level, student identifiers entered in student data tracking system to obtain student service level (Gen Ed or Sp Ed), student outcomes, and demographic info • Deidentified data provided to researchers 2013 MIS Conference

  18. Studies and RQ for this Presentation • Impact of Early Intervention on K Readiness • Who was served by EIS? • Does EIS impact later K Readiness Scores? • Tracking Children Receiving Early Intervention Services (Part C Services Birth to 3) into Elementary School to Grade 3 • What is the educational placement at Grade 3 for children who received EIS? • What is the SAA performance at Grade 3 for children who received EIS? 2013 MIS Conference

  19. Different Data Set 3. K Readiness and Grade 3 SAA Performance • Is there a relationship between K Readiness and Grade 3 RSAA and MSAA scoring? • What subscales of the WSS predict RSAAand MSAA scoring? 2013 MIS Conference

  20. Study 1: Impact of Early Intervention Services on K Readiness • Does level of service provided to children (Birth – 3) enrolled in early intervention services (EIS) programs enhance their later performance on the Kindergarten Work Sampling System (WSS-K)? 2013 MIS Conference

  21. K Readiness: Method • 2,245 children • Who were eligible and received EIS services in MD linked with MD MMSR scores • Born between Sept 1, 2001 and Aug 31, 2002 • EIS services • WSS-K • Summary scaled composite score • Hierarchical Linear Regression 2013 MIS Conference

  22. Participants 2013 MIS Conference

  23. EIS Services & WSS-K Average Scores Max WSS-K score of 90 2013 MIS Conference

  24. Variables Outcome: WSS-K Predictors: • Demographics: FaRMs, Gender, Minority • Earliest Age of Child that Part C Services Begin (-) receive services earlier better prepares child to enter K • Time in Program (+) indicates longer time in program better prepares child to enter K • Total Minutes Services (+) indicates longer time in program better prepares child to enter K 2013 MIS Conference

  25. Hierarchical Regression Results 2013 MIS Conference

  26. EIS – WSS Conclusions • Demographic Controls: WSS-K was higher for students not economically disadvantaged, higher for girls, and for White students • Age Svc Began (-): For every month earlier a child starts receiving services, he/she is expected to score .017 SD increase on the WSS-K • Supports previous findings • Time in Program (-):inconclusive; possible that children who are in the program for longer times have more severe disabilities • Total Minutes Services (-): inconclusive, possible that children with more severe disabilities will have more/longer services • Correlation Time in Program & Total Min Svc r = .37, p < .001 2013 MIS Conference

  27. Study 2 Tracking Children Receiving Early Intervention Services (Birth to 3) into Elementary School by Service Level: How do they Compare with their Peers? 2013 MIS Conference

  28. Background • Part C Services (IFSP) • Part b/B Services (IEP) • Little research examining outcomes of children receiving Part C services, mostly Part B services (Cole, Dale, Mills & Jenkins, 1993; Daley & Carlson, 2009; Peterson et al., 2004; Walker et al., 1988) • Focused on developmental progress over short term; not longitudinal or growth trajectories • Enrollment in special education changes for children as they move through elementary school • Limitations (disability, small samples, covariates) 2013 MIS Conference

  29. First Steps: Descriptive Analyses • Children in 3rd grade (within birth cohort) • What is their Education Service (Gen Ed, Sp Ed)? • Who were the children receiving EIS? 2013 MIS Conference

  30. Participants 2013 MIS Conference

  31. Total Sample, Gen Ed and HI Sp Ed *Autism, Deaf, Deaf-Blindness, Developmental Delay, Hearing Impaired, Intellectual Disability, Multiple Disabilities, Orthopedic Impairment, Traumatic Brain Injury, Visual Impairment 2013 MIS Conference

  32. 2013 MIS Conference

  33. Students in Grade 3 Historically Tracked to EIS 2013 MIS Conference

  34. Students in Grade 3 Tracked to EIS, EIS Characteristics 65.5%, n = 1,628 enrolled in general education at grade three 2013 MIS Conference

  35. Average State Assessment Scores by Educational Service at K and Grade 3 2013 MIS Conference

  36. Hierarchical Regression Analyses Outcome: RSAA Grade 3, MSAA Grade 3 Three Models: Model 1, 2, 3 Demographics: FaRMs, Gender, Minority Model 2, 3 Part C EIS Model 3 EIS x FaRMs EIS x Gender EIS x Minority 2013 MIS Conference

  37. Hierarchical Regression Results: WSS-K *p ≤ .05, **p ≤ .01 2013 MIS Conference

  38. WSS-K Results General Ed • WSS-K higher for students notFaRMs, for girls, and for White students • EIS students scored lower (.7 M diff) • No interaction effects Special Ed • WSS-K higher for students not FaRMs, for girls, and for White students • EIS students scored lower (.8 M diff) • Sig interaction EIS x Gender • F > M NO EIS (2.9 M diff) • M > F EIS (1.7 M diff) 2013 MIS Conference

  39. Hierarchical Regression Results: RSAA *p ≤ .05, **p ≤ .01 2013 MIS Conference

  40. RSAA Results General Ed • RSAA higher for students notFaRMs, for girls, and for White students • EIS students scored lower (3.1 M diff) • No interaction effects Special Ed • RSAA higher for students notFaRMs and for White students • EIS students scored lower (22.3 M diff) • No interaction effects 2013 MIS Conference

  41. Hierarchical Regression Results: MSAA *p ≤ .05, **p ≤ .01 2013 MIS Conference

  42. MSAA Results General Ed • WSS-K higher for students notFaRMs, for girls, and for White students • EIS students scored lower (1.3 M diff) • Sig interaction EIS x Gender • F > M No EIS (1.9 M diff) • M > F EIS (4.2 M diff) Special Ed • WSS-K higher for students not FaRMs, for girls, and for White students • EIS students scored lower (16.2 M diff) • Sig interaction EIS x Gender • M > F No EIS (10.4 M diff) • M > F EIS (42.9 M diff) 2013 MIS Conference

  43. General Ed EIS SAA Conclusions • Demographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged, higher for girls, and for White students • EIS: WSS-K, RSAA, & MSAAwhen FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Gen Ed peers. • EIS Interactions: WSS-K & RSAA no significant interactions; MSAA EIS x Gender (F EIS) 2013 MIS Conference

  44. Special Ed EIS SAA Conclusions • Demographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged and for White students; females scored higher on WSS-K & MSAA • EIS: WSS-K, RSAA, & MSAAwhen FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Sp Ed peers • EIS Interactions: WSS-K & MSAA significant EIS x Gender (F EIS lower in K and more at Grade 3) 2013 MIS Conference

  45. Youth in EIS have lower average scores than their peers in Gen Ed and Sp Ed 2013 MIS Conference

  46. Female EIS in Sp Ed (HI) Services have lower average scores than their peers 2013 MIS Conference

  47. Study 3: K Readiness and Grade 3 MSA Performance Is performance on the WSS-K predictive of Grade 3 high stakes testing (Reading MSA and Math MSA)? 2013 MIS Conference

  48. Background Maryland Model for School Readiness (MMSR) Pearson Work Sampling System Assessment Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development) • Personal and Social Development • Language and Literacy • Mathematical Thinking • Scientific Thinking • Social Studies • The Arts • Physical Development 2013 MIS Conference

  49. Method • Participants/Procedure • General Ed Fall K Work Sampling System student scores (2003, 2004, 2005) N = 152,105 Matched to • Grade 3 Spring MSA Math, MSA Read student scores (2006, 2007, 2008) N = 100,958 Match Rate 66% for Reading & Math • Instruments • WSS is a 30 item instrument with scaled items coded 1 - 3 • MSA is a high stakes test coded 1 - 3 2013 MIS Conference

  50. Kindergarten MMSR Composite Score Distribution by Grade 3 MSA Performance Level 2013 MIS Conference

More Related