1 / 14

Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator

Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials. Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator Jeanne Century, Co-Principal Investigator.

blake
Download Presentation

Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Scientific Validation Of A Set Of Instruments Measuring Fidelity Of Implementation (FOI) Of Reform-based Science And Mathematics Instructional Materials Dae Y. Kim, Principal Investigator Amy Cassata, Co-Principal Investigator Jeanne Century, Co-Principal Investigator IES Grantee Panel on Fidelity of Implementation September 7, 2011

  2. Instrument Development • NSF-Funded Development Project (2007-2010) • Developed a framework for measuring the enactment of five reform-based mathematics and science curricula • Created a suite of instruments based on this framework • Driving Research Questions • What are the enacted components of reform-based instructional materials in mathematics and science that are common across specific programs? • How can we rigorously and specifically measure multiple components of implementation? • How can we collect data on and accumulate knowledge about implementation across multiple programs?

  3. Instrument Development • Reform-based Curricula • Full Option Science System (FOSS), Science and Technology for Children (STC), Science Companion, Science Education for Public Understanding Program, and Everyday Mathematics • “Critical Components” • The measurable elements of the intended program model that are essential to its implementation • May be shared across programs or unique to specific programs • Can be organized into general categories • Structural: design, procedures, organization, and built-in supports • Instructional: expectations for participants’ actions, behaviors and interactions as they enact and engage in the intervention (users and recipients)

  4. Framework for Measurement • Process of Critical Component Identification • Review of written materials to identify explicit and implicit components • Interviews with program developers to articulate the “intended” program model and critical components • Conversations with program users about critical components • Iterative revision, reconciliation, and modification of critical components list and framework for organizing them

  5. FOI Framework for Instructional Materials

  6. Measuring Implementation • Suite of instruments measuring the operationalization of the identified critical components • Teacher Instructional Questionnaire • Teacher Instructional Log • Teacher Instructional Observation Protocol • Teacher Interview Protocol • School Leader Questionnaire • School Leader Interview Protocol • School-Wide Observation Protocol • Pilot tested in 200 classrooms in CPS (2008) and revised • Field-tested in 265 classrooms in CPS (2009)

  7. Measuring Implementation

  8. Classroom-Level Instruments

  9. Next Steps: Validation • IES Funding: Goal 5 (Measurement) Goals • Establish validity and reliability within and across the three classroom-level instruments • Develop a student questionnaire (grades 3-5) to triangulate measures of student engagement • Participants • 50 schools across 4 sites (IL, WA, MA) • All teachers grades K-5 (N~1000) • All consented students grades 3-5 (N~4500) • Schools use both Everyday Mathematics and a reform-based science curriculum

  10. Year 1: Student Questionnaire • 20 to 25-item questionnaire • Literature review for items measuring Student Engagement critical components • Item development and pilot testing • Cognitive interviews • Consult with experts • Field testing • 300 students in grades 3-5 • Analyze factorial validity and internal consistency • Instrument revision

  11. Year 2: Data Collection for Validation • All teachers will complete • Teacher Instructional Questionnaire – Math • Teacher Instructional Questionnaire – Science • All students (grades 3-5) will complete • Student Questionnaire – Math • Student Questionnaire – Science • 300 teachers with additional observations and logs • 100 teachers: 2 Math observations and 4 Math logs • 100 teachers: 2 Science observations and 4 Science logs • 100 teachers: 1 Math & 1 Science observation, 4 Math & 4 Science logs • Student achievement data (grades 3-5)

  12. Year 3: Analysis • Inter-rater agreement (observation) • Internal consistency of items • Factorial structure of items • Differences in scores across different groups/content areas • Science vs. mathematics • Grade level • Study site • Cross-instrument consistency • Teacher Instructional Questionnaire vs. Student Questionnaire • Teacher Instructional Observation Protocol vs. Teacher Log • Predictive validity • Mathematics and science achievement (grades 3-5)

  13. Predicting Student Achievement • To what extent do composite indices on the four subcategories (SP, SE, IP, ISE) predict student achievement in science and mathematics? • Planned analysis for predictive validity • 3-level fixed effects HLM model will explore the effect of the FOI composite indices measured at the teacher (classroom) level on individual students’ science and mathematics achievement (Grades 4-5) • Prior year’s science or math score will be entered as a covariate • Other variables: student demographics, classroom characteristics • Data for science and mathematics will be analyzed separately, then pooled • Data for questionnaires, logs, and observations will be analyzed separately

  14. Summary • The FOI framework clearly and specifically describes the nature of reform-based mathematics and science instructional materials so these elements can be measured • The framework has resulted in a set of instruments which are currently being validated • The framework, and related instruments, have the potential to inform our understanding reform-based STEM curricula and to accumulate knowledge about the elements that make them most effective

More Related