1 / 13

Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville

VALID AND RELIABLE PHYSICAL, LIFE, AND EARTH SCIENCE CONTENT ASSESSMENTS FOR MIDDLE SCHOOL TEACHERS. Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville. National Evaluation Institute conference, October 2009 Louisville, KY. Overview of Presentation.

topper
Download Presentation

Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VALID AND RELIABLE PHYSICAL, LIFE, AND EARTH SCIENCE CONTENT ASSESSMENTS FOR MIDDLE SCHOOL TEACHERS Thomas R. Tretter Sherri L. Brown William S. Bush University of Louisville National Evaluation Institute conference, October 2009 Louisville, KY

  2. Overview of Presentation • Purpose of assessments • Structure • Validity • Reliability • Initial results

  3. Purpose of Assessments • Need for skilled science teachers is high • One important component is strong science content knowledge • Indicators of teacher content knowledge often indirect (e.g. number of courses taken, GPA) • Diagnostic assessments would be useful to professional development providers and teachers themselves • Target middle school science teachers

  4. Structure of Assessments • 3 content assessments: physical, life, earth/space • Teacher content knowledge synthesized from national standards (Breadth of Knowledge) • AAAS Benchmarks NSTA • NSES PRAXIS • NAEP INTASC • TIMSS • Four categories of knowledge (Depth of Knowledge) • Li and Shavelson (2001), Shulman (1986) • 1. Declarative 3. Schematic • 2. Inquiry 4. Pedagogical • STS (crosses both breadth and depth)

  5. Item Specification Grid (Physical Science Example) Depth of Knowledge Breadth Items simultaneously evaluate two separate dimensions (and are distributed with each major content domain) Some items also simultaneously address STS knowledge 20 multiple choice (1 pt. each), 5 open response (3 pts each)

  6. Validity • Content validity from basis in standards • Test items created by teams consisting of: • science educators, teachers, scientists • Construct validity from evaluation by reviewers • Each item evaluated by national sample of approx. 30 external reviewers from same 3 groups • Six nominally parallel versions of each item passing external review were generated and randomly assembled into 6 test versions for each content area

  7. Internal Reliability (N=1787) Physical Life Earth/Space Science ScienceScience Version Alpha N Alpha N Alpha N 1 .871 289 .839 328 .754 98 2 .818 69 .868 163 .626 29 3 .873 154 .841 69 .815 17 4 .833 110 .812 110 .788 113 5 .860 33 .678 63 .848 46 6 .815 38 .784 38 .869 10 Cronbach Alpha

  8. Initial Results • Certification grade level groups • Elementary Middle High • Content certification groups • No science Other science • General science Target science

  9. Results-Grade level certification Physical Life Earth/Space Science ScienceScience 13.5 (5.96) 16.1 (5.54) 13.8 (5.44) (n=76) (n=95) (n=45) 15.3 (6.74) 19.0 (6.53) 17.5 (5.59) (n=163) (n=209) (n=143) 20.9 (6.89) 22.4 (5.26) 19.1 (4.77) (n=133) (n=189) (n=48) p<.001 p<.001 p<.001 Note. Maximum score on all assessments is 35. Brackets indicate Tukey HSD post hoc homogenous subsets. ELEM MID HIGH

  10. NO SCIENCE GENERAL SCIENCE OTHER SCIENCE TARGET Results-Science certification Physical Life Earth/Space Science ScienceScience 13.2 (5.54) 17.3 (5.77) 14.6 (4.63) (n=195) (n=226) (n=130) 19.2 (7.26) 21.1 (6.10) 18.9 (6.26) (n=116) (n=155) (n=71) 18.8 (6.08) 19.4 (5.54) 18.1 (6.48) (n=45) (n=23) (n=29) 23.8 (7.01) 23.5 (5.31) 19.1 (4.69) (n=51) (n=116) (n=35) p<.001 p<.001 p<.001 Note. Maximum score on all assessments is 35. Brackets indicate Tukey HSD post hoc homogenous subsets.

  11. Summary • Extensive development process enhances validity • Cronbach Alpha Reliability >0.8 for most versions • Feedback includes subscores on: • - content subdomains (3 or 4) • - type of knowledge subdomains (4) • - STS • Also item-by-item scores for each individual provided • Cost to score = $10 per teacher per assessment

  12. Possible Uses • Pretesting to determine focus of content area for designing workshops or courses • Pre and post testing for evaluating workshops and courses • Diagnostic tool for teacher self-assessment • Measuring content knowledge for research—both as a treatment variable and an independent variable

  13. More Information Homepage for this project (including mathematics assessments) at http://louisville.edu/education/research/centers/crmstd To request electronic copies of the assessments for use in projects, please email the assessment coordinator CRMSTD@louisville.edu

More Related