1 / 54

STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast

STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast. April 23, 2008 START TIME 9:00 a.m. Presenters Jonathan Halling - ETS Stephanie Sartain Herrera - ETS Mary Anne Arcilla - ETS Mark McLean - CDE. Agenda. Life of a Test Item Item Writer Training

lamar
Download Presentation

STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. STAR Item Writing, Assessment Review Panels, Standard Setting,and Field TestingWebcast April 23, 2008 START TIME 9:00 a.m. Presenters Jonathan Halling - ETS Stephanie Sartain Herrera - ETS Mary Anne Arcilla - ETS Mark McLean - CDE Getting Involved in STAR

  2. Agenda • Life of a Test Item • Item Writer Training • Assessment Review Panels (ARPs) • Standard Setting • Field Testing • Open Forum Getting Involved in STAR

  3. Life of a Test Item Getting Involved in STAR

  4. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test ARP & CDE review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  5. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test ARP & CDE review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  6. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test ARP & CDE review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  7. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test ARP & CDE review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  8. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test ARP & CDE review data from field test Administer operational test ARP & CDE review forms Create operational forms Getting Involved in STAR

  9. Item Writer Training Getting Involved in STAR

  10. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  11. Item WriterTraining • What will be covered by ETS in the training session? • What makes a good item • The item-writing process • How to write items specifically to meet California standards • Who should attend? Persons who: • Possess knowledge of California content standards • Are current or recent teachers • Have at least three (3) years teaching experience Getting Involved in STAR

  12. Item WriterTraining • How many are trained? • 20–30 attendees per content area in CAPA, CST, CMA, and STS • Where does training take place? • 2008: Long Beach, CA • When does the 2008 training occur? • July 15–16: CAPA items • July 17–18: CST, CMA, and STS items Getting Involved in STAR

  13. Item WriterTraining • Benefits • Professional development • Increased familiarity with California content standards • Potential for becoming professional item writer • Expenses • $150 per day or substitute teacher reimbursement • Travel and lodging expenses paid by ETS Getting Involved in STAR

  14. Assessment Review Panels(ARPs) Getting Involved in STAR

  15. Assessment Review Panels Panel members meet in an advisory capacity to ETS and CDE to review: • Item specifications and blueprints • Proposed new test items • Data collected on field-tested items • Proposed new operational test forms • Proposed released test questions (RTQs) • CST only • Differential Item Functioning (DIF) Getting Involved in STAR

  16. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  17. Assessment Review PanelsItem Specifications and Blueprints • Help define recommended blueprints for new tests • Develop foundation for future item and test form development • Examples: • Revised General Mathematics • Conducted April 2, 2008 • California Modified Assessment (Grades 9–11) Getting Involved in STAR

  18. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  19. Assessment Review PanelsItem Review • Advise if the items developed are: • Measuring the standard • Grade-level appropriate • Free from bias • Ethnic • Socioeconomic • Gender • Geographical • Interesting and appropriate to students • Suggest ways in which an item may be adjusted to meet the above criteria, if necessary • Do NOT rewrite items Getting Involved in STAR

  20. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  21. Assessment Review PanelsData Review • Recommend whether field-tested items should be included in the “bank” and hence be available to be selected for inclusion on future operational tests • Review statistics on the item’s performance as a field-test item • No editing of items is performed at this stage Getting Involved in STAR

  22. Assessment Review PanelsDifferential Item Functioning (DIF)Review • Occurs when item performance is significantly different for members of reference groups and comparison groups • Participants representing various demographic groups review test questions to determine whether or not items are appropriate • One-day meeting, conducted after field testing Note: DIF review panel consists of different participants from other ARPs Getting Involved in STAR

  23. Study state guiding documents ARP & CDE develop blueprints SBE action Train item writers CDE review of item writer materials Develop item writer materials Prepare, review, and revise items ARP & CDE review items Receive items Create field test sets with CDE review SPAR review Revise items Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms Administer operational test ARP & CDE review forms Getting Involved in STAR

  24. Assessment Review PanelsForms Review • Last stage in the process of building the test • Assembled operational forms: • Contain a range of difficulty of items • Match the blueprint and statistical specifications • Avoid clueing • No editing of items Getting Involved in STAR

  25. Assessment Review PanelsReleased Test Question (RTQ) Review • After use on operational test forms, some questions are released for review and use by the public • Contain a range of item difficulty • Are proportional to the blueprint • Released annually • No editing of items Getting Involved in STAR

  26. ARP Meeting Schedules Getting Involved in STAR

  27. 2008 ARP Meeting ScheduleCST • Item Review • April 2–5 • Data Review • July 29–30 • Forms Review • September 23 • Released Test Questions • October 29 Note: ARP members are expected to attend meetings at ALL stages of development Getting Involved in STAR

  28. 2008 ARP Meeting Schedule CMA • Item Review • Grades 3–8 • September 9–12 • Data and Forms Review • Grades 6–8 • November 19–20 • Data Review • Grades 3–5 • To be scheduled early 2009 Note: ARP members are expected to attend meetings at ALL stages of development Getting Involved in STAR

  29. 2008 ARP Meeting ScheduleCAPA • Item Review • May 13–14 • Data Review • September 25 • Forms Review • To be scheduled early 2009 Note: ARP members are expected to attend meetings at ALL stages of development Getting Involved in STAR

  30. 2008 ARP Meeting Schedule STS • Item Review • RLA grades 8–11, Algebra I, and Geometry • April 8–11 • RLA and Math grades 2–7 • July 8–11 • Data Review • RLA and Math for grades 2–7 • October 8–9 • Forms Review • Grades 2–4 • October 10 Note: ARP members are expected to attend meetings at ALL stages of development Getting Involved in STAR

  31. Becoming an ARP Member Getting Involved in STAR

  32. The Process • Review the ARP member requirements • Obtain application form: • E-mail star@ets.org • Call Kristin Elizalde 925-808-2009 • Submit a completed application form • Application reviewed by: • ETS content experts • CDE Standards and Assessment staff • State Board of Education staff • Application approved or denied Note: Process can take several weeks Getting Involved in STAR

  33. ARP Members Needed! • California Standards Tests • Science (Grades 5 and 8) • History–Social Science (Elementary and Middle School) • ELA (All grades) • California Modified Assessment • English–Language Arts (ELA) • Mathematics • Science Note: CMA – Middle and High School especially needed! Getting Involved in STAR

  34. ARP Members Needed! • California Alternate Performance Assessment • English–Language Arts (ELA) • Mathematics • Science • Standards-based Tests in Spanish • Reading/Language Arts (RLA) • Mathematics, Algebra I, and Geometry Note: STS – Middle and High School especially needed! Getting Involved in STAR

  35. Benefits of Being an ARP Member • Become more familiar with the testing program • Professional development • Learn more about what attributes a high quality test item should display • Learn more about assessing how an item performs on a test • Contribute to the development of high-quality tests • All meeting attendance expenses covered Getting Involved in STAR

  36. ARP Member Requirements Getting Involved in STAR

  37. ARP Member Requirements All Panels • Educators from • Urban areas • Rural areas • Suburban areas • Experienced in teaching K–12 students from diverse populations Getting Involved in STAR

  38. ARP Member RequirementsCST, CMA, CAPA, and STS • Bachelor’s or higher degree in content area or related subject • Knowledge of California content standards • Familiarity with student capabilities • K–12 teachers must also have: • Three or more years teaching courses in grades K–12 • ELA, math, science, or history–social science • County, district, or university content specialists must also have: • Three or more years as a district or county content specialist or university instructor • ELA, math, science, or history–social science Getting Involved in STAR

  39. Additional ARP Member Requirements for CMA • English–language arts, mathematics, and science only • K–12 teachers • Three or more years of experience teaching ELA, math, or science in grades K–12 and experience working with special education students • Experience with more than one type of disability • School administrators, district/county content/program specialists, or university educators • Three or more years of experience as a school administrator, district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science Getting Involved in STAR

  40. Additional ARP Member Requirements for CAPA • English–language arts, mathematics, and science only • K–12 teachers • Experience with students with severe cognitive disabilities • Experience with more than one type of disability • Three or more years of K–12 experience teaching special education or teaching ELA, math, or science in a general education classroom • School administrators, district/county content/program specialists, or university educators • Three or more years of experience as a school administrator, district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science Getting Involved in STAR

  41. Additional ARP Member Requirements for STS • Reading/language arts (RLA) and mathematics only • Bilingual and biliterate in Spanish • Three or more years experience teaching students in Spanish • Bachelor’s or higher degree in Spanish or a subject area related to bilingual education, reading/language arts, or mathematics Getting Involved in STAR

  42. Break10 Minutes Getting Involved in STAR

  43. Standard Setting Getting Involved in STAR

  44. What is Standard Setting? • Process used to recommend the minimum test score needed for a student to be considered performing at a certain performance level • For example, the minimum test score needed to be at the proficient level • Bookmark standard setting methodology • Participants review test items that have been sequenced from the easiest to most difficult item • Participants identify the items that likely exceed the ability of a student at each of the California performance levels • The difficulty level of these identified items form the basis for the minimum test scores needed to reach each performance level Getting Involved in STAR

  45. Standard SettingCAPA Levels I–V (ELA, Math, and Science) • Three-day meeting • September 16–18, 2008 • Participant requirements same as ARP members Getting Involved in STAR

  46. Standard SettingCMA Grades 3–5(ELA, Math, and Science) • Two three-day meetings • ELA • September 24–26, 2008 • Math and Science • October 1–3, 2008 • Participant requirements same as ARP members Getting Involved in STAR

  47. Standard SettingSTS Grades 2–4RLA and Math • Three-day meeting • RLA • January/February 2009 • Math • January/February 2009 • Participant requirements same as ARP members Getting Involved in STAR

  48. Standard SettingMember Requirements • Educators from: • Urban, rural, and suburban areas • Large and small districts • Geographically dispersed throughout California • Experienced in teaching students in appropriate grades/content areas and from diverse populations • Community members • Parent Teacher Association (PTA) • School board • Business Getting Involved in STAR

  49. Standard SettingFor CAPA, CMA and STS • Application Form • E-mail star@ets.org or • Call Kristin Elizalde 925-808-2009 • Reimbursement for: • Travel and accommodation expenses • Substitute teacher for school/district Apply now! Getting Involved in STAR

  50. Field Testing Getting Involved in STAR

More Related