Pioneering the Next Generation of Measurement: Developing West Virginia's Performance Assessments

1. Pioneering the Next Generation of Measurement:Developing West Virginia's Performance Assessments

2. WV CTE Global 21 Performance Assessments • Sandra Starr Vision and Its Background Development Process • Nathan Taylor Writing Process • Vicki Jenkins Administration Experience • Ryan Haught Student Experience • Sandra Starr Problems and Next Steps West Virginia Department of Education

3. The Vision and Its Background Sandra Starr

4. “I realized we (CTE) were missing a lot of what we do by issuing the 50 question multiple-choice end-of-course exams.” Dr. Stan Hopkins Assistant Superintendent Division of Technical and Adult Education Services West Virginia Department of Education

5. CTE terms to know • CTE is Career and Technical Education. • Clusters organize both academic and occupational knowledge and skills into a coherent course sequence. West Virginia Department of Education

6. CTE terms to know continued • Concentrations are groups of at least 3 courses that when combined make up a program. • The courses may be from a single CTE content area or a combination of content areas which may be predefined or linear in nature. • Some CTE students build meaningful cohesive concentrations based on individual interests. • Completers are students who finish a minimum of three courses, including the core required courses. West Virginia Department of Education

7. CTE present West Virginia has • 1,198 CTE programs in 55 counties/districts • 100,000+ students enrolled in CTE • 43,000 students enrolled in CTE concentrations West Virginia Department of Education

8. CTE Global 21 Performance Assessments West Virginia currently has • 86 CTE concentrations • 72 CTE concentrations that assessed completers • ≈5,600 students who completed the performance assessments in 2010 West Virginia Department of Education

9. CTE future “21st century career and technical education is purposeful, personalized and performance-based, focusing on the numeracy, literacy, employability, and technical skills required for success in postsecondary education and careers.” Dr. Stan Hopkins Assistant Superintendent Division of Technical and Adult Education Services West Virginia Department of Education

10. Carl Perkins Vocational and Technical Education Act of 1998 • Incorporates secondary/postsecondary education elements • Includes coherent and rigorous content aligned with standards • Includes ways of earning postsecondary credits • Leads to credentials/certificates/degrees • Requires accountability for technical skills West Virginia Department of Education

11. Perkins grant continued Levels of performance "shall, at a minimum-- • (I) be expressed in a percentage or numerical form, so as to be objective, quantifiable, and measurable; • (II) and require the State to continually make progress toward improving the performance of vocational and technical education students." [sec. 113(b)(3)(A)(i)] "Indicators of performance... shall be established solely by each eligible agency with input from eligible recipients." [sec. 113 (b)(2)(D)] West Virginia Department of Education

12. The Development Process Sandra Starr

13. Blueprint and design • Researched available models and fields of thought. • Compartmentalized test into three areas: • live performance (typically 3 interrelated tasks based on a scenario) • written component • oral component West Virginia Department of Education

14. Blueprint and design continued • Rubrics were standardized • Consulted with Dr. Gary Phillips, Vice President, American Institute for Research West Virginia Department of Education

15. Writing/Construction • Teachers wrote assessments with guidance from CTE Coordinators • Copy and paste with editing from teacher drafts AFTER alignment was ensured. • Alignment based on Webb method. • Consulted with Dr. Norm Webb, University of Wisconsin, prior to process. West Virginia Department of Education

16. Writing/Construction continued • Time limit of 3 hours was determined for live performance • Each task carried a suggested timing set by writers and/or state coordinators • Allowed flexibility to move time from task to task, as needed, but not to exceed three hour limit West Virginia Department of Education

17. Sample script excerpt West Virginia Department of Education

18. Sample scenario ET1980 Welding Technology The local welding shop is accepting applications for entry level welders. First you must submit a resume and complete the interview process. Then, you will complete a hands-on welding test. West Virginia Department of Education

19. Sample problem set Problem 1: Using the blueprints, layout the equipment. Problem 2: Develop a list of tools and materials needed to complete the assessment. This list of materials will be used to complete the rest of the examination, so accuracy is a MUST. Problem 3: Construct the items on a set of blueprints. West Virginia Department of Education

20. Writing/Construction continued • Written and oral components to be given prior or post live performance http://careertech.k12.wv.us/documents/WrittenComponentRubric.pdf http://careertech.k12.wv.us/documents/OralComponentRubric.pdf • Trainings for each level of administration were created and housed on Internet to ensure defensibility of trainings • Test Administration Manual created with ancillary materials http://careertech.k12.wv.us/documentsTestAdministrationManual_R.pdf West Virginia Department of Education

21. Welding West Virginia Department of Education

22. Scoring/Rubrics • Standardized rubrics used to create a “Master Rubric” from which each coordinator could pull to create rubrics for each concentration • Due to the specificity of some concentrations, some criteria had to be customized • Customized criteria were used across all concentrations when appropriate West Virginia Department of Education

23. Master rubric • Content Area Knowledge • Critical Thinking & Problem Solving • Social/Personal Accountability • Oral Presentation Skills • Visual & Media Literacy • Work Readiness Applications • Written Project Support West Virginia Department of Education

24. Sample criterion West Virginia Department of Education

25. Future revisions and changes • Use the field test from 2010 and look at the suggested revisions/corrections from the teachers and use that input to correct and improve each assessment • Use improved form as “OPERATIONAL” in 2011 • Development of second form - must undergo “social moderation” to ensure comparability West Virginia Department of Education

26. The Writing Process Nathan Taylor

27. Teacher selection • CTE Coordinators invited teachers that were specialists in the content areas being assessed. • Teachers who had completed the most recent Content Standards and Objectives (CSOs) revisions were invited first. West Virginia Department of Education

28. Planning • Dates for workshops were scheduled around convenient times for each CTE area based on CTSO activities. West Virginia Department of Education

29. Writing workshops • Training on depth of knowledge • Teachers were introduced to the depth of knowledge method of alignment and participated in evaluation of materials to identify levels of cognitive complexity. • Teachers evaluated CSOs to identify appropriate DOK levels to be assessed. • Teachers created an overarching scenario and three problems within the scenario that aligned with the CSOs. West Virginia Department of Education

30. Writing workshops continued • Teachers produced a rough draft of each assessment, peer reviewed the drafts for content and bias, and made appropriate changes. • Coordinators reviewed the drafts (content and bias) and identified areas for corrections as needed. • Final drafts were submitted at end of each writing workshop. West Virginia Department of Education

31. Workshops by cluster Agriculture, Science and Natural Resources • Coordinators—Jason Hughes, Nathan Taylor • Workshop held one time—December 2-4 • 30 Participants West Virginia Department of Education

32. Workshops by cluster continued Engineering and Technical Education • Coordinators—Rick Gillman, Kathy Gillman, Kobie Coburn • 2 workshops held • 70 participants total West Virginia Department of Education

33. Workshops by cluster continued Business and Marketing Education • Coordinators—Abby Reynolds and Betty Sias • 2 workshops • 48 participants total West Virginia Department of Education

34. Workshops by cluster continued Health Science Education • Coordinators — Cynthia Sundstrom and Becky Davis • 1 workshop • 30 participants West Virginia Department of Education

35. Workshops by cluster continued Human Services • Coordinators — Kathy Mace, Susan Rice, Donna Wilkes • 3 workshops • 48 participants total West Virginia Department of Education

36. Follow ups • Follow ups were conducted by telephone and in person when possible when questions or conflicts arose during the test building process. West Virginia Department of Education

37. Editing and revision process • CTE Coordinators reviewed performance assessments for content accuracy. • Industry professionals reviewed randomly selected performance assessments for content validity, accuracy, and applicability to field. West Virginia Department of Education

38. Editing and revision process continued • CTE Coordinators and the Assessment Coordinator revised and edited to ensure conformity to an acceptable CTE format and assessment parameters. • Assessment Coordinator made final edits for bias, mechanics/spelling, and formatting. West Virginia Department of Education

39. The Test Administration Experience Vicki Jenkins

40. Completers • A student who completes the required four courses within their unique concentration • Teacher generated list/WVEIS record • Generate roster of completers for each concentration (16 at JRTI) • Double check roster with administrator West Virginia Department of Education

41. Preparing • Rational for test-Industry standard • Have teachers who conducted pilot provide testimony/provide handout for organizing testing • Encourage teachers to “talk it up” with students West Virginia Department of Education

42. Preparing continued • Set testing dates and contact & confirm dates with home schools (check for calendar conflicts) • Publish testing schedule-distribute to all parties • Announce in newsletters, put on Edline for parents, letter to parents West Virginia Department of Education

43. Pre-assessment responsibilities • Secure materials upon arrival (in locked location) • Copy assessment booklets, gather necessary number of pens/pencils/highlighters • Make student test packets • Prepare “do not disturb” signs West Virginia Department of Education

44. Pre-assessment responsibilities continued • Prepare class packets with all required information - manual, manipulatives, signs, pencils, etc. • Order thank you gifts and materials for judges • Order supplies/materials for each test West Virginia Department of Education

45. Trainings • Copy the Test Administration Manual for each faculty member participating in testing • Complete training for teachers • Review IEP and 504 requirements • Sign agreements/maintain on file West Virginia Department of Education

46. Trainings continued • Emphasize security agreements, i.e. procedures, directions, rubric, reporting procedure breach of security • Inform examiners about checking for pertinent information • Review return procedures for all materials West Virginia Department of Education

47. Trainings continued • Review responsibilities for evaluators’ training, complete evaluator’s training and/or confirm with examiner that examiner training has been completed • Provide folder with assessment materials to each evaluator • Prepare cards containing the test irregularity site-distribute to examiners/evaluators West Virginia Department of Education

48. Post-assessment responsibilities • Secure materials after testing • Enter scores by required date • Data Management Profile System (operated by state) • Maintain hard copies for 3 years • Rubrics, answer sheets, etc. (documents with recorded scores) West Virginia Department of Education

49. Post-assessment responsibilities continued • Ask examiners/evaluators to share concerns/comments for improvement • Destroy secure materials by shredding • CD’s created for assessment • Performance assessments • Scratch paper • Projects created for assessments West Virginia Department of Education

50. The Student Experience Ryan Haught