Quality Assurance in a Chaotic World

1. Including Quality Assurance Within The Theory of Action Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012 Quality Assurance in a Chaotic World

2. Quality Assurance as a Continuous Cycle • Quality Assurance - A continuous cycle that improves and refines the processes necessary to produce valid assessments to the highest standards. Ensures that excellence is inherent in all components of the process. • Plan • Do • Study • Act

3. Edcount’s relevant work: • Evaluated assessment systems in most states and the U.S. territories through the federal standards and assessment peer review process. • Developed interpretative arguments for the validity of assessments and conducted studies to test the claims and assumptions in the interpretive argument for a number of states. • Provided technical assistance for reviewing the technical quality of assessment and data quality of assessment systems.

4. Quality Assurance Within the Theory of Action - Claims • Students are appropriately tested for identified testing options • Test scores accurately reflect student knowledge and skills • Reports provide valid scores and are used appropriately

5. Students Are Appropriately Tested for Identified Testing Options – Special Studies • The Learner Characteristics Inventory (LCI) • A survey designed to collect information about the population of students who participate in alternate assessments based on alternate achievement standards (AA-AAS). • Helps states identify and evaluate “who” is participating in the alternate assessments. Provides information to help direct participation rules and guidance.

6. Students Are Appropriately Tested for Identified Testing Options – Special Studies • Accommodation Studies • The extent to which accommodations are provided on the state assessments. • Literature review on best practices for accommodation use. • Provides information and evidence to improve best practices, guidance, and monitoring of accommodation use.

7. Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies • Performance Level Descriptor (PLD) studies • Examine the extent to which assessment cut scores appropriately distinguish between student performance levels, and whether the performance level descriptors adequately capture meaningful distinctions in grade-level performance. • Engage teachers in rating and feedback processes to help administrators understand how teachers use and interpret PLD’s • Evaluate the PLD’s for coherence, consistency, and accuracy.

8. Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies • Performance Level Descriptor (PLD) studies cont. • Provides information to help states refine their standard setting processes, focus their professional development and support their interpretations on the state assessments.

9. Test Scores Accurately Reflect Student Knowledge and Skills – Specialized Studies • Alignment Studies • Evaluations of the alignment between academic content standards and alternate assessments. • Evaluations of the alignment between academic content standards and general assessments. • Provides information to support and improve the design and development of assessments.

10. Test Scores Accurately Reflect Student Knowledge and Skills –Special Studies • Cognitive Labs • Collect information about whether assessments are eliciting the intended cognitive processes. • The cognitive labs can help states understand why students may struggle with certain items or content areas on an assessment, and, as a result, provide feedback about ways to develop accessible assessments and ways to improve instruction.

11. Reports Provide Valid Scores and Are Used Appropriately - Special Studies • Consequential Validity Studies • Evaluate and collect evidence about the intended and unintended consequences of a states large scale assessment system. • Provides evidence to inform policies and processes and focus professional development.

12. Work Within Puerto Rico • Special Studies • PLD study • Accommodation Study • Consequential Validity Evaluation • Cognitive labs • Professional Development • Data Use Workshop • Curriculum Mapping Boot Camps

13. Work Within the Laurent Clerc Center at Gallaudet University • Special Studies • PLD studies • Cognitive Process Validity Study • Technical Assistance For: • Development of a balanced assessment system with formative and interim assessments • Development of an accountability System – Small school review • Development of a data management system

14. How Did the Work Inform Quality Assurance and Processes • Provided focus for professional development • Provided information to improve the design and development activities for assessments and reports • Provided context as well as empirical evidence to support policies and processes

15. Contact Information • Jennifer Stegman, Senior Associate • jstegman@edcount.com • edCount LLC • 5335 Wisconsin Avenue NW • Washington DC 20015 • edCount.com