American Institutes for Research February 2005

1. A Brief History of the NRS American Institutes for Research February 2005

2. Overview • Wave 1: Development of the NRS • Pre-NRS: Emergence of accountability • Development process • Wave 2: Implementation • Refinement of measures and procedures (1999–2001) • State implementation(2000–2003) • Training and T/A on requirements, data quality, data use • Wave 3: Focus on Improvement

3. Emergence of Accountability • General trend toward accountability for education and employment programs (late 1980s—early 1990s) • Quality Indicators (1991–92) • Required by National Literacy Act • Federal (model) indicators • State indicators

4. Quality Indicators • Assessed local program performance in: • program planning • recruitment • retention • student learning • professional development • curriculum and instruction • support services

5. Quality Indicator Approach • Primarily qualitative measures, focused on program process—what a program does • Less emphasis on student outcomes • Lacked uniform, quantitative measures • No standards • No rewards/incentives • Not fully implemented in states—voluntary

6. Emergence of NRS • 1993: Government Performance and Results Act (GPRA)—required all Federal agencies to develop indicators of program performance to demonstrate they are meeting their goals. • 1995: Adult education program threatened with elimination into workforce development block grant system. Lack of data on program effectiveness cited. • 1996: State directors and U.S. Department of Education develop a framework for a system of program accountability. • 1997: National stakeholder meeting to validate framework and identify outcome measures.

7. 1997 Development Meeting • Included all state directors • Formal process to identify goals of adult education • Identified measures for each goal • Endorsed development of formal accountability system, leading to first NRS development project

8. Goals and Measures Identified • Educational improvement and lifelong learning • Enhancement of employability • Improve civic participation • Improve literacy and education within the family

9. NRS Development Project: 1997–98 • Collaborative process of DAEL, state directors, other stakeholders • Broad group identified basic program goals and measures • Advisory board refined measures and procedures • Key decisions brought to all state directors before final resolution

10. Key Components of Accountability Systems • Goals—reflect program purposes • Measures—tied to goals • Performance Standards—establish level of performance on measures • Rewards and/or Sanctions—for exceeding or falling below standards

11. Refinement by Advisory Board(1997–98) • Board included state directors, literacy volunteer agencies, experts in accountability • Concluded there were too many measures, some difficult to collect • Applied criteria for inclusion

12. Criteria for Including Measures • Meet WIA requirements • Maintain state and local control, flexibility to meet learner needs • Minimize burden • Accommodate methodology to produce valid and reliable national data • Definitions and procedures compatible with other agencies

13. WIA Passage • Required four types of measures—from NRS development work • Requires measures tied to state performance standards. • Rewards for states exceeding standards • Allows states sanctions—states must “consider” local performance.

14. Optional Measures and Tables • Include other measures that state directors had identified and considered • Could not be required due to WIA • Also concern on increased burden, difficulty in collecting and defining • Optional tables created, with possible formal inclusion if widespread state use

15. Wave 2: Implementation • Transitional year 1999 • Guidelines published, national training, state technical assistance • Performance standard setting process • Refinements to measures and procedures • Definitions, procedures, tables refined • Data quality checklist • Data quality guide and training

16. Wave 3: Improvement • Implement new polices and procedures: WIA reauthorization • Continue to refine NRS procedures and measures • Stronger focus on state monitoring of local programs • Use data for program administration and improvement