Mindy Crain-Dorough, PhD Southeastern Louisiana University mindy.dorough@selu

1. The Evolution of Accountability Reporting:Lessons Learned Over Nine Years By a State Education Agency Mindy Crain-Dorough, PhD Southeastern Louisiana University mindy.dorough@selu.edu Robyn Madison-Harris, EdD Louisiana Department of Education robyn.madison@la.gov

2. Overview of presentation • Background on report development and what Louisiana does • History of Accountability Changes • Problems/Solutions • Recommendations for future directions

3. Purpose of this study • To compile a list of lessons learned as a reference for other states • To document the history of accountability reporting in Louisiana • To explain the complexities of the report development process • To provide recommendations for improvements accountability reporting in Louisiana and other states

4. Report development – Not an easy thing to do • Present complicated information and data in a format that is aesthetically pleasing and understandable • Negotiate many conflicting opinions about content • Turn complicated and detailed policy into computer programs that calculate scores • Coordinating many large data files • Extensive checking of all aspects of calculations and report generation has to be conducted. • All of these must be done in a very rigid timeframe.

5. Why do we go to all this trouble? • Not just to meet federal and state mandates • Parents make decisions about which schools to which they will send their children and even where to live. • Parents are also driven to take action at their children’s schools. • Teachers, principals, and district staff use the results to make decisions about school improvement efforts and funding priorities

6. Louisiana’s choices • Reports produced at the state level for districts and schools • School Report Card for Parents in paper format and sent home with every child • Electronic PDFs of reports are available on LDE website • Assessment and accountability reporting are separate • LDE works with a contractor to produce reports

7. Back to the beginning • Louisiana had report cards as far back as 1996-1997 as part of a legislative mandate • Transformed report cards into accountability reports in 1998-1999

8. School Report Card for Parents

9. Inside

10. Back

11. Changes to reports over the years

12. Along comes NCLB • First reports containing NCLB mandates in 2002-2003 • Louisiana had to incorporate NCLB into policy and thus the existing reports • A dual system created – SPS Component and Subgroup Component • Conducted an evaluation of existing reports for what was and was not working • Prioritized contents • Put highlights in report cards and details in a new report

13. Incorporation of NCLB

14. School Report Card for Parents

15. Inside

16. Back

17. Methodology • Qualitative – authoethnographic, case study, and phenomenological approaches • Data Collection • Authoethnographic recount by authors • Interviews of current and former accountability staff • Document analysis • Data Analysis – cross-case examination and content analysis strategies

18. Problems/Solutions • Accountability system changing every year • Be organized • Planning meetings with key parties • Preparation for planning meetings “Regrouping” at the end of each year

19. Problems/Solutions • Errors in the reports • Improve the checking process every year using a log of problems found in previous year(s) • Include cross-report consistency checking in the process • Create thorough processing rules, “data maps,” and record layouts that work together • Communication between report developers, data calculation staff, and the contractor on these issues is key • Allow districts to review data in aggregated form before inclusion in reports, especially for new indicators • Have emergency plans in place should errors occur • Use everyone’s time effectively – hire temps if necessary to do more tedious checking

20. Problems/Solutions • Too many conflicting opinions about content • Use focus groups to obtain evidence of what works best for consumers • Develop a protocol for making decisions regarding conflicting viewpoints

21. Problems/Solutions • Miscommunication within a state agency • Is the agency structure conducive to functional report development? • Develop an issues database to track issues and their resolutions as well as a process for maintaining the database • Develop a thorough timeline with a process for ensuring everyone is on task and a protocol for revising the timeline should unexpected events arise • Find a method of meeting regularly to discuss issues and timelines that is most effective for all involved – we are currently having weekly conference calls with the contractor and impromptu meetings in-between

22. Problems/Solutions • Vendor relations • Good vendor qualities include flexibility, availability to devote extensive time to the project, advanced technical skills, and good quality assurance practices • Balance of working for us and working with us • Establish protocol for staff turnovers and management changes • Set standards and roles for quality assurance procedures and processes • Regular communication and established communication procedures

23. Problems/Solutions • New mandates (internal and external to state) • Evaluate the current reports for what works and what doesn’t • Prioritize contents of the report (current and anticipated) • Take advantage of the opportunity to make enhancements to the reports that are undergoing major revision anyway • Look at what other states are doing to get ideas

24. Problems/Solutions • Disasters occur • Meet with those most impacted by the events to obtain all information before making decisions • Consider all aspects of the accountability system and brainstorm possible impacts

25. Problems/Solutions • Giving one individual too much responsibility • Have a team of report developers • Rotate lead responsibility for individual reports every two or three years

26. Problems/Solutions • Promoting and supporting report release • Use Press Conferences, written press releases, and radio announcements • Create a “Help Desk” to answer questions about accountability and other agency programs • Provide accountability trainings on policy changes, the importance of data quality, and description of report contents to school and district staff

27. Problems/Solutions • Making reports useful • Brainstorm areas in the agency that can be highlighted in the report cards • Provide a message to parents in the report card from the superintendent to inspire parents to become more involved in their child’s education

28. Validity and reliability • Validity • Confirm valid use of reports through focus groups and training • Reliability • Thorough and detailed checking process

29. The current “Parent Report Card”

30. Recommendations • Transition to interactive accountability reporting and tie to school improvement efforts • Combine assessment and accountability data • Reduce the number of releases and release the Parent Report Cards earlier • Re-evaluate the content and purposes of the current reports • Have an “Accountability Czar”