Contemplating the End: How Assessment Can Help with Retention

1. Contemplating the End: How Assessment Can Help with Retention Robert Mundhenk AAACL Annual Conference 8 April 2009

2. The Age of Accountability • Once upon a time, colleges and universities were like doctors. . . . • Then tuition and greater consumer awareness rose • Then higher education become a political topic

3. The Age of Accountability • The Spellings Commission • Early stirrings: • Threatened regional accreditation • Proposed standardized testing • Argued that colleges were failing to serve their public responsibility—and that they cost too much • Results: • None of the above, but---

4. The Age of Accountablity • The charges remained • Higher Education Act required increased reporting of outcomes, especially in teacher training • State legislatures and boards become increasingly interested in outcomes • “Voluntary” System of Accountability: • Enrollment and graduation data • Institutional climate data • “Engagement” information • Standardized test data on critical thinking

5. The Age of Accountability • ….but exactly what these “outcomes” are is still very murky • Thus “success” differs state to state and institution to institution

6. Context: Accreditation • Original purpose was to ensure quality • Peer Review • Extreme confidentiality • Focus traditionally on quantity and processes rather than quality and results • Rarely concerned with learning

7. Context: Accreditation and Assessment • Mid-80’s: Accreditors began to require some attention to assessment of learning • Mid-90’s: Focus on structures and data collection • Mid-00’s: Focus of use of information to improve outcomes

8. Assessment and Accountability: Serving Two Masters • Not always aligned values institutionally; thus not always aligned activities • Assessment has moved toward improvement of learning outcomes, which tends to appeal to faculty while nonplussing external stakeholders

9. Assessment and Accountability: Serving Two Masters? • Accountability efforts seem aimed at external stakeholders seeking “transparency” through broader, statistically based institutional outcomes—but these efforts seem pointless and bureaucratic to faculty

10. Why the Shift toward Accountability? • A need to determine degrees of success • Cost • Accountability data are relatively easy to collect—and have been collected and reported for years—while learning outcome data are new and not particularly comparable

11. Can Assessment of Learning and Accountability Efforts Be Compatible? • They ought to be • Serving two audiences but essentially the same purpose, the success of students

12. Aligning Assessment and Accountability: Data Set 1 Public Four-Year Totals Re-en AnnRetCumGr Success 1992 8,929 64.5 72.2 0.1 72.3 1993 10,296 64.5 72.4 0 72.4 1994 9,804 63.4 72.4 0 72.4 1995 9,655 64.7 73.8 0 73.8 1996 9,624 65.7 74.7 0.1 74.8 1997 9,928 65.2 74.7 0.1 74.7 1998 10,406 66.6 75.5 0 75.6 1999 10,684 66.5 75.7 0 75.8 2000 10,480 68.6 77.3 0 77.3 2001 10,643 68.6 77.2 0.1 77.3 2002 10,861 68.4 76.7 0.1 76.8 2003 11,527 67.7 75.9 0.3 76.2 2004 11,497 67.5 75.5 0.1 75.6 2005 12,042 68.5 76.6 0.2 76.8 2006 12,083 68.3 77.0 0.2 77.1 2007 12,248 67.0 75.4 0.5 75.8 2008 12,781

13. Aligning Assessment and Accountability: Data Set 2 Public Two-Year Totals Re-en AnnRetCumGr Success 1992 3,885 35.9 41.4 16.0 57.4 1993 3,785 38.8 44.7 12.3 57.0 1994 3,018 41.2 46.1 10.5 56.6 1995 3,498 44.3 49.2 6.5 55.7 1996 3,718 44.5 50.3 5.6 55.9 1997 4,133 45.5 51.4 4.0 55.4 1998 4,639 46.7 53.0 4.2 57.1 1999 5,041 48.2 53.9 3.5 57.4 2000 4,688 49.2 55.4 2.9 58.3 2001 5,336 51.4 56.7 3.2 59.8 2002 5,726 49.6 55.4 2.9 58.3 2003 6,107 48.0 53.2 5.3 58.4 2004 6,059 48.8 54.0 4.9 58.9 2005 6,179 46.9 52.3 4.8 57.1 2006 5,828 48.0 53.2 4.9 58.1 2007 6,678 49.4 54.0 5.7 59.7 2008 6,594

14. Aligning Assessment and Accountability: Data Set 3 Total Public Re-en AnnRetCumGr Success 1992 12,814 55.8 62.9 4.9 67.8 1993 14,081 57.6 65.0 3.3 68.3 1994 12,822 58.2 66.2 2.5 68.7 1995 13,153 59.3 67.2 1.8 69.0 1996 13,342 59.8 67.9 1.6 69.5 1997 14,061 59.4 67.8 1.2 69.1 1998 15,045 60.4 68.6 1.3 69.9 1999 15,725 60.6 68.8 1.1 69.9 2000 15,168 62.6 70.5 0.9 71.5 2001 15,979 62.9 70.3 1.1 71.5 2002 16,587 61.9 69.3 1.0 70.4 2003 17,634 60.9 68.0 2.0 70.0 2004 17,556 61.0 68.1 1.8 69.8

15. Using Retention Data • What conclusions can we draw from these data? • What kinds of decisions can we make on the basis of these data? • How can these data help us improve learning?

16. Aligning Assessment and Accountability: Data Set 4 • Graduation Rates at Arkansas Public Universities (Source: U.S. Department of Education College Navigator) A 38% B 39% C 33% D 34% E 20% F 33% G 33% H 58% I 18% J 44% (Data do not include transfer-out students)

17. Drawing Inferences • In what ways do the graduation data qualify the retention data? • What conclusions can be drawn from these data?

18. A Qualification: • Alexander Astin (“How good is your institution’s retention rate?,” 1993): Graduation rate depends on the characteristics of the students who initially enroll • “…some institutions with ‘high’ graduation rates should really have rates that are even higher….other institutions with graduation rates that appear to be very modest are actually retaining their students at a significantly higher rate…”

19. Finding Answers • Retention is a worthy end in itself • Raw data on retention or graduation rates give no clues about ways to improve or increase retention and graduation rates • Assessment data and strategies may be a way to process raw data

20. Using Assessment to Improve Retention • What data sets or assessment assumptions currently in place might be used to help in understanding retention information? • What data sets or assessment assumptions might be employed to improve retention?

21. Thinking about the End: Why We Assess • Self-justification (focus on data) • Improvement of instruction (focus on teaching) • Improvement of learning (focus on outcomes)

22. An Assertion • We assess the achievement of student learning outcomes because we want to know the degree to which we have succeeded in helping students achieve the learning goals we have set for them—and caring about learning and its improvement may actually be a way to improve retention

23. A Validation of the Assertion • Frank Heppner, “On the bottom line, good teaching tops good research,” Chronicle of Higher Education, 13 March 2009: • “Can faculty members be trained to be more effective teachers and so have an impact of retention? Absolutely.”

24. A Validation of the Assertion • If better teaching produces better retention, one way to improve teaching is to provide faculty with meaningful information that tells them the degree to which students are achieving student learning outcomes • Thus better information about learning can produce better teaching, which may improve retention

25. What about Other Ways in Which Assessment Information May Have a Positive Effect on Retention?

26. How Assessment Work Can Help Retention • Focus on outcomes can improve: • Student understanding of expectations • Institutional articulation of expectations • Syllabi and planning of experiences so that they emphasize outcomes • Grading and/or prioritization of grades

27. How Assessment Work Can Help Retention • Formative assessment can help students understand expectations and grading strategies—and to make progress toward important outcomes over time (Mantz Yorke, “Formative Assessment and Its Relevance to Retention,” Higher Education Research & Development, 2 (2001), 115-126.

28. How Assessment Work Can Help Retention • Concentrating on outcomes focuses the conversation internally and externally: • Learning becomes center of improvement efforts • Learning (not numbers) becomes center of conversations about effectiveness

29. How Assessment Work Can Help Retention • “Drilling Down”: Assessment data can provide detailed information about the raw retention rates: --breakdown of cohorts by various characteristics --tracking of student experience --correlating predicted retention with actual retention

30. How Assessment Work Can Help Retention • Correlating institutional mission, outcomes, and resources with graduation and retention data --to what degree are outcomes achieved? --have students achieved their desired objectives?

31. Assessment and Retention • Assessment data cannot directly affect retention rates, but they can clarify their meaning • Assessment data and strategies can affect retention indirectly by improving student and staff understanding of their common learning goals

32. Assessment and Retention • Assessment data and strategies can help institutions understand why their retention rates are where they are—and to determine how best to deal with them

33. For Further Information: Robert T. Mundhenk Consultant, Higher Education Assessment asmt357@aol.com