Assessing and Improving Student Outcomes

1. Assessing and Improving Student Outcomes Presentation for TENNAIR Conference August 7, 2008 Gwen P. Aldridge, Ph.D. Director of Assessment J. Nevin Robbins, Ph.D. Executive Director for Planning and Analysis

2. Presentation to include: • Importance of SLOs • Defining Learning Objectives • Selecting Measures & Setting Benchmarks • Comparing Outcomes with Intended Objectives • Using Results to Improve Learning • Online Documentation – Southwest’s System • SLO Hints

3. Current Focus on Importance of Student Learning Outcomes This focus represents a convergence of two major trends in higher education: the assessment movement and the accountability movement.

4. Assessment and Accountability • Assessment has evolved from efforts to improve the quality of teaching and learning by improving the quality of educational outcomes • Accountability has evolved from the efforts of state legislatures to prove institutional effectiveness in making higher education more cost effective

5. Tennessee Academic Audits for Programs • Criterion 1.1: The faculty completed an honest analysis of their process for developing learning objectives for the program, considering measurability, clarity, and whatstudents need to know. • Criterion 1.2 : The faculty have documented or proposed a process for developing learning objectives that are based on realistic and appropriate evidence. • Criterion 1.3: The faculty have documented or proposed specific plans to take best practices and appropriate benchmarks into account in the analysis of learning objectives.

6. Good Assessment Practice • First - assess what is most important • Realize that anything that can be taught or learned can be assessed • Apply assessment at the course, program, and institutional levels • Organize every course and every program around clearly defined learning goals and objectives, explicit assessment methods, and measurable outcomes.

7. Program Outcomes • Are needed for internal assessment of continuing improvement • Are needed for external assessment of institutional effectiveness for accountability purposes • Are useful in communicating program results for purposes of marketing • Insure that the program is more than just a cluster of courses

8. NWCC The Purpose of Student Outcomes Assessment Effort Academic Program Improved Accreditation Requirements Satisfied Effectiveness Demonstrated

9. What SLOs can accomplish: • To tell students what is expected of them and thus… • help students learn more effectively • make it clear what students can gain • help instructors design their materials more effectively • help instructors select the appropriate teaching strategy • assist in setting examinations based on the material developed • ensure that appropriate assessment strategies are employed

10. SACS Compliance Standard 3.3 Institutional Effectiveness • 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas (Institutional Effectiveness):

11. 3.3.1.1 educational programs, to include student learning outcomes • 3.3.1.2 administrative support services • 3.3.1.3 educational support services • 3.3.1.4 research within its educational mission, if appropriate • 3.3.1.5 community/public service within its educational mission, if appropriate

12. Simply… the 4 Column Model: • Learning outcome • Method of assessment • Assessment results • Use of results

13. Assessment Learning Cycle

14. From SACS-COC Resource Manual for Review Teams “The expectation is that the institution will engage in ongoing planning and evaluation to ensure that, for each academic program, the institution develops and assesses student learning outcomes.” Note the “each” in the expectations – it’s not “some”.

15. From SACS-COC Resource Manual for Review Teams - continued “Program and learning outcomes are grounded in the faculty’s knowledge of the content and coherence of the discipline as well as in the learning process and reflect expectations for performance consistent with the level of the program and the mission of the institution.”

16. From SACS-COC Resource Manual for Review Teams - continued Sample Documentation for this requirement: Representative examples of program and learning outcomes for each educational program Descriptions of methods for evaluating student achievement of these outcomes Reports of the results of the evaluation, examples of how the results have been used for program improvement, and examples of how methods of evaluation have been improved over time

17. From SACS-COC Resource Manual for Review Teams - continued “At appropriate intervals, program and learning outcomes and evaluation methods are evaluated and revised as appropriate.”

18. From SACS-COC Resource Manual for Review Teams - continued “Program and learning outcomes specify the knowledge, skills, values, and attitudes students are expected to attain in courses or in a program.” “Methods for evaluating the extent to which students achieve these outcomes are appropriate to the nature of the discipline and consistent over time to enable the institution to evaluate cohorts of students who complete courses or a program.”

19. NWCC UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS

20. Column 1: DefiningLearning Objectives The dream begins with a teacher who believes in you, who tugs and pushes and leads you to the next plateau, sometimes poking you with a sharp stick called "truth."  • ~Dan Rather

21. NWCC Promising practices in Defining Learning Objectives • Make certain that Learning Objectives • Are specific to the program • Are focused on what is critical to the program • Are descriptive of the what a student should gain as a result of their completion of the program • Are clear and understandable • Are written to a level of specificity while allowing a certain amount of interpretation leeway for faculty • Use action verbs • Are realistic given the typical student in the program • Are assessable

22. NWCC Is outcome singular? Singular – Statement of One Outcome Don’t bundle multiple outcomes in a single statement. This leads to problems in formulating means of assessment.

23. NWCC When writing SLOs, the focus should be on observable outcomes and an “action verb” can provide that focus. SLOs usually begin with something like: By the end of the secondary education program, students will be able to design curriculum and appropriate instruction. Design is the “action verb” in this example.

24. NWCC

25. NWCC

26. NWCC Statements of Educational Outcomes should answer WHAT? What students should: Know (cognitive) Think (affective) Do (behavioral) when they have completed a degree program.

27. NWCC How many SLOs are enough? A program may have only a few SLOs or a long list of them. A total of five to seven outcomes is typical for a program. Most career-technical programs have standardized student competency profiles and instructors may select the learning outcomes from these. An instructor should not list each competency, but should include the ones deemed most important.

28. NWCC SLOs are: Specific _______ (who or what will be changed) Time-Based __________ (when will the changes take place) __________ (how much change will there be) Measurable Goals are necessary for improvement because when we shoot at nothing, we usually hit it.

29. NWCC Evaluating SLO Statements Are they reasonable given student ability? Are they clear? Is accomplishment measurable? Is outcome singular (not a bundle of outcomes)? Do they focus on what students can know, think, or do as a result of completing the program?

30. A Citation for One Institution Regarding Learning Outcomes: 3.4.1 Non-Compliance: The On-Site Committee should seek evidence that the college establishes and evaluates student learning outcomes for each of its educational programs. • Comments from the Off-site team: The execution of the student learning outcome system varied from effective and accurate in the description of student learning outcomes and their assessment to ineffectively broad in defining the student learning outcomes and assessments not keyed to the desired outcomes. For example, certain programs effectively and descriptively focused on what students could do, know or believe as a result of completing their programs and also used assessment tools directly addressing the expected outcomes. Other programs presented groups of student learning outcomes from courses making up the program or from a course sub-section as one learning outcome.

31. Identify the problem here: Unit: Commercial Truck Driving • The students will demonstrate the ability to safely operate commercial trucks.

34. Column 2: Selecting Measures & Setting Benchmarks • Are Assessments Keyed to the Desired Outcome? • Are Assessments Reasonable Given Student Ability? • “Fledgling skiers make the most progress when they are pushed outside their comfort zone, but not so far that they are scared off the slopes altogether.” a ski instructor We’re #1!!! Mississippi - #1 in Poverty, #1 in Obesity, #1 in Teenage Pregnancies……

35. Assessment of the Outcomes Each SLO should have its own assessment. At the end of the assessment, an instructor should know which SLOs were achieved, and which were not achieved. If goal related questions are used on a final exam to assess more than one outcome, an item analysis should be done. For example, 80% of students achieved Outcome 1, 60% of students achieved Outcome 2, 75% of students achieved Outcome #3…

36. NWCC Without such individual assessment of outcomes, the instructor fails to know in the end what was accomplished and thus what individual changes in instruction should be made in improving Student Learning in the program. The ultimate purpose of writing and assessing SLOs is Program Improvement.

37. NWCC • Some acceptable measures to evaluate SLOs: • standardized tests with questions tied to specific SLOs • locally developed tests with SLO goal-related questions identified (use item analysis to judge performance on each individual outcome) • analysis of theses, portfolios, recitals, speeches • student performance in lab activity

38. NWCC Assessments…Are they reasonable given student ability? Make sure criteria (bench marks) for success and expected student achievements are not set at levels farbeyond the ability of most students. We should STRETCH, but not STRAIN.

39. Individual Student Grading and Educational Outcomes Assessment

40. NWCC Class Grades or G.P.A. as a Means of Assessment • Why not just use the course grades as assessment? • With course grades, we have assessment of individuals, not programs • Also, the objectivity of the evaluator is questioned • Finally, an instructor is not provided with information that allows that instructor to assess student accomplishment on the individual SLOs • (Generally grades are not accepted as a mean of assessment for SLOs).

41. Biology I Classes Grading and Educational Outcomes Assessment

42. NWCC Citation for NWCC from SACS-COC 3.4.1 Non-Compliance Recommendation: The On-Site Committee should seek evidence that the college establishes and evaluates student learning outcomes for each of its educational programs. Comments from the Off-site team: …One program listed all student learning outcomes from each course, but used class management and course requirement standards as assessment means. In assessing learning outcomes, most programs used a retention indicator instead of a learning outcome indicator, that of course completion, or a measurement tool that did not address the desired outcome. In those cases a “C” grade in a course or a comprehensive exam grade without targeting portions of the exam relevant to the learning outcome were presented as proof of achieving the outcome.

43. Art: Students will demonstrate mastery of hand building pottery techniques by creating forms with pinch pots, coiled symmetrical cylinders, and crafting coil and slab combinations.

44. Art: Students will demonstrate mastery of hand building pottery techniques by creating forms with pinch pots, coiled symmetrical cylinders, and crafting coil and slab combinations.

45. With “Assessments not keyed to desired outcomes…”

46. Column 3: Comparing Outcomes with Intended Objectives At some point, you just have to decide that you are not going to be afraid of the data. It may not be pretty… But you can use it.

47. NWCC Column 4: Using Results to Improve Learning How might we use the results? We may decide to… Change the related SLO Adjust the Program/Unit Activities Modify the Means of Assessment Who Kissed the Mirror?

48. NWCC The End Product of Assessment Activities Assessment results are used for Improving Educational Programs…. Thus Column 4, our “Use of Results”. Ever tried. Ever failed. Try again. Fail again. Fail better. Samuel Beckett

49. Statement of “use of results” • The Use of Results should be stated in past tense - an action that has already taken place to improve the program. • But, what if the improvement won’t take place until the following year (semester)? You may say, “A decision was made to begin…….. in the upcoming semester.” Make the decision now.

50. NWCC UNIT ANNUAL OUTCOME OBJECTIVES EDUCATIONAL PROGRAMS