Making Our Way from Anecdote to Evidence

1. Making Our Way from Anecdote to Evidence Presentation for Georgia College & State University Caroline R. Noyes, Ph.D. Assistant Director, Office of Assessment

2. Background and Introduction

3. Four Purposes of Evaluation (Mark, Henry, & Julnes ,2000) • Assessment of merit and worth: the development of warranted judgments, at the individual and societal level, of the value of a policy or program. • Program and organizational development: the effort to use information to directly modify and enhance program operations. • Oversight and compliance: the assessment of the extent to which a program follows the directives of statutes, regulations, rules, mandated standards or any other formal expectations. • Knowledge development: the discovery or testing of general theories, propositions, and hypotheses in the contexts of policies and programs.

5. Program Mission History: The department has as its mission through the BA in History to provide students with a basic knowledge of selected areas of history; to teach students to think critically in analyzing historical issues, to write clearly and cogently on historical topics, and to conduct historical research; and to provide students with a basic understanding of history as a discipline. Theatre: We seek to provide the best education possible in theatre and dance for our students, preparing them for careers in the professional and educational arenas both in and out of the performing arts. We cultivate imagination, passion, discipline, and collaboration in the creation of arts and artists. We emphasize interdisciplinary connections and a global perspective as essential components of a liberal arts education.

6. Sample Program Goals • Graduates will be able to communicate their knowledge effectively, both orally and in writing. • Graduates will demonstrate knowledge of the major authors, periods, movements, and issues in drama and theater. • Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation.

7. Properties of Effective Student Learning Outcomes • Have a clear purpose • Use action words • Describe meaningful learning • Are easily understandable • Represent high level learning • Result in observable behaviors/products

8. Formulating Student Learning Outcomes • Method • Inventory of program content • Rank most important items • Categorize by outcome type • Draft outcome statement • Revise using criteria for quality outcomes • Use action verbs • Should be observable (either directly or indirectly inferred)

9. Bloom’s Taxonomy of the Cognitive Domain

10. Action Verbs Accompanying Bloom’s Taxonomy

11. Bloom’s Taxonomy of the Affective Domain

12. Action Verbs Accompanying Bloom’s Taxonomyhttp://www.abet.org/_TrainingCD/data/references/Assessment%20Links/Goals_revised_Blooms.pdf

13. Student Learning Outcomes: Oral Communication Graduates will be able to communicate their knowledge effectively, both orally and in writing.

14. Student Learning Outcomes: Written Communication Graduates will be able to communicate their knowledge effectively, both orally and in writing.

15. Student Learning Outcomes: Research Methods (Psychology) Graduates will understand and apply basic research methods in psychology, including research design, data analysis, and interpretation

16. Working on Learning Outcomes in groups

17. GC&SU Core Outcomes • Communicate clearly in written English, demonstrating comprehension, analysis, and critical interrogation of a variety of texts. (A: Essential Skills; English) • Use description, analysis, and synthesis of data, ideas, or information appropriate to the purpose. (A: Essential Skills: English) • Express and interpret mathematical information, concepts, and thoughts in verbal, numeric, graphical, and symbolic form, and to shift among these different modes when solving problems. (A: Essential Skills; Math) • Use appropriate technology in the location, evaluation, analysis, synthesis, and application of information in problem-solving situations (A: Essential Skills; hybrid of Math and English)

18. Finding the Evidence

19. Things to Consider: • What’s already in place? • Exams, assignments, or projects? • Common exams, assignments, or projects? • Writing samples (common question) • Capstone experience products • Look for logical assessment points in the curriculum • At program entrance • Program courses required for the major • At program conclusion • The evidence you collect depends on the questions you want to answer

20. Kinds of Evidence, Part 1 Direct Evidence Indirect Evidence Measure opinions or thoughts about students’ (or alumni) own knowledge, skills, attitudes, learning experiences, etc. Also includes external evaluations. Alumni surveys Exit surveys Employer surveys Focus Groups Measure student learning outcomes directly • Embedded questions/tasks • Pre-post testing • Evaluation of projects/work • Standardized tests (e.g., MFT) • Portfolios • Presentations

21. Kinds of Evidence, Part 2 Quantitative Evidence Qualitative Evidence Observations of student behavior w/ systematic notes Summaries of discussion threads Student reflections Portfolios evaluation Written work, performances, or presentations scored with a rubric Analysis of comments on surveys • Ratings of student skills/performance • Scores and pass rates on tests (licensure/certification) • Scores on locally-designed tests (final exams, qualifiers, comprehensives) when accompanied by test “blueprints” • Classroom response systems (clickers)

22. Student Learning Outcome: answer questions directly and accurately

23. Oral Presentation Rubric (http://www1.uprh.edu/cruzmigu/OralRubric.pdf)

24. Student Learning Outcome: articulate strengths and limitations of various research designs

25. Identifying Sources of evidence in Groups

26. Sharing your Findings

27. At Last: Reporting your Findings • Analyze the data…identify the results • Integrate results from various assessment methods • What conclusions can you draw from that evidence? • What recommendations arise from that data? • What actions should/will you take based on the recommendations?

28. Student Learning Outcome: answer questions directly and accurately

29. Closing the Loop, Option 1 Outcome Practice

30. Closing the Loop, Option 2 Practice Outcome

31. Conclusions The “systematic collection, review, and use of information about educational programs undertaken for the purpose of improving learning” allows us to address three fundamental questions: • What have our students learned? • How well have our students learned that information? • How successful have we been at what we are trying to accomplish?

32. Contact Information Caroline R. Noyes, Ph.D Assistant Director, Office of Assessment Georgia Institute of Technology (404) 894-0510 cnoyes@gatech.edu