WASC Evaluator Workshop Fall Visits 2009

1. WASC Evaluator WorkshopFall Visits 2009

2. Workshop Outcomes Understand WASC’s three-stage process and how your visit fits into the process Be familiar with the WASC Standards and CFRs and how to use them Know how to prepare for and conduct an effective visit and produce a useful, high-quality team report Be prepared to make sound judgments about institutions under the Standards Be familiar with resources that support your work on a team

3. Agenda • Context for the Visit/Accreditation • WASC Three-Stage Review Process • Standards and CFRs • Preparing for the Visit • Conducting the Visit • Developing Team Recommendations • Writing the Team Report • After the Visit

4. Context for Accreditation and Visits • The Continuing Evolution of the WASC Process and Standards • The Accountability Movement • Retaining Peer Review • The Impact of the Economy • Value Added, Collaboration, and Ongoing Efforts to Refine and Improve

5. The WASC Review Process

6. Three-Stage Process 1. InstitutionalProposal: Identifies priorities, themes/areas of emphasis, and outcomes. Aligns work with institutional plans and needs. 2. Capacity/Preparatory Review: Focuses on capacity (systems, policies, resources) and readiness for educational effectiveness. 3. Educational Effectiveness Review: Focuses on results, findings.

7. The Two Reviews

8. The CPR and EER as a Whole The CPR evaluates what an institution has for infrastructure (staff/faculty, resources, processes, facilities, systems, structures). The EER evaluates how well that infrastructure works and the results that the institution achieves.

9. Navigating Multiple Purposes (1)

10. Navigating Multiple Purposes (2)

11. The Special Visit • Intended to monitor institutional progress on issues identified by the Commission • May or may not be connected to a sanction • Limited to a few specific areas of concern • Intended to assess how institution will move into compliance (if on sanction)

12. Understanding the Team’s Impact Why were you chosen for a team? • Peer review is the foundation of accreditation. • The team report forms the basis for the Commission action and its letter. • The team report and action letter inform the work of the institution for years to come.

13. Working with the Standards and CFRs

14. Understanding the Standards and CFRs • Two Core Commitments: Capacity and Educational Effectiveness • Standards: Broad, holistic, encompassing • Criteria for Review: More specific and detailed • Guidelines: Ways to demonstrate compliance with the relevant CFR

15. Using the Standards and CFRs • Team judgments must be linked to specific Standards and CFRs • CFRs must be cited in reports • Standards and CFRs form the basis for Commission decisions • Standards and CFRs provide a context for continuous quality improvement

16. STANDARD 1:Defining Institutional Purposes and Ensuring Educational ObjectivesInstitutional PurposesIntegrity

17. STANDARD 2:Achieving Educational Objectives Through Core FunctionsTeaching and LearningScholarship and CreativitySupport for Student Learning

18. STANDARD 3:Developing and Applying Resources and Organizational Structures to Ensure SustainabilityFaculty and StaffFiscal, Physical & Information ResourcesOrganizational Structures & Decision-Making Processes

19. STANDARD 4:Creating an Organization Committed to Learning and ImprovementStrategic Thinking and Planning Commitment to Learning and Improvement

20. Tool: Standards at a Glance

21. Comparing the Two Visits: Different Views of a CFR (2.6)The institution demonstrates that its graduates consistently achieve its stated levels of attainment and ensures that its expectations for student learning are embedded in the standards faculty use to evaluate student work.

22. Capacity and Preparatory Has the institution defined expected levels of attainment for SL? Are they embedded in the standards and measures for student work? How does the institution know if students are meeting expectations? What data are collected and how analyzed? How is student learning measured? Educational Effectiveness What do data show about student learning? Are data disaggregated and analyzed? Did the students learn what the faculty intended them to learn? At what levels of performance? Has the institution used data to make changes and/or improvements? CFR 2.6: Two Views

23. Changes in 2009: • Implement changes to Institutional Review Process re: Student Success, Program Review and EE Sustainability • Implement changes to CFRs • Clarify the scope of the CPR visit to review the “infrastructure” for assessment of student learning • Examine Program Review and Program-Level Student Learning in a systematic way • Allow teams more time together on visits Tool: Table A & B(RB pg. 47)

24. Preparing for the Visit(Visit Guide, Part II, pp. 29-52)

25. Timeline For CPR/EER Reviews 12 weeks 2 months Institution mails report to team and WASC Team holds conference call Site visit held and team report written Institution responds to errors of fact in team report Institution responds to final team report Commission acts at February or June meeting

26. Roles and Responsibilities of Team Members and Staff • Role of team chair (RB pg. 189) • Role of team assistant chair (RB pg. 191) • Role of assigned WASC staff liaison (VG pg. 7) • Team assignments

27. Pre-visit Preparation • Read all the documents from WASC • Standards, CFRs, policies, visit guide, rubrics • Background documents re: institution and purpose of the visit, including Proposal and/or last action letter/team report • Read the institutional report • Review the data portfolio and exhibits • What to look for and highlight? Tools:Timeline (VG pg. 8, VG pg. 29)

28. Reviewing the Exhibits • Enrollment data • Headcounts and FTE • Graduation data • Faculty data • Key financial indicators • Inventory of Educational Effectiveness Indicators • Inventory of Concurrent Accreditation and Key Performance Indicators Tool: How to Review WASC Data Exhibits (RB pg. 61)

29. Reading the Report • Has the institution done what it said it would do in its Proposal? • Has it collected and analyzed data effectively? • Are its conclusions supported by evidence? • Are there serious problems or potential areas of noncompliance? • Does the report contain recommendations for further institutional action?

30. Developing Visit Strategies and Lines of Inquiry • What are areas needing clarification and/or more information? • What are the major issues challenging the institution? • What is raised by the themes that needs to be verified or explored? • What are the strategies that will be most effective?

31. Worksheet for Team Conference Call • Organizes team’s responses to institutional materials • Helps team make preliminary evaluation under the Standards • Provides basis for team to work toward consensus • Should be submitted in advance of call Tool:Team Worksheet (VG pg. 43)

32. Team Conference Call • Evaluate quality of institutional report and alignment with Proposal and previous action letter(s) • Identify areas of good practice, improvement, and further inquiry • Identify issues, strategies, evidence needed • Identify persons and entities to be interviewed • Make or refine team assignments • Plan visit logistics

33. Off-Campus Sites and Distance Education Programs Prior to Visit: Sites will be identified and assignments made • Review substantive change action letters to determine if issues have been identified • Develop plan for the review of the programs and/or sites During Visit • Interview faculty, administrators and students • Evaluate facilities OR online infrastructure • Observe classes • Document visit and findings in appendix • Discuss important findings with team for inclusion in report, as appropriate Tools:Protocols (RB pg. 158, RB pg. 160) Forms(RB pg. 55, RB pg. 58)

34. Compliance Audit • Required for: • Institutions seeking Candidacy and Initial Accreditation • Some institutions under sanction • Additional report submitted by institution in advance of the visit—with links to documents Tool: Compliance Audit Checklist (RB, pg. 51)

35. Determining Strategy for CPR Visit • What evidence is provided to show capacity and readiness for EE? • Why was it chosen? • What are the strengths and weaknesses of the evidence? • What other evidence do you want to review to evaluate capacity and preparation for EE? • Do any issues arise with regard to the Standards? • Meetings: format/methodologies

36. Determining Strategy for EER Visit • What evidence is provided to show EE? • Why was it chosen? • What are the strengths and weaknesses of the evidence? • What other evidence do you want to see to evaluate effectiveness? • Do any issues arise with regard to the Standards? • Meetings: format/methodologies

37. Drafting in Advance of the Visit • Assistant Chair draft outline of team report and context sections • Team members draft outline or text for which they are responsible, using data from institution, with space for additional data, analysis and conclusions Tool:Team Reports (VG pg. 53)

38. Conducting the Visit

39. Process of Visit • Team meets at start of visit to confirm roles, assignments, logistics, and agenda • Team meets frequently re: observations, emerging recommendations, and issues • Team members draft sections of report and turn in to assistant chair on the last day • Team agrees on report recommendations and confidential recommendation to Commission

40. Visit Schedule • Executive sessions and debriefings with team only • Meetings and interviews with key individuals and groups • Open meetings with students, faculty and staff • Document review • Time for drafting report sections • Final exit meeting Tool: Sample Visit Schedule

41. Confidential Email Account • Set up by WASC as extension of open meetings • Checked by assistant chair during visit • Important emails shared with team and investigated • Comments included in team report only if the institution has a chance to address them

42. Approaches Used on Visits • Document review • Interviews and meetings • Mini-questionnaires • Techniques for small and large meetings • Fishbowl exercises • Audits Plan visit methodologies in advance as part of schedule.

43. Document Review Use to: • Check compliance • Evaluate the level of institutional engagement • Examine the evolution of a policy or process • Identify direct and indirect evidence of student and organizational learning • Confirm report claims DO as much as possible in advance

44. Interviews Use to: • Gather information • Explore issues • Build relationships with members of the institution • Validate impressions and observations

45. Tips for Good Interviews • Decide on a protocol for interview • Prepare questions and lines of inquiry in advance • Ask questions that elicit information, stimulate discussion, or require judgment • Avoid interrogation, leading questions, or loaded language • Avoid consultation, giving solutions, or talking about your institution • Let them do the talking

46. Alternative Forms of Interview • Fishbowl • Brainstorm/free discussion on a salient topic • Go-round • Bundling • Audit

47. Evaluating Program Review and Student Learning (EER Visits) Tool: EE ToolkitTool: Suggested Approaches to Evaluating Program Review on EER Visits

48. Rubrics: Assessment of Student Learning • Academic Program Learning Outcomes • Use of Portfolios in Assessing Program Outcomes • Use of Capstones in Assessing Program Outcomes • Integration of Student Learning Assessment into Program Review • Assessing General Education Tool: Program Learning Outcome Rubric

49. Expectations for Two Reviews Use: • ‘Student Learning’ questions (p. 2) • as a monitor to be sure you are within the proper scope of the visit Tool: Expectations for Two Reviews

50. Educational Effectiveness Framework • Use with team to evaluate institution’s “place” • Use language of rubric to describe the institution in the report • Ask the institution to evaluate itself and discuss • Confer with team toward end of visit to mark a copy of the EEF • Submit the marked EEF confidentially to WASC Tool: EE Framework