Jody Greene Associate Vice Provost for Teaching and Learning Director of CITL

1. citl.ucsc.edu | citl@ucsc.edu | McHenry Library 1330A WDYT Information Session & Workshop Stevenson Conference Center Nov. 7, 2018 Jody Greene Associate Vice Provost for Teaching and Learning Director of CITL Professor of Literature Executive Sponsor for WDYT Project Rebecca Peet SETs Service Manager Faculty Instructional Technology Center Learning Technologies Information Technology Services, UCSC Kimberly Adilia Helmer Chair of COT Teaching Professor of Writing Nic Brummell Member of COT Professor of Applied Mathematics

2. Welcome & Agenda • Revisioning a Teaching Culture: COT and SETs • WDYT: A New Platform, A New Approach • WDYT: An ITS Service Provider Perspective • Crafting Custom Questions: During & End-of-Quarter

3. Revisioning a Teaching Culture: Committee on Teaching & SETs COT SET Workflow: Study Informed SET Changes WDYT Implementation (2018) SETs @ UCSC: Chair Survey (2018-19) COT Charged with SET Policy WDYT Platform Selected & Contract Signed (2015-17) COT SET Study: Winter 2019 SETs @ UCSC: Instructor Survey (2018-19)

4. citl.ucsc.edu | citl@ucsc.edu | McHenry Library 1330A What Do You Think? A New Platform and a New Approach Jody Greene Associate Vice Provost for Teaching and Learning Director of CITL Professor of Literature Executive Sponsor for WDYT Project

5. What’s New in WDYT? • Name change: from “evals” to “SETs” • All end of term instructor surveys are now online • Instructors can see response rates but not individual respondents • TA surveys are online and TAs can access their reports online and can export their own reports • Instructor customization is possible with this system (although not in F18) • Gradual move to questions that are designed to elicit better, less biased, more “formative” information • Some improvements for students as well—fewer emails, better questions, more targeted feedback opportunities

6. The Name Change From “Course Evaluations” to “Student Experience of Teaching Surveys [SETs]” WHY?

7. The Instructor Experience • Your courses appear automatically in student portals as before • This includes graduate seminars • You can access WDYT through Canvas, WDYT, and, after the fact, the DivData portal

8. Response Rates You can see the percentage of students who have responded anytime after the beginning of 9th week You cannot see whether individuals have responded Why is this important? • To help you increase response rates and in case you use Extra Credit

9. The TA Experience • TA surveys are now online • Graduate students will be able to immediately electronically access their own past teaching record without going through department staff. • Grads will need to export this record when they leave the campus, in order to be able to represent their UCSC teaching in future.

10. Customization • You can do mid-quarter surveys OR you can add questions at the end of term. • Mid quarter surveys are known: — to help people develop their teaching — to get students to reflect on their learning — to produce better student responses at the end of term • Anything you customize is private, meaning only you can see the responses

11. Why No Customization in Fall 2018? To give ITS and department staff time to come up to speed and To give COT time to get a really simple user friendly guide to writing good questions

12. Gradual shift in how departmental survey questions are asked & scaled • Declarative statements that represent only the individual student’s experience • Scale (5 possible responses): Can’t evaluate | Disagree | Somewhat disagree | Somewhat agree | Agree • Emphasis on writing questions that focus on student learning rather than faculty’s teaching style or satisfaction with the course • Effort to reduce the possibility for implicit bias Examples: • The lectures were clear helped me clearly understand key concepts • The instructor’s comments on students’my written work were helpful helped me improve my writing • The Teaching Assistant was effective in supporting my learning in the course as a whole. • The instructor was available and helpful outside of class time for scheduled office hours or by appointment

13. Advice for Increasing Your Return Rates and Limiting Bias • Talk to your students about how SETs are used (both to improve our teaching and in every personnel action) • Ask them to give you constructive and specific feedback that you can use to improve your teaching • Incorporate the feedback they give you at mid quarter to make changes to the course, signaling that you take their input seriously • Inform them that bias is a common problem with SETs and ask them to be thoughtful and respectful – this isn’t YELP. • Use class time to allow students to complete SETs

14. citl.ucsc.edu | citl@ucsc.edu | McHenry Library 1330A What Do You Think? An ITS Service Provider Perspective Rebecca Peet SETs Service Manager Faculty Instructional Technology Center Learning Technologies Information Technology Services, UCSC

15. ITS Service Management • Service Management - • Allows ITS to organize the way we manage and present technology • Provides a common language to talk about what we do • Defined Roles, in particular - • Service Manager • Service Executive Sponsor

16. Role - Service Manager • Responsible for overseeing all technological aspects of a service • Integration with other infrastructure services • Training and Support • Continuous improvement • Service Manager for SETs - Rebecca Peet

17. Role - Service Executive Sponsor • Provides direction for the service and executive acceptance of the service. • Jody Greene - is the Executive Sponsor for SETs and is responsible for the policy and pedagogical practice questions related to the service.

18. The SETs service includes • Login with your cruzid and gold password at sets.ucsc.edu • Direct access to WDYT from all Canvas course sites • Training, support and all tiers of troubleshooting • Data integration with myucsc

19. Benefits for Staff • Department staff workload is significantly decreased • Courses are uploaded from MyUCSC with their session and default survey form attached

20. Benefits for Instructors • Login to your dashboard to access reports • See real time response rates • Send your own reminders to your students • W19 - Customization options • Send private mid-quarter surveys to your students • Add course specific questions to the standard end-of-term surveys, results of which are private to the instructor

21. Benefits - Reports • Access - Instructors and TAs will be able to login to WDYT the day after grades are due and see their reports. • Consistency - The same summary and per-respondent reports that are available in WDYT as a pdf will be exported as a pdf and loaded into DivData

22. WDYT - Department Staff Role • Confirm courses are loaded and setup properly • Add department specific questions to a survey • Setup the TA survey • Check response rates and track progress • Download standard end-of-term reports

23. WDYT - The Student Experience • Student receives one announcement • Expresses the value of student feedback • Provides a list of surveys to complete • Provides instructions for completing the form • Student receives one reminder every other day until their surveys are completed or the survey session closes

24. The Student Dashboard Body Text

25. Student Options in the Survey • Move forward and backward through pages of the survey • Decline the survey • Save the survey and continue it later

26. WDYT - TA Role • TAs are instructors of “Affiliated Courses” • Can login • See Response Rates • Send messages to students • See reports the day after grades are due • Download reports

27. WDYT - Instructor Role • Can login • See Response Rates • Send messages to students • See reports the day after grades are due • Download reports • W19 - Customization • End of term questions • Mid-quarter surveys

28. Gateway page - sets.ucsc.edu The logo is a live login link

29. Single sign-on from Canvas • Click the link in any site that says “SETS (was course evaluations)” Click this link

30. Click the WDYT launch button • After you click the “SETs” link you will see the link to WDYT. Click it.

31. Land at your WDYT dashboard

32. Dashboard - no prior terms The typical dashboard will look like Jody’s-

33. Dashboard - Upcoming surveys

34. Send a Reminder

35. Dashboard - Response Rates

36. Reports - Viewing

37. Reports - Downloading

38. Got a question? Need support? • Check with your department manager or staff who works with WDYT • Send an email to sets@ucsc.edu to generate a help request in ITR • Call 9-5506 - The Faculty Instructional Technology Center (FITC) help line • Drop by FITC - 1330 McHenry Libray

39. citl.ucsc.edu | citl@ucsc.edu | McHenry Library 1330A Crafting Custom Questions: During & End-of-Quarter Kimberly Adilia Helmer Chair of COT Teaching Professor of Writing Nic Brummell Member of COT Professor of Applied Mathematics

40. Why Write Custom Questions? • Reflective opportunity for teachers and learners • Midterm questions, electronic or otherwise, help improve teaching-learning in progress; prime Ss for end-of-quarter SETs • End-of-term questions provide pointed feedback for future course modification • Confidential: Instructor driven

41. Metacognition, Self-Efficacy, & Reflection • Metacognition: “Thinking about thinking” that is, the process of checking understanding and choosing appropriate strategies for improving learning. • Leads to better learning & transfer • Focuses on active learning • Promotes self-efficacy: Believing in one’s capacity to use effective metacognitive strategies

42. Metacognition, Self-Efficacy, & Reflection • Self-Efficacy: Reflects beliefs about one’s capabilities to complete a specific task successfully, that is, “self-perceptions of competence” • Leads to greater student effort, tenacity, & achievement • Developed through: • Positive feedback & encouragement • Metacognitive strategy development

43. Writing Questions • Questions should: • Focus on learning • Focus on the student NOT the teacher (to avoid bias & comments on grades) • Be precise & simple (no jargon, i.e., avoid: “outcomes,” “metacognition,” but not too colloquial) • Connect to specific skills & knowledge • Encourage reflection

44. Writing Questions: Open-Ended • Returns more information, but time consuming to read & analyze • Appropriate for deeper reflection; a tool for developing metacognition

45. Metacognitive Examples: A. What learning experiences or tools help you learn how to apply, solve, or explain X? B. What strategies have you used to learn how to apply, solve, or explain X? C. How did your writing change after group-writing conferences? D. How did your problem-solving skills change after you have done X? Writing Questions: Open-Ended

46. Writing Questions: Scaled (1-5) or Open To measure student learning in terms of their Self-Efficacy in relation to specific tasks in the near future: A. How confident do you feel about doing well on the upcoming midterm? 1-Not at all confident 2-A little confident 4-Somewhat confident 4-Confident 5-Very confident B. Please explain how you may use X (knowledge, skill, strategy) in your next (Writing, Physics, Math, etc.) course.

47. Writing Questions: Scaled (1-5) To evaluate specific learning tools or activities’ impact on learning of specific skills/topics: A. How useful in YOUR learning of X were outside study groups? 1-Can’t evaluate 2-Not useful 3-Somewhat useful 4-Useful 5-Very useful B. How helpful for YOUR learning of X were iClickers? 1-Can’t evaluate 2-Not helpful 3-Somewhat helpful 4-Helpful 5-Very helpful

48. citl.ucsc.edu | citl@ucsc.edu | McHenry Library 1330A Customization & WDYT: Steps

49. Step 1: Get into questions for a course Step 1(a)Click on “My Questions” in your login dashboard Step 1(b): Choose term and course Click on “Show Questions”

50. Step 2: You will see questions for that course ...