Course Evaluation

1. Course Evaluation Amitava ‘Babi’ Mitra, MIT Maneesha Aggarwal & Robert Cartolano, Columbia University William Plymale and Aaron Zeckoski, Virginia Tech www.sakaiproject.org

2. Introduction Review Common Ground Features at Columbia and Virginia Tech Next Steps Q & A Agenda www.sakaiproject.org

3. How did this begin ? Late Feb 05 began exploring who was using/planning to use online course eval within SEPP March 05, SEPP Course Eval Working Group set up What’s the WG been doing since then ? Decided to use Columbia's existing feature set as a baseline to start with CU and VT prepared a detailed functional layout and work flow,  and posted it in April so that we can clearly show how the existing systems might operate. MIT, e.g., has studied this and responded with their own needs CU and VT have made sandboxes available Why are we doing this ? Getting to the common ground - the baseline that most of us can agree on Introduction www.sakaiproject.org

4. Administration of survey to students across one or more courses May span course, department, school, university Hierarchy, permissions, access levels are important Provide results to University, School, Department, Faculty, Students to evaluate: Instructors Curriculum Academic Program (Program Evaluation) Department, School, University (accreditation) What is Course Evaluation? www.sakaiproject.org

5. Evaluation has historical aspects at each school There are different motivations for evaluation System must be flexible enough to deal with different needs, but… There are common features that we have identified among our working group members. Course Evaluation www.sakaiproject.org

6. Anonymity - student privacy must be preserved Evaluation administrator - someone with appropriate authority must create and administer evaluation. Course Selection - must be very flexible As little as 1 course… Or the entire department, program, school, etc. Common Features --- from the WG’s discussions www.sakaiproject.org

7. Ability to combine core and specific questions Course-specific, instructor-specific, etc. Requires extensive reporting capabilities Administrator, Instructors, Students, Public accessible Requires workflow and automation to make it easy, reduce burden to administrators. More Common Features --- from the WG’s discussions www.sakaiproject.org

8. In production since Fall 2003 Built into CourseWorks, campus CMS Flexible approach needed to support multiple schools, multiple departments, etc. Carrot vs. Stick Open vs. Closed reports Many Reporting types; built-in reports and export data Columbia - Current System www.sakaiproject.org

9. E-mail tool very important; both auto and manual Aggregate data across semesters to support accreditation review Very popular, well-received by faculty, students, and administrators 87,825 evaluations administered to date 54,480 evaluations completed 157 evaluation templates created CU Current System (cont’d.) www.sakaiproject.org

10. Columbia Live Demo www.sakaiproject.org

11. --- Current System www.sakaiproject.org

12. Implemented - 1981 Different forms for different colleges Business, Architecture, Engineering Statistics 500 class sections per term 230,000 sheets per year Response Rate: 78% during past 3 years --- Current System www.sakaiproject.org

13. Columbia University Partnership SEPP Conference – Summer, 2004 – Denver Robert Cartolano / Maneesha Aggarwal Virginia Tech Development Aaron Zeckoski Kapil Ahuja / Justin Gawrilow SEPP Course Eval Discussion Group VT / CU Evaluation System Summary VT Evaluation System – “sandbox” release https://courseeval-dev.cc.vt.edu SEPP Contribution – VT / CU www.sakaiproject.org

14. Virginia Tech extensions Administrative hierarchy (Super, University, School, Dept, Instructor) Dynamic reporting (flexible formatting, trend analysis, sub-setting) Ability to incorporate existing data collected with VT’s current evaluation system. Oracle database tuning SEPP Contribution – VT / CU www.sakaiproject.org

15. VT Pilot release – May 2005 Columbia System with VT extensions Very limited response to VT pilot study due to late release. VT pilot study will resume Fall 2005 --- Pilot System www.sakaiproject.org

16. Faculty training will be incorporated into FDI Change to online evaluation system needs approval by Deans and Department Heads Online system will provide better access for Institutional Research and the administration Online system offers faculty flexibility in adding questions to their evaluations --- Pilot System www.sakaiproject.org

17. --- Demo www.sakaiproject.org

18. Institutional subject evaluation evaluates between 650-700 subjects each term, about 2/3 of which are undergraduate subjects and 1/3 graduate. This is out of approximately 1700 subjects with students enrolled and 3000 total subjects offered for credit each term. Evaluations are end-of-term only, on 2 paper forms separately for Science/Engineering subjects and for Humanities, Arts, and Social Science subjects, as chosen by the department. Quantitative data are captured and analyzed, written comments are available to each department from originals. Three departments, Electrical Engineering and Computer Science, Management, and Aeronautics and Astronautics (online) have their own forms and systems for subject evaluations. --- Current scenario www.sakaiproject.org

19. Must haves for a new system (to replicate current system): • Stand alone system (with the capability of being tied to the Course Management System) • Anonymity, confidentiality, security • Overall institutional administrator, plus levels of school/departmental administrators • At least 2 different forms (1 for Science/Engineering subjects and 1 for Humanities, Arts, and Social Sciences subjects) with preference for many • Some questions consistent across all schools with flexibility within school/dept./subject. Flexible questions would need to be administered at the local level but available to all. • Rating both subject and instructors on the same form • Access to individualized reports for each person evaluated • Assigning roles to instructors (lecturer, recitation instructor, etc.) • Integration with other MIT systems for data downloads (student, subject, and instructor information) and appropriate access (e.g., only to students registered in subject) • A 7 point rating scale, with flexibility to change that if needed • Blocking access to data/reports until after grades are turned in • Analysis and reporting at a centralized point • Student access to appropriate report information through a centralized site Needs --- Must haves www.sakaiproject.org

20. Necessary in any new system: Searching data/reports on instructor name • Assigning rank/status to instructors (prof./assoc. prof./assist. prof./lecturer/grad. student, etc.) • Individual and other reports that include overall subject, department, school, and institutional data • Longitudinal data/reports • Analysis and reporting at local points in addition to centralized one, with capability to print subject/department/school level with 1 click • Linking to evaluation results by department and by subject • Linking to evaluation results from subject listings/selection (i.e., our on-line catalog) * Integration with course management system with necessary security • Having students go to a single site for all their subjects to be evaluated Needs --- Necessary www.sakaiproject.org

21. Nice to have capability for: Evaluating sections of a subject (with the capacity for students to enter instructor's name) • Instructors' photos together with their name • Mid-term evaluations • Evaluations that only faculty would have access to • Capturing, editing, and distributing students' comments • Having a single email sent to a student listing all subjects to be evaluated by that student Needs --- Nice to have www.sakaiproject.org

22. Create a functional specification for ‘Course Eval ver 1.0’ built upon Columbia base and Virginia Tech extensions Develop Java-based version based on Sakai framework Run as standalone or within Sakai Consider using SAMigo as assessment engine, but probably not in version Course Eval ver 1.0 What’s Next www.sakaiproject.org

23. Questions? See Demonstrations at 5:30 today Join Evaluation Working Group Contacts: Robert Cartolano rtc@columbia.edu Amitava ‘Babi’ Mitra babi@mit.edu Open Discussion www.sakaiproject.org