Presented by Kathleen Norris Plymouth State University October 8, 2010

1. Evaluating a New Doctoral ProgramUsing Formative AssessmentNational Evaluation InstituteConsortium for Research on Educational Accountability and Teacher Evaluation Presented by Kathleen Norris Plymouth State University October 8, 2010

2. Purposes of the Evaluation • Engage the doctoral program students in assessment of their program • Gather information from the students about their experiences with all aspects of the program early in their studies • Use information from the students to make adjustments in the program as it unfolds • Collect rich information early in the program and continue over time to discover patterns or trends • Use information to adjust the survey instrument

3. Audiences for the Evaluation • Plymouth State University (PSU) Doctoral Program Advisory Board (includes the program faculty) • New England Association of Schools and Colleges (NEASC) • PSU College of Graduate Studies Staff • PSU Administration

4. PSU EdD Advisory Board • The Advisory Board includes: • Doctoral Program Faculty (all) • College of Graduate Studies Administrative Team • PSU, Granite State College and UNH Admin/Staff • Dean of the PSU Library and Student Services • Doctoral Students (one from each cohort) • Representatives of K-12 School Districts • Representative from the NH Department of Ed • Representatives from the NH School Administrators • PSU Certificate of Advanced Graduate Studies Alumni

5. A Brief History • The idea for a doctoral program at PSU had its first implementation through doctoral program partnerships with the CAGS program that began in 1997. The first partner was The University of Sarasota, in Florida (now Argosy University/Sarasota). Four members of the first CAGS cohort received their EdD degrees from the University of Sarasota by 2001 and many more have completed through Argosy.

6. A Brief History • When Franklin Pierce University developed their Doctor of Arts program, Plymouth State collaborated with them and developed a partnership that, like Argosy’s, included an articulation with PSU’s 18 credit CAGS core. Several PSU graduates have continued in their doctoral work at FPU and some have already graduated.

7. A Brief History • With the College of Graduate Studies firmly established at Plymouth State University, and these partnerships demonstrating the success of our CAGS graduates in doctoral programs, a natural next step was the development of a Doctor of Education degree program at PSU. The process of development and approval, from within the university through all levels outside, including the NH legislature, began in earnest in 2006 and was completed in 2008. The EdD program began at PSU in 2009.

8. Where We Are Now • Two cohort models have been established. • The Year-Round model requires that students have completed all 27 of their post-master’s specialization credits before beginning the eight doctoral core courses, which are offered through a summer, fall, winter, spring and following summer. • The Summers model also requires 60 credits, but the doctoral core courses are offered during 4 summers, with time during the intervening academic years for students to complete their specialization courses.

9. Doctoral Core Courses • There are eight core courses in the doctoral program, and these eight courses are taken with the cohort. Students who enter the program are committing to their own programs of study, but also to being a member of a cohort learning community through their core courses as well as the dissertation. • EP 8000 Emerging Perspectives on Learning and Development; • EP 8010 Program Evaluation: Theory and Practice; • EP 8020 Ethical Leadership and Advocacy; • EP 8030 Leadership in a Diverse World; • EP 8040 Resource Planning and Stewardship; • EP 8050 Vision: Synergy and Synthesis; • EP 8060 Dissertation Seminar: Writing the Proposal; and • EP 8820 Externship

10. Cohort Model Schedules Year Round Cohort I (repeated beginning in odd year summers) • Summer 2009 • EP 8000, EP 8010 • Fall 2009 • EP 8020 • Winter 2010 • EP 8040 • Spring 2010 • EP 8030, EP 8050 • Summer 2010 • EP 8060, EP 8820 • Fall 2010 • Comprehensive Examinations • Winter 2011 • EP 8070 Dissertation I • Spring 2011 • EP 8080 Dissertation II • Summer 2011 • EP 8090 Dissertation III • Dissertation Defense • Graduation Summer Cohort I (repeated beginning in even year summers) • Summer 2010 • EP 8000, EP 8010 • Fall 2010, Winter and Spring 2011 • Specialization Courses as required • Summer 2011 • EP 8020, EP 8030 • Fall 2011, Winter and Spring 2012 • Specialization Courses as required • Summer 2012 • EP 8040, EP 8050 • Fall 2012, Winter and Spring 2013 • Specialization Courses as required • Summer 2013 • EP 8060, EP 8820 • Fall 2013 • Comprehensive Examinations • Winter 2014 • EP 8070 Dissertation I • Spring 2014 • EP 8080 Dissertation II • Summer 2014 • EP 8090 Dissertation III • Dissertation Defense • Graduation

11. Numbers • Cohort I, Year Round • 25 Applicants • 12 Admitted • 11 Enrolled Members • All 11 Cohort Members qualified to sit for their comprehensive examinations this fall. • Cohort I, Summers • 34 Applicants • 16 Admitted • 14 Enrolled Members • All 14 Cohort Members have completed their first two courses and are following their individual programs of study for fall and winter.

12. Formative Evaluation • In addition to regular course evaluations and conversations during advising sessions, a formative evaluation was developed to solicit feedback from the doctoral students in these areas: • Program Information and Marketing • Admissions • Program Orientation • Doctoral Program Handbook • Program of Study • Briefcase and Enclosures • Course Scheduling • Course Evaluations • Independent Study • Program Advising • Cohort Model • College of Graduate Studies and EdD Program Hallmarks • Additional Comments, Suggestions

13. Formative Evaluation • The first year-round cohort was given this survey after their first four courses, and the first summer cohort was given this after their first two courses. Information gained from this evaluation has been used to update and adjust programming between the first and second cohorts and will be followed up with additional formative evaluations each winter and summer, as well as a summative evaluation of the first cohort’s experience through the comprehensive examinations but before the dissertation. After degree completion and in years following, other surveys will be provided.

14. Feedback from the First Cohorts • In the areas of Program Information and Marketing, all students in the year-round cohort learned about the program through personal contact though one student reported becoming interested in applying only after reading about the program in a newspaper. However, for the summer cohort, students reported finding the program online and through publications as well as by “word of mouth.” • As to the reasons for choosing PSU, these quotes represent the cohorts’ answers:

15. Student Voices • What were the main reasons you chose to apply to PSU’s EdD program? • From the Year Round Cohort: • “Because I had already completed my CAGS at PSU and knew that credits would be carried over. It would cost me less than starting a traditional program at another university. I didn’t want to drive to Boston and liked the more personal approach of PSU.” • “I liked what I had heard about the program being ‘student-centered’ with concrete timelines for coursework being completed, exam schedule and dissertation blocks.”

16. Summer Cohort Voices • “This has been a lifetime goal to achieve this degree and the option to pursue the EdD over consecutive summer sessions which would allow me to maintain my professional responsibilities.” • “Plymouth State’s emphasis on holistic thinking and applied practice, and the ties to state and local agencies which are making a difference in education in New Hampshire.” • “Location and the type of program. I like the idea of a cohort group.”

17. Program Information • Generally students in the first cohort reported a difference between what they had heard during their program orientation and what they found in the handbook and online and wished for clearer information on timelines. • The students of the first cohort did not have a full orientation day on campus. • Based on these comments, the handbook and the information on the Web site have been updated, admissions information has been adjusted, and a complete orientation day was held for the 2010 summer cohort. The summer cohort responded favorably to all of these updates.

18. Admissions • Both cohorts gave overall positive feedback on the iGrad online admissions process as well as the interview process. The online application process has been monitored and any technical issues have been resolved. • The cohorts gave mixed responses on whether or not PSU should go paperless with admissions materials, so we have continued to use a combination of email and paper communications. • Each cohort member is given a PSU briefcase with a thumb drive and some other enclosures. All of the cohort members expressed a positive response to this.

19. Course Scheduling • Year-round cohort members expressed the importance of knowing the schedule of upcoming courses for time and financial planning. They noted differences in pacing between the courses and while some students appreciated the intensity, others found it challenging to absorb the material without more time between them. • “I have often felt torn between getting ready for my next class and finishing up the last class. The pace is challenging at times but it is a doctoral program so I guess it should be!” • Based on this feedback, the full 4 year schedule for the summer cohort was provided with the admissions packet, and the schedule will be set with attention to the pacing and provided in the same way for the next year-round cohort. The summer cohort responded positively to the scheduling of their program.

20. Course Evaluation • In addition to the evaluations students provide at the end of each course, the formative evaluation asked them to consider the coursework in general. • Most students in the year-round cohort reported finding the courses to vary in their writing and research requirements and that they would like to see more connections made to their own dissertation ideas. Suggestions were made to include the assignment of an annotated bibliography related to the dissertation in each course, and to spend some time in each class discussing the dissertation process with students and these suggestions already have been incorporated into the summer cohort courses. • The year-round cohort found it challenging to find time for their own independent research and for reaching out to program faculty while working full time and completing work for the core courses. Anecdotal feedback indicates that they were highly communicative with each other and preferred to develop their cohort without faculty involvement. • Annotated bibliographies and dissertation discussions were incorporated into the summer courses. The summer cohort reported finding the expected amount of program rigor, reading, writing and research. They also mentioned getting involved in their own research and reaching out to the faculty for advising. The summer cohort has initiated more contact than the year-round.

21. Cohort Model • These comments are typical of the year-round cohort: • “I think we have gotten close as a cohort, although it seems some are more engaged than others, dependent upon their family life.” • “The cohort has been great and a nice part of the program. It has worked out better than I expected.” • “A great group of people, all very sharp and insightful, mostly well-chosen for this experience.”

22. Cohort Model • From the Year Round Cohort: • “I have found it supportive and interesting as we have so many folks from different fields. At first I wondered if it would help because there were few people from the [K-12] education perspective. I have grown to see the classes from their eyes and have appreciated their perspectives. I think we have a good supportive cohort and am glad it has worked out.” • “Love my cohort.I like the diversity and the ability of the group to say what they think. I have really enjoyed being a member of our cohort. I think we got very lucky with the group that we have.”

23. Cohort Model • From the Summers Cohort • “Loved it- a great model-really enjoyed the team atmosphere of the program-helps people get through the challenging parts of the work.” • “I was more than pleasantly surprised at the relationship that quickly developed between cohort members, without the usual cliques that tend to form. The experience is profound. My impression is one of being a part of a whole, and I am amazed by the balance of erudition and interpersonal ability that everyone in the cohort has. It is exhilarating to share and discuss ideas with 13 other people who have similar points of reference, who share the same values, and who experience “flow” when talking about ideas and actively making changes to the world around us.” • “Exciting, stimulating and so far a very positive learning experience. The admissions process must have worked hard to identify a group that would be committed to the cohort concept and working within a professional learning community. It was fabulous and extremely supportive. Everyone has so many things to offer professionally, personally and recreationally. This is going to be a wonderful group to be part of the next 4 years.”

24. Program HallmarksYear-Round Cohort (n=10) I recognize them in myself • Enhanced leadership capacity (10) • Fostering of inquiry (8) • Reflective practice (9) • Expansion of knowledge (8) • Global aspects of leadership (2) • Leadership and Advocacy (8) • Application of research (8) • Intellectual challenge (7) • Mastery of course content (8) • Current issues in leadership (7) • Preparing to be a system leader (5) • Current issues in community (7) • Professionalism and Service (6) • Refinement of skills (7) • Current issues in learning (7) • Reflection and Innovation (7) • Creative scholarship (4) • Preparing to be an agent of change (6) • Retrieval of scholarly materials (7) • Scholarship and Application (6) • Global Awareness and Social Responsibility (4) I recognize them in the program • Enhanced leadership capacity (8) • Fostering of inquiry (8) • Reflective practice (7) • Expansion of knowledge (7) • Global aspects of leadership (2) • Leadership and Advocacy (6) • Application of research (6) • Intellectual challenge (8) • Mastery of course content (6) • Current issues in leadership (7) • Preparing to be a system leader (7) • Current issues in community (5) • Professionalism and Service (5) • Refinement of skills (6) • Current issues in learning (7) • Reflection and Innovation (4) • Creative scholarship (1) • Preparing to be an agent of change (7) • Retrieval of scholarly materials (6) • Scholarship and Application 6) • Global Awareness and Social Responsibility (3)

25. Program HallmarksSummer Cohort (n=14) I recognize them in myself • Enhanced leadership capacity (13) • Fostering of inquiry (14) • Reflective practice (12) • Expansion of knowledge (13) • Global aspects of leadership (6) • Leadership and Advocacy (12) • Application of research (12) • Intellectual challenge (14) • Mastery of course content (8) • Current issues in leadership (10) • Preparing to be a system leader (10) • Current issues in community (10) • Professionalism and Service (13) • Refinement of skills (12) • Current issues in learning (12) • Reflection and Innovation (12) • Creative scholarship (10) • Preparing to be an agent of change (13) • Retrieval of scholarly materials (14) • Scholarship and Application (12) • Global Awareness and Social Responsibility (10) I recognize them in the program • Enhanced leadership capacity (13) • Fostering of inquiry (14) • Reflective practice (13) • Expansion of knowledge (13) • Global aspects of leadership (10) • Leadership and Advocacy (13) • Application of research (14) • Intellectual challenge (14) • Mastery of course content (14) • Current issues in leadership (12) • Preparing to be a system leader (13) • Current issues in community (12) • Professionalism and Service (14) • Refinement of skills (14) • Current issues in learning (12) • Reflection and Innovation (12) • Creative scholarship (11) • Preparing to be an agent of change (12) • Retrieval of scholarly materials (14) • Scholarship and Application (11) • Global Awareness and Social Responsibility (8)

26. Conclusions • Overall, students from both cohorts are highly positive about the program. • Over 200 inquiries have been logged, and more inquiries are coming from potential applicants who live outside of New Hampshire. The influence of cohort members who live outside of New Hampshire and our marketing efforts overseas will be seen in future application groups. • The improvements we made to the course descriptions, handbook, website, admissions materials and orientation based on comments from the first year-round cohort have been beneficial to the first summer cohort and we will look for these changes to benefit the upcoming year-round cohort. • The inclusion of a full orientation to the library, opportunities to network with the Pakistani Leadership Institute members, and attention to the dissertation in the first two courses for the summer cohort have provided opportunities for the students to recognize more of the program hallmarks. • Programming to emphasize Global Awareness and Global Leadership should be increased throughout the program so these hallmarks become more easily recognized by students in the program and in themselves. The inclusion of cohort members who have had significant international experience made a difference in the summer cohort and recruitment of international students will continue.

27. Challenges and Opportunities • Gathering data while the program is operating (and new) provides both the challenge of interpreting the data to make informed decisions and the opportunity to make adjustments as needed. • Gathering data about all aspects of the program and graduate studies literature, from the recruitment through taking courses, involves working with a much broader team than just with the faculty. • Working with faculty during and after their first offerings of a course can be a challenging opportunity. • For this new, and the first, doctoral program at the university, there are different levels of internal and external interest in various aspects of the program and high expectations at all levels.

28. Next Steps • The formative evaluations will continue with both of these first cohort groups and with those in the future so that trends can be identified and continual improvements made. • The doctoral faculty will work on the scope and sequence of the courses and the long term schedule, and continue to update their courses. • Processes throughout the program will be monitored to ensure that they are as smooth as possible for the students. • Doctoral faculty will meet after monthly Graduate Faculty Meetings for updates, information and program monitoring. • Faculty will monitor the dissertation process as we move into that phase with this first cohort beginning in winter of 2010-2011. • The recent NEASC Substantive Change Focused Visit report responses will be made as the institution heads toward the next comprehensive review.

29. Thank you for coming! Please contact me at any time with comments or questions and opportunities to collaborate on evaluations. knorris@plymouth.edu 603-535-3023