Professional Development Activity Log: A New Approach to Design, Measurement, Data Collection, and Analysis

1. Professional Development Activity Log: A New Approach to Design, Measurement, Data Collection, and Analysis AERA Annual Meeting San Diego April 13, 2004

2. Longitudinal Study to Measure Effects of MSP Professional Development on Improving Math and Science Instruction Math and Science Partnership

3. A collaborative study conducted by: Council of Chief State School Officers (CCSSO) American Institutes for Research (AIR) Wisconsin Center for Educational Research (WCER)

4. Authors • Kwang Suk Yoon, AIR • Reuben Jacobson, AIR • Mike Garet, AIR • Bea Birman, AIR • Meredith Ludwig, AIR

5. Research Questions • To what extent is the quality of the professional development supported by MSP activities consistent with research-based definitions of quality (e.g., content focus, active learning, coherence, collective participation, and sustained efforts) (Garet et al., 2001)? • What effects do teachers' professional development experiences have on instructional practices and content taught in math and science classes? Are high-quality professional development activities more likely than lower-quality activities to increase the alignment of instructional content with state standards and assessments?

6. Logic Model Teacher Characteristics: Background Variables, Prior PD Experiences Target Class Students: Diversity School Culture: Trust Implementation of Professional Development Content Focus Collective Participation; Active Learning; Coherence; Sustained Effort PDAL Pre-PD: Alignment of Instruction with Content Standards; Instructional Practice Post-PD: Alignment of Instruction with Content Standards; Instructional Practice Survey of Enacted Curriculum wave 1 Survey of Enacted Curriculum wave 2 Year 0 Year 1 Year 2 Year 3

7. Participants • Four MSP projects were selected for the study. In each project, we are collecting data with teachers mostly in middle schools or middle grades about their professional development in mathematics and science education. • N=472 teachers

8. Survey of Enacted Curriculum (SEC) • Instructional practice (e.g., instructional time in target class) • Content coverage and alignment: • Instructional time on topics and subtopics • Expectation for students (e.g., memorize facts, perform procedure, or solve non-routine problems) • Past experiences in professional development • Teacher characteristics (e.g., gender, teaching experience)

9. Why PDAL? • Gathers accurate, time-sensitive information; Minimizes recall problem with retrospective reports • Collects disaggregate information about specific PD activities – Reduces bias introduced by gross data aggregation • Generates context sensitive questions • Is able to tailor technical assistance to teachers based on their response patterns • Allows teachers to review their own logs – Teachers can reflect on their own PD experiences

10. Professional Development Activity Log (PDAL) • Teachers create an ongoing monthly log of any professional learning activity in which they participate • Longitudinal data collected over 15 months • Web-based, self-administered log • Aligned with SEC (e.g., content coverage) • Inclusive approach to professional development • Includes MSP-sponsored and non-MSP-sponsored activities • Documents one-time and recurring activities • Captures both formal and informal activities

11. PDAL Entries • Name of activity • Number of hours spent on each activity and its duration • Whether the activity is a one-time or continuous event (e.g., recurring over a number of months) • Type of activity (e.g., workshop, summer institute, study group) • Purpose of activity (e.g., strengthening subject matter knowledge) • PD quality features (e.g., active learning, coherence, collective participation) • Content focus (e.g., algebraic concepts: absolute values, use of variables, etc.) • Instructional practice – instructional topics covered in each activity (e.g., use of calculators, computers, or other educational technology)

13. Analysis of PDAL Data • Implementation analysis • Patterns of responses to monthly logs • Response rates; sample attrition; extent of missing data • Descriptive analysis • Patterns of teachers’ PD experiences • Correlates of high-quality PD activities • Latent classes of teachers based on their PD experiences • Impact analysis • Assess the impact of PD on math & science instruction

14. Table 1: PDAL Data Structure:Disaggregated log data: Teacher by activity by time

15. Table 2: PDAL Data Structure:Activity-level data aggregated across teachers

16. Implementation Analysis • # of Activities (over 9 months) • Range: 1 - 21 • Mean: 3.1 • # of Logs (over 9 months) • Range: 1 - 31 • Mean: 4.9 • Sessions • Per session: mean 12 min. • Per teacher over 9 months: mean of 50 min. 83% finished in one session; 13% finished in two sessions.

17. Implementation Analysis • Timing of log entries: • Weekends are least popular. Tues. - Thurs. are most popular • Spike at the beginning of each month • User understanding of the instrument • 95% of logs created were fully completed • Communication: • Info Packets • Phone calls, emails, letter reminders • Helpline

18. Table 3: Duration and Contact Hours of PD Activities

19. Table 4: PD Quality Construct Active Learning

20. Table 5: PD Quality Construct Coherence

21. Table 6: PD Quality Construct Collective Participation

22. Conclusion:Revisiting the Logic Model Teacher Characteristics: Background Variables, Prior PD Experiences Target Class Students: Diversity School Culture: Trust Implementation of Professional Development Content Focus Collective Participation; Active Learning; Coherence; Sustained Effort PDAL Pre-PD: Alignment of Instruction with Content Standards; Instructional Practice Post-PD: Alignment of Instruction with Content Standards; Instructional Practice Survey of Enacted Curriculum wave 1 Survey of Enacted Curriculum wave 2 Year 0 Year 1 Year 2 Year 3

23. Contact Information Kwang Suk Yoon (202) 403-5358 ksyoon@air.org Reuben Jacobson (202) 403-6925 rjacobson@air.org Visit us in the future www.pdal.net