Retrospective and predictive evidence enhancing PISA assessments

1. Retrospective and predictive evidence enhancing PISA assessments Satya Brink, Ph.D. Learning Policy Directorate Human Resources and Skills Development Canada Government of Canada OECD Japan seminar Raising the Quality of Educational Performance at School Tokyo, June 23-24, 2005 Learning Policy Directorate, HRSDC, Canada

2. Retrospective and predictive evidence enhancing PISA assessments • PISA assessments measure performance due to cumulative years of schooling • PISA performance can also predict future performance because it measures competence rather than curriculum knowledge. Canadian students performed well in PISA assessments, but there were some research results on early and later performance that were unexpected. Learning Policy Directorate, HRSDC, Canada

3. NLSCY - Grade 6 Inter-Provincial Differences in Mathematics Scores NF NS PE NB QU ON MN SK AB BC NLSCY - Grade 2 NLSCY - Grade 4 NLSCY - Grade 6 TIMSS - Grade 7 TIMSS - Grade 8 SAIP93 - Age 13 SAIP97 - age 13 SAIP93 - Age 16 SAIP97 - Age 16 IALS - Youth Aged 16 to 25 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 Years of Schooling Learning Policy Directorate, HRSDC, Canada Willms, Vulnerable Children, 2002

4. Early differences in reading outcomes persist through schooling Mean Reading Score by Readiness Group (Assessed in Kindergarten) Canadian Test of Basic Skills 80 70 60 50 40 Low reading Average reading Variable reading 30 High reading 20 10 0 Grade 10--Reading Comprehension Grade 6--Reading Grade 8--Reading Learning Policy Directorate, HRSDC, Canada Elkin, Class of 79, 2002

5. Policy questions • What problems affect learning outcomes? • Universal or targeted? Mix? • How do we target? • How durable are the results? Investment in a national system of surveys with data on learning • Measured outcomes within surveys • Longitudinal surveys • Life course coverage • Trace trajectories (outcomes) and chains of risk (negative determinants) and protective factors (positive determinants Learning Policy Directorate, HRSDC, Canada

6. Vulnerability was defined as the occurrence of low measured current outcomes that indicate a higher chance of negative outcomes later in life. Children were vulnerable if they had at least one learning or behaviour problem that they could not overcome without an intervention. • In 1998-1999, 24.4 % of children had at least one learning or behaviour problem. • There were approximately 1.1 million vulnerable children between the ages 0 to 11 nationwide in 1998-1999. • In every cycle, close to 75 % of Canadian children aged 0-11 had no identifiable behaviour or learning problems. Source : NLSCY Learning Policy Directorate, HRSDC, Canada

7. Behavioural problems were more prevalent than learning problems • 18% of Canadian children have a behavioural problem • 11.8% of Canadian children have a learning problem • Only 2.9% of children have both a learning and a behavioural problem. • Because of the higher prevalence of behavioural problems among boys, 30.8% were vulnerable compared to 24.4% of girls. Percentage of Children in Canada Cycle 2 NLSCY Learning Policy Directorate, HRSDC, Canada

8. Vocabulary performance of 4 to 5 year olds Source: National Longitudinal Survey of Children and Youth Learning Policy Directorate, HRSDC, Canada

9. Why some policies have not worked. Targeting by income can result in mis-targeting Children with high outcomes who don’t need help but benefit when policies target by low income Children with low outcomes who need help but do not benefit when policies target by low income Adapted. Willms, vulnerable children, 2002 Learning Policy Directorate, HRSDC, Canada

10. Outcomes of younger children were impacted more by home language than older children Source: National Longitudinal Survey of Children and Youth Learning Policy Directorate, HRSDC, Canada

11. Differences in math scores of children with disabilities Math scores of children by disability or chronic illness Source: NLSCY 1998-99 Learning Policy Directorate, HRSDC, Canada

12. Grade repetition among aboriginal and non-aboriginal students during school years Learning Policy Directorate, HRSDC, Canada

13. Vulnerability is not a permanent state for most children 1994 1996 56.2% 14.9% 71.1% 71.9% 15.7% Vulnerable 28.1% 28.9% 13.2% Resilient Positive Development Long term Vulnerable Newly Vulnerable Source: NLSCY Learning Policy Directorate, HRSDC, Canada

14. Trajectories suggest Universal preventive approaches and early intervention 1994-95 1996-97 1998-99 13,5% (~0-2 years old) 10,6% (~3-4 years old) 14,2% (~5-6 years old) 26% 103 Low Score 49% 50 1.7% Low Score Not Low Score 51% 53 1.8% 14% 400 Low Score 15% 44 1.5% Low Score Not Low Score Not Low Score 85% 8.6% 253 74% 297 86% 2557 8% 209 Low Score 40% 83 2.8% Not Low Score Low Score Not Low Score 60% 126 4.3% 8.2% 243 Percent (%) Low Score 10% Not Low Score Number of obs. Not Low Score 90% 2105 71.2% 92% 2348 Total Percent (%) Source: NLSCY, all cycles, MSD and PPVT Scores Learning Policy Directorate, HRSDC, Canada

15. Learning and behaviour problems of graduates and dropouts Learning Policy Directorate, HRSDC, Canada

16. High School 75.8% GRADUATES 70% Post-Secondary Education DROPOUTS* 11.4% 8.5% GRADUATES LEAVERS CONTINUERS CONTINUERS 12.8% 6.9% 9.2% 83.9% 100% 100% N/A 91.5% 30% % No Longer in School N/A Labour Market % Employed Full-time 10% 59.3% 51.1% 63.1% 58.2% 11.4% % Employed Part-time 53.2% 18.2% 34.3% 22.2% 22.8% 34.8% % Not Working 36.7% 22.5% 14.6% 14.6% 19.0% 34.8% * This figure presents estimates for 18- 20-year-olds, including the high school dropout rate (11.4%). This rate differs from the 20-year-old dropout rate reported in Chapter 2 (12%). The latter is preferable as it accounts for the fact that some youth complete high school at a later age. Yet, for the purpose of presenting reliable estimates of education and labour market pathways, it is necessary to present 18-20-year-old results here. Source: At a Crossroads, HRDC and Statistics Canada, 2002 Start of the pathways of 18-20-year-olds in December 1999 Learning Policy Directorate, HRSDC, Canada

17. Pathways of youth : December 1999 and December 2001 Working part-time - 8% In high school – 3% In PSE – 52% Working full-time – 28% In school – 55% Not in school and working – 36% Not in school and not working – 9% 7% 23% 6% 3% 3% 3% 35% 15% 5% Not in school and not working – 14% Not in school and working – 41% In school – 44% In high school – 1% In PSE – 43% Working full-time – 34% Working part-time - 7% Age 20 – December 1999 Age 22 – December 2001 Source: Education and labour market pathways of young Canadians between age 20 and 22: An overview, Statistics Canada and HRSDC, 2004 Learning Policy Directorate, HRSDC, Canada

18. There is an inverse relationship between age and literacy performance, even after controlling for educational attainment Relationship between age and literacy scores on the document literacy scale, with adjustment for level education, 2003 Norway Canada Source: ALL, 2003 Learning Policy Directorate, HRSDC, Canada

19. Some policy directions based on longitudinal evidence • Importance of early childhood learning • Mix of preventive, universal investments and targeted interventions • Behaviour problems may require interventions outside school • Quality of initial education ensures that full benefit is drawn from further learning • Policies for second chance and life long learning Learning Policy Directorate, HRSDC, Canada