Jeff Evans j.evans@mdx.ac.uk Radstats, York, 23 Feb. 2013

1. The PIAAC International Survey of Adults’ Skills: Policy Context, Aims, Methods, and Opportunities Jeff Evans j.evans@mdx.ac.uk Radstats, York, 23 Feb. 2013

2. Aims of presentation To describe the PIAAC international survey: its aims, definition of ‘numeracy’, its methods, and their ‘validity’ To describe features of the relevant educational policy context, esp. the turn to ‘governance by numbers’ To sketch some possible developments related to statistics and social and educational research

3. PIAAC - Project for the International Assessment of Adult Competencies • previous international surveys : IALS (1994-98) and ALL (2004-06), but few countries • PIAAC: commissioned by OECD • Domains: Literacy, Numeracy, and Problem solving in technology rich environments • design and planning overseen by subject-matter ‘expert groups’ and by international consortia of consultants • fieldwork done by national teams, in 24 countries • first run in 2011-12, full results available in Oct. 2013

4. PIAAC Aims To “measure the skills & competencies needed for individuals to participate in society and for economies to prosper”: ‘inclusive liberalism’ • Comparative: between [groups of] individuals and across countries in key “competencies” (Literacy, Numeracy, Problem Solving) • Correlational: with individual outcomes, e.g. further learning / earnings; & with aggregate outcomes, e.g. economic growth, social equity in labour market • Evaluative: potentially, of e.g. performance of education / training systems (Schleicher, 2008)

5. PIAAC concepts and measures Numeracy: the ability to access, use, interpret, communicate mathematical information & ideas, to engage in / manage mathematical demands of a range of situations in adult life. (PIAAC Numeracy Expert Group, 2009) Also: Background Questionnaire: attitudes to, and experience with, numeracy (literacy, etc.) in various adult practices Job-Related Assessment: use of skills in workplace

6. PIAAC survey design (1) • adults 16-65: 5000+ sampled per country; various incentives to participate • unlike PISA (& TIMSS): no ‘captive population’ in schools  combines household survey methodology with educational testing • default method of administration is via PC • ‘adaptive testing’ to find appropriate ‘level’ of respondent, to guide selection of further test items  Issues of survey validity & reliability, esp. for international comparisons of performance

7. PIAAC survey design (2) Construct validity: extent to which measure represents all facets of concept and in a theoretically ‘appropriate’ way Reliability of testing across countries & across interviewers Ecological validity: whether setting of research is ‘representative’ – i.e. whether tasks on-screen representative of tasks to which one wishes to generalise External validity: overall representativeness of findings for population sampled and for tasks / skills of interest … Similar dilemmas for most educational assessment. Possible approach: Radical Statistics Education Group, Reading between the Numbers: a critical guide to educational research (1982), 1stedn. [needs revision – see Epilogue, below] http://www.radstats.org.uk/books/ReadingBetweenNumbers.pdf … BUT also need to discuss how results will likely be interpreted and used, by politicians, producers, media (as e.g. with PISA?)

8. Educational policy context Competitive global economic context  countries / regions like EU must display appropriate ‘levels’ of skill and productivity … educational values increasingly interpreted through neo-liberal imperatives (Rizvi & Lingard, 2010) Lifelong learning (LLL) aims to promote development of knowledge, & to enable citizens’ active participation in all spheres of social & economic life BUT holds individuals responsible for their own education • stress on the need to acquire and update range of abilities, attitudes, knowledge and qualifications over the life-course – and to demonstrate having done this

9. Role of international organisations OECD & the EU are disseminating ideas & practices that strongly influence nat’l policy making worldwide (Dale & Robertson, 2009) • Authoritative definition – and deployment – of concepts like adult numeracy (cf. Cockcroft, 1982; Crowther, 1959; etc., etc.) • Promotion of “expertise in creating comparable datasets … countries can measure relative success of education systems & shift policy orientations accordingly” (Grek, 2010) • … and ’governing by data’ (Lingard, 2011)

10. PIAAC survey design (3): further issues Validity also  how survey’s measure scores interpreted or reconceptualised in reports of the sponsors, or of the media … • Can an adult’s performance score be related to putative ‘level of numeracy’ in a one-dimensional sense? Many adults have “spiky profiles”, which are different due to distinctive life experiences: some find type A items (“data & chance”) more difficult; others find type B items (“dimension & shape”). • Is it legitimate for, say, pressure groups to try to stipulate a ‘minimum level of numeracy needed to cope with the demands of adult life’, for adults within their countries – based on scores from an international comparative study – and then to count how many adults have a relevant ‘deficit’?

11. Epilogue: relation to statistics & social research • Big Data ? … PIAAC data at least ‘moderately big’ • Open Data … OECD undertakes data deposited on website by October 2013 launch; data for other surveys (PISA) already there • Clearly space for groups to investigate issues not investigated by OECD and national partners/ in a different way … a role for RS? * • Raises issues for concerns explored in Rad Stats Education Group (1982, see above) … However, the context has changed radically: for one thing, educational policy is much more globalised, responsive to ideologies of economic competition – and key studies are international and comparative, including this one * * Conference participants or others interested in working on either of these themes are invited to contact the speaker: Jeff Evans j.evans@mdx.ac.uk

12. References Cockcroft Report (1982). Mathematics Counts: Report of the Committee of Enquiry into the Teaching of Mathematics under the Chairmanship of Dr. W H Cockcroft. London: HMSO. Dale R. & Robertson S. (2009) Globalisation & Europeanisation in Education. U.K.: Symposium Books. Grek, S. (2010) International Organisations and the shared construction of policy ‘problems’: problematisation and change in education governance in Europe. European Educational Research Journal, 9(3), 396-406. Lingard, B. (2011). Policy as numbers: Ac/counting for educational research. Australian Educational Researcher, 38, 4, 355-382. Online: http://www.springerlink.com/content/u515384526754608/fulltext.pdf OECD (2012). Literacy, Numeracy and Problem Solving in Technology-Rich Environments: Framework for the OECD Survey of Adult Skills. Paris: OECD Publishing. Online: http://dx.doi.org/10.1787/9789264128859-en Rizvi, F. & Lingard, B. (2010) Globalizing education policy. London: Routledge. Schleicher, A. (2008). PIAAC: A New Strategy for Assessing Adult Competencies. International Review of Education, 54(5-6), 627-650. Online: http://www.oecd.org/dataoecd/48/5/41529787.pdf