290 likes | 936 Views
Writing assessment descriptors. Ensuring validity and reliability in assessment Olwyn Alexander February 2015. Outline. Principles of test/assessment usefulness Focus on validity: assessment construct Focus on reliability: assessment criteria. P rinciples of test usefulness. Validity
E N D
Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn Alexander February 2015
Outline • Principles of test/assessment usefulness • Focus on validity: assessment construct • Focus on reliability: assessment criteria
Principles of test usefulness • Validity • Reliability • Authenticity • Interactiveness • Impact/Washback • Practicality Bachman & Palmer,1996
Principles of test usefulness • Validity – the test actually measures the performance it claims to measure and this is appropriate for the test purpose • Reliability – measurement is consistent between different assessors and different test takers • Authenticity – the tasks in the test are representative of tasks in the target situation Bachman & Palmer,1996, Alexander et al., 2008
Principles of test usefulness • Interactiveness – the tasks in the test engage the test takers’ communicative language ability, background knowledge and strategic competence • Impact – the test will affect classroom teaching and learning (and wider institutional policy – entry levels) • Practicality – resources will be required to develop the test and train assessors to administer it Bachman & Palmer,1996, Alexander et al., 2008
Test specification • Who individual/one class/all classes? • Why achievement/proficiency? • What construct/knowledge/skills? • How format/rating scales? Bachman & Palmer,1996
Test specification • Who students in10-week pre-sessional classes at or near CEFR B2 • Why to measure achievement of targetperformance in subject-specific context • What construct/knowledge/skills • How 2,000 word literature reviewexploring asubject-specific research question Bachman & Palmer,1996
Test specification – construct • What is a literature review? • Audience/Purpose/Structure • Level of target discipline knowledge • Level of critical engagement with sources Bachman & Palmer,1996
Literature review construct • Involves more than linguistic knowledge and skills • Requires students to engage with texts in the disciplines & struggle to understand them • Requires evaluation and management of information • Raising awareness of graduate attributes • Target performance described at SQA level 10: exit level of UG and entry level of PG degrees
Graduate attributes • a sense of ‘research-mindedness’ enabling a wider more analytical perspective on individual practice • the ability to identify problems, formulate research questions & interpret complex data to seek answers • the ability to derive meaning from complexity and make informed judgments on the basis of evidence • an openness to learning and positive orientation to new opportunities, ideas and ways of thinking • a tolerance for ambiguity and unfamiliarity.
SCQF level 10 knowledge • A critical understanding of the principal theories, concepts and principles in a subject/discipline. • Detailed knowledge and understanding in one or more specialisms some of which is informed by or at the forefront of a subject/discipline. • Knowledge and understanding of the ways in which the subject/discipline is developed, including a range of established techniques of enquiry or research methodologies.
SCQF level 10 application • Execute a defined project of research, development or investigation and identify and implement relevant outcomes. • Practise in a range of professional level contexts which include a degree of unpredictability and/or specialism.
SCQF level 10 cognitive skills • Critically identify, define, conceptualise, and analyse complex/professional level problems and issues. • Critically review and consolidate knowledge, skills, practices and thinking in a subject/discipline. • Make judgements where data/information is limited or comes from a range of sources.
Test specification – construct • What is a literature review? • Audience/Purpose • Structure overall & within sections/paragraphs • Level of target discipline knowledge • Level of critical engagement with sources • Task: Choose one of the areas above and brainstorm ideas for this aspect of the construct
Audience • peers, novices (non-specialists) in the community of practice • lecturers (specialists) who assess research projects, dissertations, theses • educated non-specialists, e.g. in funding councils, NGOs or government departments who award grants or use the findings
Purpose • in-depth exploration of a specific aspect of a research area • to indicate the current state of the art and suggest new research directions • OR to define and limit the scope of and provide a context and framework for a piece of research. • In-depth = reference to wide range of sources • Context = current state of the art • Framework = theoretical basis for research design
Structure • Thematic comparison and classification of sources, based on clear criteria usually specified in advance. • Relates key papers & ideas to each other, e.g. shows which make similar/opposite/more developed claims; which ideas stimulated by which earlier contributions • Paragraphs develop from general to specific with claims supported by evidence from relevant sources. • Review (& sections/paragraphs) structured from familiar to new; structure made explicit for the reader by summarising at the end of each section what has been discussed and how this links to what follows.
Subject knowledge • Responsibility of student (or subject expert if joint marking opportunity). • Includes recognition of key figures/papers which have moved the field on and therefore should be cited. • EAP tutors do not have this knowledge • Nevertheless have to be able to read a text they do not fully understand to assess whether it achieves its purpose for the stated audience through its structure and the level of critical engagement it demonstrates.
Critical engagement • Not simply description/summarising • (SCQF, 2012) identify, define, conceptualise, & analyse complex/ professional level problems & issues. • (Bruce, 2014) evaluative judgement made within any field of human activity about some aspect, object or behaviour of that field. • (Dodd, 2014) the different usefulness of knowledge…, appreciating how knowledge in its various forms holds potentially different value for different people in different places at different times.
Critical engagement • Not simply description/summarising • (Argent, 2013) showing awareness of different perspectives/stances (Argent, 2014) arguing cogently why some work makes a more relevant, useful or powerful contribution to the field generally or in the context of specific research. • (Spencer & Alexander, 2014) relates not to finding fault (criticising) but to criteria; evaluation must be in terms of /based on specific criteria, stated explicitly.
Test specification • How 2,000 word literature reviewwithsubject-specific research question Submitted to Turnitin Assessed using customised holistic assessment descriptors Bachman & Palmer,1996
Assessment descriptors Analytic (e.g. IELTS) or holistic (e.g. TOEFL) Holistic – quicker/easier to give an impression mark • A response at this level largely achieves all of the following • A response at this level is marked by one or more of the following • A response at this level may reveal one or more of the following weaknesses
Assessment descriptors • Task: develop a set of statements which capture the aspects outlined in the construct • A response at this level largely achieves all of the following • Content & task achievement (audience/purpose) • Structure overall & use of sources & stance • Paragraph structure & use of language
Assessment descriptors • Task: develop sets of statements which capture varying levels of achievement • A response at this level largely achieves all of the following • A response at this level is marked by one or more of the following • A response at this level may reveal one or more of the following weaknesses
Assessment descriptors • Example: task achievement and content • Fullyaddresses the specific area (concept, problem, method) to be explored, showing achievements/gaps. • Addresses the specific area in sufficient depth to cover the main points, showing someachievements/gaps • The main points are covered but with some redundant ideas. Achievements/gaps are not shown clearly. • Not all aspects of the area are covered and achievements/gaps in research are not shown.
Discussion Any questions & comments?