Developing Language Assessments and Justifying their Use

1. Developing Language Assessments and Justifying their Use Lyle F. Bachman Department of Applied Linguistics University of California, Los Angeles lfb@humnet.ucla.edu Presentation based on Bachman, L. F. & Palmer, A. S. (2010). Language assessment in practice: Developing language assessments and justifying their use the real world. Oxford: Oxford University Press.

2. When you need to assess your students, where do you begin? The type of assessment task I will use How I will maintain test security How I can help my students do well so that they will succeed after they finish my class How I can help my students do well so that they will make me look good All of the above.

3. Teachers’ questions about assessment Teachers almost always ask: When should I assess? How often should I assess? How should I assess? Teachers seldom ask: What should I assess? Teachers almost never ask: Why should I assess?

4. Topics in this presentation The purposes of teaching and assessment in the classroom Two modes of classroom assessment Deciding why we want to assess: identifying intended beneficial consequences Deciding why we want to assess: identifying decisions to be made Deciding what we want to assess: defining constructs Deciding when to assess Deciding how we want to assess: Designing assessment tasks: Relating classroom assessment to assessment purpose: 5 things to think about

5. Why do we teach? The primary purposes of language teaching are to (e.g.): promote or facilitate learning; enhance learners’ linguistic, cognitive, emotional, and social development

6. Why do we assess? The primary purpose of classroomassessment is: to gather information to help us make decisions that will lead to beneficialconsequences for stake holders (learners, teachers).

7. Beneficial Consequences Decision(s) Interpretation(s) about learner’s language ability Assessment Report Assessment Performance

9. Teaching & learning tasks, assessment tasks

10. Why do we assess? In the classroom, we use language assessments to inform two kinds ofdecisions: formative and summative: Formative decisions relate to making changes in teaching and learning activities in support of, or to promote or enhance learning. Formative decisions are made during the processes of teaching and learning.

11. Why do we assess? In the language classroom, we use assessments to inform two kinds ofdecisions: formative and summative: Summative decisions relate to passing or failing students on the basis of their progress or achievement, or certifying them based on their level of ability. Summative decisions are made after the processes of teaching and learning.

12. Why do we assess? Formative decisions: Teachersmake decisions about: changing their teaching (materials, activities). presenting, revising, contextualizing, and scaffolding new material; placing learners into appropriate groups or levels; guiding their students’ learning; challenging and motivating their students to learn.

13. Why do we assess? Formative decisions: Learnersmake decisions about making changes: in their approaches to or strategies of learning; in the particular areas on which they may need or want to place greater emphasis.

14. Why do we assess? Summative decisions: Teachers and administratorsmake decisions about: which students pass and fail a course. which students are certified at a particular level of ability

15. Two modes of classroom assessment: Implicit Explicit Modesof classroom assessment

16. Modesof classroom assessment Implicit mode: (“dynamic assessment”, “on-line assessment”, “continuous assessment”) Instantaneous and cyclical: assessment – decision – instruction; assessment – decision – instruction Learners are largely unaware that assessment is taking place. Used primarily for formative decisions.

17. Modesof classroom assessment Explicit mode: Assessment as “assessment” Separate activity from teaching Both teacher and learners know this activity is an assessment. Used for both formative and summative decisions.

19. When do we assess? What are our intended consequences? What decisions do we need to make? What information about learners’ language ability do we need to collect? How will we collect this information? Explicit modeof classroom assessment

20. When do we assess? Whenever we need to make an instructional decision, or a decision about learners, we need to assess.

21. When do we assess? Occasions for classroom assessment Warm-up, revision (self-assessment, implicit assessment) Presentation (implicit assessment) Guided practice (implicit assessment) Independent practice (self-assessment) “Assessment” (explicit assessment)

22. Types of decisions for which language assessments are used • Guiding teaching and learning • Entrance, readiness • Placement • Achievement/progress • Certification • Selection (e.g., employment, immigration)

23. Uses of language assessments • Many of these decisions are “high stakes”. • Need to ask: • What beneficial consequences do we want to bring about? • What decisions do we need to make to help promote the intended consequences? • What information about learners do we need to make the most appropriate decision? • How can we gather this information? • Teachers’ judgments? • Classroom assessments? • Self assessments? • Formal tests?

24. Intended Consequences ASSESSMENT DEVELOPMENT Decisions to be made Interpretation(s) about learners’ language ability Assessment Record Assessment Performance

25. Intended Consequences ASSESSMENT INTERPRETATION AND USE ASSESSMENT DEVELOPMENT Decisions to be made Interpretation(s) about learners’ language ability Assessment Record Assessment Performance

26. Uses of language assessments Your Turn: For your assessment project, answer these questions: What beneficial consequences do I/we want to bring about? What decisions do I/we need to make to help promote the intended consequences? What information about learners do I/we need to make the most appropriate decision? How can I/we gather this information?

27. Accountability • We must be able to justify the use we make of a language test. • That is, we need to be ready if we are held accountable for the use we make of a language test. • In other words, we need to be prepared to convince stakeholders that the intended uses of our test are justified.

28. Whom do we need to convince? All Stake holders: • Ourselves • Our fellow teachers • Test takers (our students) • Program/department/university administrators • Parents, guardians • Other stake-holders (e.g., potential employers, funding agencies)

29. Uses of language assessments Your Turn: For your assessment project, describe the stake holders.

30. How do we do this? • We need a conceptual framework that will enable us to justify the intended uses of our assessments. • An “Assessment Use Argument” (AUA) provides such a framework.

31. How do we do this? Two activities in justifying the uses of our assessments: • Develop an Assessment Use Argument (AUA) that the intended uses of our assessment are justified, and • Collect backing (evidence), or be prepared to collect backing in support of the AUA.

32. Assessment Use Argument Provides: • the rationale and justification for the decisions we make in designing and developing the test, and • the logical framework for linking our intended consequences and decisions to the test taker’s performance.

33. Parts of an Assessment Use Argument • Claims: statements about our intended interpretations and uses of test performance; claims have two parts: • An outcome • One or more qualities claimed for the outcome • Data: information on which the claim is based.

34. Parts of an assessment Use Argument • Warrants:statements justifying the claims • Rebuttals: statements about possible alternatives to the outcomes or to the qualities that are stated in the claims. • Backing:the evidence that we need to collect to support the claims and warrants in the AUA.

35. Consequences Equitable Values-sensitive Decisions Beneficial Meaningful Impartial Generalizable Relevant Sufficient Interpretations about test taker’s language ability Assessment Reports/Scores Consistent Assessment Performance

36. Articulating Claims for Intended Uses (Table 1 in the Handout)

37. Qualities of Claims in an AUA Claim 1 • Outcome: Consequences • Quality: Beneficence Articulate Claim 1: list and describe: • The intended consequences • The stakeholders

38. Qualities of Claims in an AUA Generic version of Claim 1: The consequences of using an assessment and of the decisions that are made are beneficial to stakeholders. {EXAMPLES OF CLAIM 1, pp. 2, 24}

39. Qualities of Claims in an AUA Your turn: Adapt Claim 1 to your project

40. Qualities of Claims in an AUA Claim 2 Outcome: Decisions Qualities: Values-sensitivity Equitability

41. Qualities of Claims in an AUA Generic version of Claim 2: The decisions that are made on the basis of the interpretation take into consideration existing educational and societal values and relevant legal requirements and are equitable for those stakeholders who are affected by the decisions. {EXAMPLES OF CLAIM 2, pp. 3, 27}

42. Qualities of Claims in an AUA Your turn: Adapt Claim 2 to your project

43. TAKE A BREAK!

44. What do we assess? Learning objectives “Content” of the syllabus or curriculum “Content” of lesson plans “Content” of teaching and learning materials “Content” of teaching & learning activities Language ability, proficiency

45. Qualities of Claims in an AUA Claim 3 Outcome: Interpretation Qualities: Meaningfulness Impartiality Generalizability Relevance Sufficiency

46. Qualities of Claims in an AUA Generic version of Claim 3: The interpretations about the ability to be assessed are: meaningful with respect to a particular learning syllabus, a needs analysis of the abilities needed to perform tasks in the TLU domain, or a general theory of language ability or any combination of these. impartial to all groups of test takers, generalizable to the TLU domain, relevant to the decision to be made, and sufficient for the decision to be made. {EXAMPLES OF CLAIM 3, pp. 5, 28}

47. Qualities of Claims in an AUA Meaningfulness warrants define the ability we want to assess, with respect to one of more frames of reference, and specify the conditions under which test takers’ performance will be elicited. Meaningfulness Warrant 1 provides a descriptive label and definition of the ability to be assessed.. Generic version of meaningfulness Warrant 1: The definition of the construct is based on a frame of reference such as teaching syllabus, a needs analysis, or current research and/or theory of language use, and clearly distinguishes the construct from other, related constructs {EXAMPLES OF MEANINGFULNESS WARRANT 1, pp. 5, 28}

48. Qualities of Claims in an AUA Meaningfulness Warrant 2 provides the conditions under which we will observe or elicit test takers’ performance. Generic version of meaningfulness Warrant 2: The assessment task specifications clearly specify the conditions under which we will observe or elicit performance from which we can make inferences about the construct we intend to assess. {EXAMPLES OF MEANINGFULNESS WARRANT 2, pp. 5, 28}

49. Qualities of Claims in an AUA Your turn: Adapt Claim 3 to your project. Adapt meaningfulness Warrant 1 to your project. Adapt meaningfulness Warrant 2 to your project. Create an example assessment task for your project.

50. How do we assess? Think about the following: Why we want to assess (decisions and consequences) What we want to assess (interpretations about learners’ language ability) The “target language use domains” to which we want the interpretations to generalize Language classroom School—other classes Home, community or workplace