When What You Have Is Not Enough

1. When What You HaveIs Not Enough NCOLCTL 24 April 2009 Ray Clifford

2. How Do You Understand theTitle of This Session? • In the LCTLs there is a shortage of…. • Expertise. • Time. • Textbooks. • Tests. • Patience. • Students. • Something else.

3. And the answer (from a test development perspective) is …

4. And the answer (from a test development perspective) is … there are often too few students to follow “normal” validation procedures for Reading and Listening proficiency tests.

5. But are you sure you want to use a proficiency test? • Proficiency testing is not always the right choice.

6. Testing is complicated – but it is important! • Language tests can motivate. • Language tests can demotivate.

7. Language Testing and Motivation • Appropriate tests can motivate learners to improve their skills. • Appropriate tests can motivate teachers to refine their teaching to match their students’ needs. • Inappropriate tests can de-motivate both students and teachers.

8. “Washback” Effects • Testing has a negative impact when: • Educational goals are reduced to those that are most easily measured. • Testing procedures do not reflect course goals, for instance… • Giving multiple choice tests in writing classes. • Using grammar tests as a measure of general proficiency. • Basing speaking ability on pronunciation alone.

9. Washback Effects of Tests • Testing has a positive impact when: • Tests reinforce course objectives. • Tests act as change agents for improving teaching and learning.

10. If Tests Are to bePositive Motivators • We have to select the right type of test for each testing purpose.

11. 3 Major Types of Tests • Achievement • Performance • Proficiency

12. 3 Major Types of Learning • Limited Transfer • Near Transfer • Far Transfer

13. Aligned Test and Learning Types • Achievement (Limited Transfer) • Memorized responses using the content of a specific textbook or curriculum. • Performance (Near Transfer) • Rehearsed ability to communicate in specific, familiar settings. • Proficiency (Far Transfer) • Unrehearsed general ability to accomplish real-world communication tasks across a wide range of topics and settings.

14. More on Types of Tests • Achievement Tests measure: • Rehearsed, memorized responses. • What was taught. • Content of a specific textbook or curriculum.

15. Sample Achievement Test Item Complete the following with the correct verb form in the past tense. (go) I _____________ to the United States last year. (be) My seat on the plane _______ in business class. (have) My associates and I _________ meetings each day. (eat) We _________ at typical American restaurants.

16. More on Types of Tests • Performance Tests measure: • Semi-rehearsed and rehearsed responses. • Ability to communicate in constrained, familiar, and predictable settings. • What one can do with what has been taught and practiced.

17. Sample Performance Test Item Complete the following sentences about an upcoming business trip. Add a minimum of 5 additional words to each sentence. For an upcoming business trip I plan to __________________________________________. I am certain that the trip will be successful, because __________________________________________.

18. More on Types of Tests • Proficiency Tests measure: • Spontaneous, unrehearsed communication ability. • General ability to accomplish communication tasks in a variety of settings. • Whether skills are transferable from one context to another.

19. Sample Proficiency Test Item You will be taking a business trip abroad. Plan an itinerary that spends at least two days in each of the three cities you must visit and costs less than $4,000 for all travel expenses. Then negotiate with a travel agent to purchase the airplane tickets, arrange hotel reservations, and obtain sufficient information about local transportation options to be able to complete the trip within your budget.

20. What Distinguishes Proficiency Testsfrom other tests? • They test real world tasks. • They measure a person’s ability to function in a language. • They provide an overall evaluation across a range of real-world tasks. • They rate a person’s unrehearsed ability against a set of task, conditions, and accuracy criteria.

21. ACTFL Proficiency ScaleNovice Memorized language • Lists words/phrases • Telegraphic • Attempts at conversation • Reactive • Limited topic areas • Social courtesies • Dates, numbers, colors • Family, home, common objects • May be difficult to comprehend beyond memorized material.

22. ACTFL Proficiency Scale Intermediate Survival Proficiency • Has sufficient language to create and express own meaning • Engage in simple conversation • Deal with a simple social transaction • Ask and answer questions • Comprehensible to a sympathetic conversation partner

23. ACTFL Proficiency ScaleAdvanced Limited Work Proficiency • Speaks with confidence • Can narrate and describe in all major time frames • Can elaborate, clarify, illustrate • Can handle a situation with a complication • Can be a “Story Teller” • Fully comprehensible to native speakers

24. ACTFL Proficiency ScaleSuperior Professional Proficiency • Can support opinions and hypothesize • Converse both formally and informally • Handle abstract treatment of subject • No pattern of linguistic errors

25. ACTFL Criteria: Speaking

26. Quick Review • 3 main types of Tests. • Ac… • Pe… • Pr…

27. A Summary that Contrasts: Achievement, Performance and Proficiency

28. Matching Test Type with Testing Purpose…

29. Some CommonTesting Purposes • Assigning grades in a class. • Placing students into a sequence of courses. • Selecting an applicant for a job with limited, static language requirements. • Screening employees for future jobs with broad, general language requirements.

30. What would happen if students,who were studying the same textbook,were given achievement tests by different teachers? • Unless the two tests asked exactly the same questions, the students’ would have different responses on one test than on the other. • Even if the questions were the same, unless the teachers graded using exactly the same criteria, each student’s score would be different.

31. What would happen if the same students were tested on their rehearsed performance by University A and University B? • Unless tests A and B covered exactly the same performance areas, the students’ performance on one test would be different from their performance on the other test. • Even if the tests were identical, unless the raters from both Universities applied the same performance standards, the students would be given different ratings.

32. And what would happen if you compared students’ classroom achievement ratings with their performance ratings on a university test with their proficiency ratings? • Those who can pass an unrehearsed, general proficiency test can also pass a performance test and an achievement test. • Those who can pass an achievement test, or a rehearsed performance test may not be able to pass a general, unrehearsed proficiency test.

33. And what does all this mean?

34. All Three Types of LanguageTests are Needed.

35. 3 Major Types of Tests • Achievement = Memorized responses using the content of a specific textbook or curriculum. • Performance =Rehearsed ability to communicate in constrained, familiar settings. • Proficiency = Unrehearsed general ability to accomplish real-world communication tasks across a wide range of topics and settings.

36. Activity # 1 • You will be asked about 8 different testing purposes. • For each of those test purposes, which type of test would you choose? a. Achievement b. Performance c. Proficiency

37. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course?

38. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses?

39. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course?

40. Which type of test would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks?

41. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be your spokesperson on a news show with a “hostile” moderator?

42. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be your spokesperson on a news show with a “hostile” moderator? • To document employees’ language ability in their personnel files?

43. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be your spokesperson on a news show with a “hostile” moderator? • To document employees’ language ability in their personnel files? • To compare the learning of your students with those of other students using the same text book?

44. Which test type would you choose:Achievement, Performance, or Proficiency? • To assess students’ language learning after Chapter 3 of a beginning language course? • To place students into a university’s sequence of courses? • To test students completing a year-long, intensive language course? • To screen job applicants for a specific job with well-defined, repetitive tasks? • To select someone to be your spokesperson on a news show with a “hostile” moderator? • To document employees’ language ability in their personnel records? • To compare results of my students with those of other students using the same text book? • To compare the skills of students in Study Abroad programs with “regular” students?

45. Solving Testing Problems • “The solutions to our problems should be as simple as possible, but no simpler.” Albert Einstein • There is no answer for the overly simple question of “Which test is best?” • There is an answer to the question, “Which type of test is best for a given purpose?”

46. Which type of test is best? • The test that matches the purpose for which the results will be used. • Use achievement tests for testing mastery of lessons in a textbook. • Use performance tests for checking rehearsed abilities within specific contexts. • Use proficiency tests for determining general, unrehearsed ability in real-world situations.

47. If You Do Want to TestReading and Listening Proficiency • It is not as easy as you might think. • Start by answering the question, “What is reading?”

48. A ProposedDefinition of Reading • Reading: The process of deriving meaning from the written symbols used to represent a given language.

49. But What isReading Proficiency? • Reading for achievement purposes may be defined differently for each curriculum. • Reading for specific performance purposes can result in a different definition of reading for each purpose.

50. But What isReading Proficiency? • “Proficient Reading” has some consistent, core expectations: • Understanding of texts for the purpose(s) for which they were written. • Automatic comprehension rather than laborious decoding. • Comprehension abilities that are sustained beyond one’s own areas of specialization.