1 / 30

Users’ reactions to innovative computerised feedback – the case of DIALANG

Users’ reactions to innovative computerised feedback – the case of DIALANG. Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland. What is DIALANG?. computerized diagnostic language assessment system in 14 languages (www.dialang.org)

hasad
Download Presentation

Users’ reactions to innovative computerised feedback – the case of DIALANG

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Users’ reactions to innovative computerised feedback – the case of DIALANG Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland

  2. What is DIALANG? • computerized diagnostic language assessment system in 14 languages (www.dialang.org) • feedback (& activities) that help learners to • diagnose their strengths and weaknesses • plan further language learning • raise awareness • unusually wide range of feedback for a test; partly unique / innovative, e.g. self-assessment

  3. What is ’feedback’? Why feedback? In educational contexts FB can be: • ”… any communication or procedure to inform a learner of the accuracy of a response, usually to an instructional question” (Knowledge of the Correct Response) • but also ”… other information such as precision, timeliness, learning guidance, motivational messages, ... critical comparisons, and learning focus” • ”… any message or display that a computer presents to the learner after a response” (Mory 2004) • improve performance & learning (directly) • but also to motivate, make learning more interesting, encourage different learning, raise awareness, …

  4. Why study (DIALANG) feedback? • feedback is an integral part of all behaviour, performance and learning • social psychology, organizational theory, education ... • feedback from tests is not much researched (cf. teachers as assessors) • feedback from computerized (language) tests is particularly little researched

  5. DIALANG FEEDBACK

  6. Research topics / questions This study: • What are users’ reactions to different types of feedback in DIALANG? How do new types of feedback compare with more traditional feedback? • Are users’ reactions associated with their background characteristics (e.g. sex, age, language use, language proficiency)? How might possible associations be explained? -------------------------------------------------------- • related, more in-depth studies users’ reactions to self-assessment and Vocabulary Size Placement Test

  7. Research methods & data • Quantitative & qualitative (cross-sectional) • Questionnaire • 553 respondents in two countries • mostly non-language majors at universities (typical users) • Interviews of users and their teachers • Written reports from users & teachers

  8. Users’ ’reactions’? Defined in terms of • 1) Reading / using particular feedback? • 2) Interest in particular feedback? • 3) Perceived usefulness of particular feedback? • 4) Other reactions (e.g. ease of use) • Likert scale + open-ended questions • cf. Technology Acceptance Model

  9. Who were the respondents?

  10. Age of respondents

  11. # of languages & tests taken

  12. Most common test languages

  13. Results: Overall reaction to FB- use & interest & usefulness

  14. Reactions to Level (test result)

  15. Reactions to Level (test result) • the most popular part of feedback • the fact that the meaning of A1 etc was defined was appreciated • associations with user background: • correlated with proficiency: higher proficiency – more positive reactions • men were less interested in this than women

  16. Reactions to Vocabulary Size Placement Test feedback / result

  17. Reactions to Vocabular Size Placement Test & its feedback • VSPT and its result was the most controversial part of DIALANG although, overall, the reactions were positive • strong affective reactions, opinions rather divided • “Interesting, I had never thought of a way like this to test language proficiency. Fun to do!” • “Absolute nonsense! This can’t be my level!” • background: in general, those who got low VSPT results regarded it very negatively, whereas those who got good results regarded it very positively • in particular, women at level C1/C2 liked VSPT feedback • explanation: VSPT scoring is affected by test-taking strategy and produces very low scores too easily

  18. Reactions to immediate item review • almost 60% preferred immediate review, 30% post-test review, 10% preference • Why like: want to know immediately, remember better if see the result right away, want to learn (during test) • Why don’t like: disturbs test-taking, can be depressing • background: • beginners liked it more than advanced learners • in particular, C-level (C2) learners considered it less useful and interesting • C-level men used it particularly infrequently • possible explanation: • many beginners & intermediate learners want to learn from tests?? • advanced learners make so few mistakes that they may feel immediate feedback is waste of time??

  19. Reactions to post-test item review • less popular than immediate review but a fair number of users absolutely want to have it • Background: • more experienced users (# of languages used & length of language studies) considered this more interesting & useful • possible explanation (very tentative): • more experienced language learners may find information about subskills interesting??

  20. Reactions to self-assessment feedback (comparison of SA and test result)

  21. Reactions to self-assessment feedback • according to users, SA is important part of (a system like) DIALANG, • as ‘popular’ as e.g. item review (joint second) • background: • men had more often very negative opinions compared with women • the youngest group (under 18) found it less interesting than the older groups • possible explanation: • younger learners may not (yet) be so interested in the kind of reflection on one’s skills involved in self-assessment??

  22. Reactions to self-assm. information (why SA and test result may not match)

  23. Reactions to SA information • the least used / interesting / useful part of feedback • Background: negative reactions were associated with • higher proficiency, especially C1/C2 women • more experience in lg learning (more lgs studied) • male respondents (more very negative opinions) • young age (under 18) • positive reactions were associated with: • lower proficiency, especially A1/A2 men • less experience in lg learning (fewer lgs studied) • men were divided in their opinions of SA information • possible explanations: • if SA and test agreed, no need to read this • beginners have less experience in what is described in SA information and may find it new & potentially useful; advanced learners may feel they already know all that

  24. Reactions to extended descriptions

  25. Reactions to extended descriptions • fairly interesting, useful and used but less so than e.g. item review • background: • men made more often very negative evaluations of this feedback than women

  26. Reactions to advice

  27. Reactions to advice • quite popular but not among all users – surprise? • Background: • high proficiency (C1/C2) & more experience  more negative views of advice • men had more very negative views than women • A1 / A2 learners were divided: they often chose it as one of the 3 most useful types of FB but many also chose it as one of the least useful types of FB • possible explanation: • C-level learners no longer need advice, whereas A-level learners do, but some of them may feel the particular advice in DIALANG is not useful??

  28. Implications / questions • implications for DIALANG and/or other such computerized systems • users clearly like to have feedback about their self-assessment • younger learners may need to be motivated first • good to have two types of item review: immediate and post-test, based on choice • preferences differ; level of proficiency

  29. Implications / questions • VSPT divides opinions: love – hate • scoring needs revising • advice appears useful for beginners in particular • challenges ”I don’t bother to read the advice because I know I’ll do things my way anyway” ”... you don’t remember them any more when you study” • longitudinal studies • how best integrate systems like DIALANG into teaching • (long-term) effectiveness of feedback?

More Related