Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

1. eAssessment for student learning: short free-text questions with tailored feedbackSally Jordan University of Chester Staff Conference, 2nd May 2008 Centre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT)

2. My plan • Introduction to the Open University and the Open University CETLs; • What is assessment for? • How and why I became involved in eAssessment • Our questions – demonstration • Our evaluation • Your turn – try writing a suitable question in your own subject area • Discussion of possibilities and limitations • Assessment for learning; learning from assessment?

3. The Open University • Supported distance learning; • 150,000 students, mostly studying part-time; • Undergraduate courses are completely open entry, so students have a wide range of previous qualifications; • Normal age range from 18 to ?? • 10,000 of our students have declared a disability of some sort; • 25,000 of our students live outside the UK.

4. OpenCETL An informal grouping of four Centres for Excellence in Teaching and Learning: • Personalised Integrated Learning Support Centre (PILS); • Practice Based Professional Learning CETL (PBBL); • Physics Innovations CETL (piCETL) – joint with the universities of Reading and Leicester; • Centre for the Open Learning of Mathematics, Science, Computing and Technology (COLMSCT).

5. COLMSCT Working primarily in three areas: • Online experimentation; • eLearning communities and identities; • Assessment. The E-Assessment for Learning Initiative aims to bridge the gap between academic aspirations and systems development. It supports 15 sub-projects from across the University. In each, academic staff undertake innovative e-assessment implementation projects within their own course context in order to explore and inform future development and pedagogy.

6. What is assessment for? • grading • teaching and learning • self reflection • engaging/pacing Summative (Assessment of learning) or Formative (Assessment for learning)

7. The Formative Assessment in Science Teaching (FAST) Project Gibbs and Simpson identified 11 conditions under which assessment supports student learning. (Gibbs, G and Simpson, C (2004) Conditions under which assessment supports students’ learning. Learning and teaching in higher education, Issue 1, pp3-31) 3 of the conditions relating to pacing and 4 of those relating to feedback underpin our work.

8. FAST conditions - pacing • Condition 1: Sufficient assessed tasks are provided for students to capture sufficient study time. • Condition 2: These tasks are engaged with by students, orienting them to allocated appropriate amounts of time and effort to the most important aspects of the course. • Condition 3: Tackling the assessed task engages students in productive learning activity of an appropriate kind.

9. FAST conditions - feedback • Condition 4: Sufficient feedback is provided, both often enough and in enough detail. • Condition 6: The feedback is timely in that it is received by students while it still matters to them and in time for them to pay attention to further learning or receive further assistance. • Condition 9: Feedback is received and attended to. • Condition 10: Feedback is acted upon by the student.

10. Feedback matters! • Within the Open University context, learners are geographically separated and we cannot assume that they will meet their tutor in order to receive feedback. Feedback on assignments therefore has an important teaching and learning function. • We are seeking to provide students with feedback which is personalised and received in time to be used in future learning and assessment tasks.

11. S104 : Exploring Science • A new interdisciplinary science course for students entering undergraduate study; • Students have a wide ranges of previous study backgrounds (open entry); • 60 CATs points, spread over 9 months; • We want to help students keep up to date in their studies; • Assessment aims to drive student engagement with the learning process; • Frequent small assessment tasks to help students monitor their own progress.

12. Interactive online assessment • So we are using regular iCMAs (interactive computer marked assignments) with tailored feedback (alongside conventional tutor marked assignments, marked against learning outcomes); • The iCMAs marks count towards the course score (but they are low stakes – 20% of overall score), so students are motivated to tackle them. However their primary function is to provide pacing and immediate feedback. • Assessment is both summative and formative.

13. The OpenMark system • Uses a range of question types, going far beyond what is possible with multiple choice; • Question types include: • Numerical input • Text input • Drag and drop • Hotspot • Students are allowed three attempts with an increasing amount of teaching guidance, wherever possible tailored to the student’s previous incorrect answer; • Different students receive variants of each question so each has a unique assignment. Demonstration

17. The OpenMark system • Is being used by a number of courses across the university in summative, purely formative and diagnostic use; • Underpins most of the projects in the COLMSCT ‘E-assessment for learning initiative’; • Has good accessibility features; • In purely formative use the different variants of each question provide students with extra practice; • OpenMark has been incorporated into Moodle, the open source virtual learning environment being used by the OU.

18. Questions for S104 : Exploring science • We want to be able to ask questions that require free text answers of a phrase or sentence in length; • We are working with a commercially provided authoring tool (from Intelligent Assessment Technologies); • Student responses are being used to refine the questions; • We are providing targeted feedback on incorrect and incomplete answers.

19. Short answer questions • We want to mark many different answers as correct; • ..but we want to mark incorrect answers as incorrect; • The fact that the software is linguistically based means that it recognises ‘dog bites man’ as different from ‘man bites dog’; • It copes well with poor spelling and, usually, with poor grammar. • Demonstration

24. Evaluation 1:User lab observations • Six students were observed in June 2007; • They reacted to the free-text questions in interesting ways e.g. because they thought we were just looking for keywords, some tried to ‘help’ the computer by giving answers in note form; one of the students appeared to ‘brain dump’; • Use made of the feedback provided was also variable; one student said he found it useful but clearly didn’t use it; others read the feedback carefully, checked things in their course book, and altered their response successfully.

26. Evaluation 2:Human-computer marking comparison • The computer marking was compared with that of 6 human markers; • For most questions the computer’s marking was indistinguishable from that of the human markers; • For all questions, the computer’s marking was closer to that of the question author than that of some of the human markers; • The computer was not always ‘right’, but neither were the human markers.

27. Writing questions:Our choice of software • We were looking for software to mark the content of short free-text questions; • This is characteristically different from systems which mark essay-writing, from systems which mark essays and from systems which ‘pre-group’ answers for human marking; • We chose the IAT software because it had an authoring tool that I stood a chance of learning how to use; • The software is based on the Natural Language Processing technique of ‘Information Extraction’.

28. Writing questions:The IAT software • incorporates a number of processing modules aimed at providing robust marking in the face of errors of spelling and grammar; • represents mark schemes as templates; • synonyms can be added and verbs are usually lemmatised.

30. Writing questions:what makes a good question? • Answers that are difficult to parse include those that are very long and those in note form. • Questions have to be quite tightly constrained e.g. ‘You are handed a rock specimen that consists of interlocking crystals. How would you decide, from its appearance, whether it is an igneous or a metamorphic rock?’ has become ‘You are handed a rock specimen that consists of interlocking crystals. How could you be sure, from its appearance, that this was a metamorphic rock?’

31. Writing questions:Have a go! Either Have a go at writing a question (or two) in your own discipline that could be marked in this way (remember to think about what correct and incorrect answers students are likely to give. What targeted feedback would you like to give?) Or Think about the answer matching for the ‘Alice’ question on the handout. What correct and incorrect responses can you think of? What keywords are important in these answers? What targeted feedback would you give?

32. Discussion • Could you use questions like this in your teaching? • What would the advantages be? • What are the limitations? • What’s stopping you?

33. OpenMark – some other developments • OpenMark is being incorporated into Moodle, and Moodle Gradebook enables tutors to ‘replay’ the questions each student has received and to view their responses; • New question types are being developed (OpenMark examples site) • OpenMark is being used in interesting ways on various courses (mostly for purely formative use) e.g. • Feedback on course learning outcomes • Clinical decision making

36. Assessment for learning: learning from assessment? • We have records of student answers (used for Award Board decisions for borderline cases); • We can analyse these answers on a question by question basis. What are the common misconceptions? • The evidence is stronger when the assessment is summative and not multiple choice.

37. Analysis of responses from ‘Maths for Science’ questions: Science students’ mathematical misconceptions • e.g. students have problems with units; • Detailed analysis revealed that this is frequently because they do not know how to divide by a fraction; • Some of the findings have been expected, some have been surprising; • But the evidence is indisputable!

38. What do we do with what we have learnt? Alter the assessment tasks: • make the questions clear; • improve the feedback; • sometimes don’t ask that question! Improve our teaching.

39. Acknowledgments • The Centre for Open Learning of Mathematics, Science, Computing and Technology (COLMSCT) especially Barbara Brockbank, Phil Butcher and Laura Hills; Tim Hunt (LTS) • Tom Mitchell of Intelligent Assessment Technologies Ltd.

40. Our demonstration questions are at https://students.open.ac.uk/openmark/omdemo.iat/ Or if you want more… https://students.open.ac.uk/openmark/s103-07b.block3world/ https://students.open.ac.uk/openmark/s103-07b.blocks45world/ https://students.open.ac.uk/openmark/s103-07b.block7world/ https://students.open.ac.uk/openmark/s103-07b.block8world/ https://students.open.ac.uk/openmark/s103-07b.block9world/ https://students.open.ac.uk/openmark/s103-07b.block10world/ https://students.open.ac.uk/openmark/s103-07b.block11world/ For more about OpenMark http://www.open.ac.uk/openmarkexamples/index.shtml For more about Intelligent Assessment Technologies Ltd. http://intelligentassessment.com/

41. Sally JordanCentre for Open Learning of Mathematics, Science Computing and Technology (COLMSCT) The Open UniversityWalton HallMilton KeynesMK7 6AAs.e.jordan@open.ac.uk http://www.open.ac.uk/colmsct/projects/sallyjordan