Heriot Watt University Breakout Session MCQ with Confidence

1. Heriot Watt UniversityBreakout SessionMCQ with Confidence Dr Phil Davies Division of Computing & Mathematical Sciences Department of Computing FAT University of Glamorgan

2. E-Assessment can be viewed from different perspectives • Commercial / Large Scale Implementations • Question Mark’s Perception • MOODLE • Triads (25 different types of Multiple Choice Type Questions!) • TOIA (FREE!) Tool for On-line Interoperable Assessment • Blackboard Assessment (version!) • Standards (IMS QTI Specification) • Try it and see

3. How easy to do MCQ?Cost? • Question • <normally> • Key = • Distracter One, Distracter Two, …………….. • Note • Key(s) must always be correct • How many students would you expect to get it right for it to be a GOOD question? • Distracter One / Two / Three equal # of selections • Throw away BAD questions?

4. TOO HARD .. ? • Too time consuming • Often needs to be an iterative process • Does it matter if a question is NOT perfect? • Seeking perfection not feasible • I suggest that assessment MUST be included within a module, not only at end > make it embedded within TEACHING & LEARNING experience • Formative / Summative / Both

5. My Entry into CAA • 1st Attempt TOTAL FAILURE – 1998 OLAL • 5 minutes all finished apart from one student • Guessing • Why should the student care if incorrect? • How to make it worthwhile – slow students down – make them think • Negative marking used +2 and -1 (actually becomes -3 –> double whammy!) • Allow TWO PASSES of the test • By concentrating on first pass • will get reward on second pass • Formative • IMPORTANT – WORTH MARKS

7. Own Background = Objective Testing via MCQ • Four Tests MCQ .. Two passes 50/50 (became graduated) • Cumulative Tests 5%, 10%, 15%, 20% • Amateur .. No Research Pedigree (CAA Conference) • Improved Class results > not easier – more directed • Weaker students improved dramatically • Assessed the learning outcomes of subject • “END JUSTIFIES THE MEANS”

8. What did the students say about it? • Excellent way of gaining knowledge • 2nd Test Allowed us to learn • Good assistant to learning • Encourages Revision • Very good, promotes learning of a subject early on • Better than sex!!!!

9. What decides the QUALITY of a question? • In the 2004 World cup qualifier between England and Greece, who scored the all important equalizing goal? • David Beckham • Victoria Beckham • Brooklyn Beckham • Romeo Beckham

10. Rewards? • Know it • On the tip of my tongue • If I see the answers I’ll know it • Deduction • Guess • Know it … oh no I’ve got it wrong .. misinformation

11. TRIED ONE CONFIDENCE TEST ONLY SHOWS ANSWERS ONCE CONFIDENCE SELECTED RELATES TO MARKS +4, +2, +1, -2, -1, 0

12. Proportion of Questions Correct • High Confidence => 84% correct • Fairly Confident => 58% correct • Not confident => 47% correct • 72% of all questions answered with high confidence

13. Student thoughts on Confidence • “Eliminates guesswork” • “I was waiting for a leggy blond to bring on my prize at the end” • “separated the lucky students from me” • “at least I didn’t lose marks for selecting no confidence” • “are you testing my confidence or knowledge”

14. “Bite The Bullet” • Use Confidence Testing for all four tests (10%, 15%, 20% & 25%) • Real SHOCK to students • Weighting of Passes (50/50, 60/40, 70/30 & 80/20) • How to judge if the students had ‘really’ learned? • Select five questions that were answered ‘poorest’ in test one & pass through to test two • Select five questions that were answered ‘poorest’ in test two & pass through to test three ( + original five) …. Etc.

16. Identifying Quality of a Question • Negative on first pass e.g. > -180 • Difference between first and second pass • Too big an improvement may just signify a very poor question e.g. >500? • Create heuristic that identifies degree of difficulty of a question

17. Passed through to Test Four • 13 of the ‘hardest’ question • 6 new questions based upon additional work between tests 3-4 • 21 questions (cross section of questions and topics from previous tests)

18. Improvement of Results Through Tests

19. Improvement Pass One of Questions

20. Improvement for Question 5 • Stronger Students improved quickly through the tests • Became more confident in answering the questions • Weaker Students improved less quickly than stronger through the test progress, but DID improve

21. Is it just Confidence or Knowledge i.e. questions right or wrong?

22. Quality of a multiple choice question? • In the 2004 World cup qualifier between England and Greece, who scored the all important equalizing goal? • Teddy Sheringham • Michael Owen • Robbie Fowler • QUALITY OF QUESTION NOT DISTRACTERS

23. Developmental v Assessment • Need not be at alternative ends of the scale • Student motivation / attendance increased considerably • Testing knowledge (not done at moment for higher order skills) • Good students have best results i.e. number of questions correct

24. COMMENTS? • HMCQ • Confidence • Two Pass • Formative & Summative • Graduated Assessment • Continual Assessment