We need to know about … Computer Assisted Assessment (CAA)

1. We need to know about …Computer Assisted Assessment (CAA) Dr Andrew Oliver LTDU & BLU University of Hertfordshire

2. Objective Testing (in StudyNet) • StudyNet Quizzes are ‘objective’ • In other words … • Student provides a response to a question which has a pre determined answer • Which means • Select solution from a list of choices (MCQ) or • Enter short text or numerical responses (matched to correct answer)

3. Objective Tests • Diagnostic • Measure understanding in-class & adjust teaching, e.g. Electronic Voting Systems (EVS) • Formative • Provide ongoing feedback in-course and motivation, e.g. StudyNet • Summative • Exam type forming part of final course grade, e.g. QuestionMark

4. Advantages • Immediate feedback to students (and staff) • Decreased marking time • More frequent assessment • Provide statistics (improve question design) • STUDENTS LIKE IT ! (formative that is)

5. Challenges • Time investment for design of questions & tests • As above but with technology • Measure high order skills ? • Yes – but time required to design • Difficult to assess essays (and hence creativity) • Well maybe, may be not • CHEATING • Network stability

6. Objective Testing at UH • StudyNet: designed for formative use • QuestionMark: summative (conditions apply, contact r.a.shaw@herts.ac.uk)

7. StudyNet Quizzes: Question Types • Multiple Choice • True/False • Multiple Response • Text / Numerical input • Ranking

8. StudyNet Quizzes: Where are they? • Every Module website • Teaching Resources > Quizzes & Forms • Step 1: Question Wizard • Create questions with images, text formatting, variable feedback/scoring etc. • Step 2: Quiz Wizard • Create quizzes with using previously created questions, adding feedback tailored to score • Each question is a single resource & can be used in > 1 quiz

9. Question features • You can: • Include images (stem or option) • Feedback to option chosen • Negative scoring • Confidence based scoring

10. Quiz features: • You can: • Add extra info at the top & base of quiz • Tailor feedback according to user score • Provide remedial or complex material based on score • Number questions per page • Allow re – try before viewing detailed results • Anonymous or named submission

11. Viewing Results • Go to Teaching Resources > Quizzes & Forms • Find Quiz & click ‘R’ • Click ‘Submissions’ to list students • Click student name for details

12. Statistics! • Identify & remove weak questions • e.g. too easy or too difficult • Also find incorrect or poorly worded questions • Pre test questions for future • AND • Problem areas in module • Struggling students • Students who are NOT being challenged

13. StudyNet Quiz Statistics • Go to Teaching Resources > Quizzes & Forms • Find Quiz & Click ‘R’ • Click ‘Question Statistics’ & ‘More Statistics’ • Per questions: mean score, SD, Difficulty Index, Correlation (Pearsons), Mean for outcome (MCQ only)

14. Guessing • Corrective scoring • +1 correct, -1 incorrect, 0 no answer • But contentious • NB relevance decreases with as questions increase

15. Confidence Based Scoring • Choose: low medium high • For question correct score 1: • Correct 1 (low) 2 (med) 3 (high) • Incorrect -1 (low) -2 (med) -3 (high) • Tell the students

16. Why? • Gets students to: • Think about reliability • Understand issues (not instant) • But also: • Fairer (confident vs guess) • Wake up students (confident incorrect)

17. Question Design • Part 1: basic tips • Question stem • Question distractors • Part 2: Bloom’s taxonomy

18. Design: the question stem • single, definitive statement • avoid unnecessary or irrelevant material • use clear language • use negatives sparingly • put as much of the question in the stem • Any more?

19. Bad question I • Morphemes • a. are made up of smaller units called phenomes • b. are NOT found in grammatical function words • c. create meaning or have a grammatical function • d. can be described as free or bound

20. Good question I • Morphemes are the smallest contrastive unit of • a. compound words • b. prefixes • c. grammar • d. intransitive verbs • Clear, shorter, homogenous choices. • Provide single clear statement.

21. Bad Question II • Paul Muldoon, an Irish postmodern poet who uses experimental and playful language, uses which poetic genre in “Why Brownlee Left”? • a. sonnet • b. elegy • c. narrative poem • d. dramatic monologue • e. haiku

22. Good Question II • Paul Muldoon uses which poetic genre in “Why Brownlee Left”? • a. sonnet • b. elegy • c. narrative poem • d. dramatic monologue • e. haiku • Contains material irrelevant to the question. • Avoid unnecessary and irrelevant material.

23. Bad Question III • As mortality rates approach the zenith, what is the most likely ramification for the citizenry of an industrial nation? • a. an increase in the labour force participation rate of older people • b. a dispersing effect of population concentration • c. an upward trend in the youth dependency ratio

24. Good Question III • A major increase in mortality rates in an industrial nation is likely to cause • a. an increase in the labour force participation rate of older people • b. a dispersing effect of population concentration • c. an upward trend in the youth dependency ratio • Too complex – tests reading comprehension ? • Use clear, straightforward language

25. Bad Question IV • Which of the following is not a symptom of osteoporosis? • a. decreased bone density • b. frequent bone fractures • c. raised body temperature • d. lower back pain

26. Good Question IV • Which of the following is a symptom of osteoporosis? • a. decreased bone density • b. raised body temperature • c. hair loss • d. painful joints • Use negatives sparingly. • If needed then capitalize, underscore, embolden or otherwise highlight.

27. Bad Question V • Maslow’s theory of growth motivation asserts that • a. humanistic needs are physiological • b. humanistic needs are emotional • c. humanistic needs are intellectual

28. Good Question V • Maslow’s theory of growth motivation asserts that humanistic needs are • a. physiological • b. emotional • c. intellectual • Put as much of the question in the stem as possible, DON’T duplicate in each of the options

29. Design: the distractors • ensure there is only one correct answer • use plausible distractors • avoid clues to the correct answer • base distractors on common student errors (phenomenographics) • Any more?

30. Bad Question VI • Which of the following texts is considered to represent the pinnacle of modernist achievement? • a. The Waste Land • b. Middlemarch • c. “Ode to a Nightingale” • d. Ulysses • e. Ethan Frome

31. Good Question VI • Which of the following texts represents one of the high points of modernist achievement? • a. The Waste Land • b. Middlemarch • c. “Ode to a Nightingale” • d. Ethan Frome • e. “My Last Duchess” • ensure that there is only one correct response. • both options ‘a’ and ‘d’ could be considered to be correct.

32. Bad Question VII • The Dichotic Listening Test determines which side of the brain is directly involved in • a. tone perception • b. selection attention • c. cognition • d. hearing sounds

33. Good Question VII • The Dichotic Listening Test determines which side of the brain is directly involved in • a. tone perception • b. selection attention • c. pitch constancy • d. hearing sounds • Use only plausible and attractive alternatives as distracters. • ‘c’ not plausable

34. Bad Question VIII • A fertile area in the desert in which the water table reaches the ground surface is called an • a. mirage • b. oasis • c. water hole • d. polder

35. Good Question VIII • A fertile area in the desert in which the water table reaches the ground surface is called a/an • a. mirage • b. oasis • c. water hole • d. polder • Avoid giving clues to the correct answer • uses the article “an” which identifies choice “b” as the correct response.

36. Bloom’s Taxonomy est. 1956 • hierarchy of educational objectives • divides cognitive objectives into subdivisions • range from the simplest behaviour to the complex • other systems or hierarchies exist

37. Cognitive subdivisions • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

38. Bloom’s Taxonomy: Knowledge • Recall of information • Bringing to mind the appropriate information • Learning objectives: know facts, know theories, know terms, know events, know basic concepts, know places, know methods • Question cues: list, define, label, describe, name

39. Aim - to recall a specific date Question: in which year did the American Civil War end? a. 1832 b. 1865* c. 1857 d. 1888

40. Bloom’s Taxonomy: Comprehension • Ability to grasp meaning of the material – to interpret, to predict, to estimate • Learning outcomes: understand facts, interpret graphs, translate verbal to math formulae, estimate consequences from data. • Question cues: interpret, discuss, predict, summarise, classify

41. Aim - to understand and interpret definitions, relationships & analogies Question: which one of the following describes what takes place in the so-called PREPARATION stage of the creative process, as applied to the solution of a particular problem? 1. The problem is identified and defined. 2. All available information about the problem is collected. 3. An attempt is made to see if the proposed solution to the problem is acceptable. 4. The person goes through some experience leading to a general idea of how the problem can be solved. 5. The person sets the problem aside, and gets involved with some other unrelated activity. KNOWLEDGE is recalled and understanding (COMPREHENSION) tested of the meaning of each term (preparation).

42. Second Example: comprehension Question: In the following, a related pair of words is followed by five more pairs of words. Choose the response pair that best expresses a relationship similar to that expressed in the original pair. QUENCH : THIRST a. staunch : wound b. douse : fever c. antidote : poison d. extinguish : fire*

43. Bloom’s Taxonomy: Application • Ability to use learned materials in NEW situations • Application of rules, methods, concepts and theories. • Learning outcomes: apply principles / theories to new situations, solve math problems, construct graphs. • Question cues: apply, demonstrate, show, relate, calculate

44. Aim: to calculate velocity Question: End A of the cord is moving 6m/s to the left. Compute the velocity of the block B and select one of the responses below. a. 2 m/s (to left) b. 3 m/s (to left) c. 6 m/s (to left) d. 12 m/s (to left) e. none of the above Nb there would be a diagram here!

45. Second Example: application Which one of the following values approximates best to the volume of a sphere with radius 5m? • 2000m³ • 1000m³ • 500m³ • 250m³ • 125m³

46. Bloom’s Taxonomy: Analysis • Ability to break down material into components and understand patterns (underlying structure) • Recognition of parts and their relationship to each other • Learning objectives: recognise unstated assumptions, evaluate the relevancy of data • Question cues: analyse, arrange, order, explain, connect, infer, compare, categorize

47. Aim: to analyse and infer from a geological map Question 1. LANDSLIPS At which of the following contacts between strata would you expect MOST landslips to occur? Choose 2 of the following options. a. Chalk above Gault Clay b. Gault Clay above Lower Greensand c. Corallian limestone above Oxford Clay d. Cornbrash limestone above inferior Oolite limestone

48. Bloom’s Taxonomy: Synthesis • Use old ideas to create new ones • Organise and relate knowledge from several areas • Learning objectives: writing themes & stories, formulating a new scheme for classifying objects (or events, or ideas). • Question words: integrate, modify, invent, design, compose, plan, formulate, arrange

49. Aim: to organise and arrange appropriate critical terms in order to construct a geological analysis of the following photographic image Question: Move the appropriate descriptive terms from the list to the ‘form’ and ‘attitude’ boxes below. Question created by Professor Don MacKenzie, TRIADS.

50. Bloom’s Taxonomy: Evaluation • Make judgements of / evaluate data • Assess value of ideas, theories • Compare and discriminate between ideas • Learning outcome: judge the adequacy with which conclusions are supported by data • Question cues: appraise, evaluate, defend, rank, conclude, discriminate, recommend