How to convince your friends NOT to misuse raw scores

1. How to convince your friends NOT to misuse raw scores Benjamin D. Wright Institute for Objective Measurement & MESA Psychometric Laboratory bd-wright@uchicago.edu COMET

2. THE TROUBLE WITH RAW SCORES • The BROKEN BUCKET of Missing Data • The CRUMBLING CATEGORIES of Lumpy Ratings • The DIRTY DATA of Unpredictable Responses • The PERVERSE PRECISION of Extreme Scores • The RUBBER RULER of Irregular Intervals and Squashed Extremes COMET

3. The BROKEN BUCKET of Missing Data • Compare Two Patients on an 8-item, 7-category Functional Independence Measure: • Patient A: 2 3 3 m m m m 4 = 12 • Patient B: 1 2 2 2 2 2 2 3 = 16 • Which Patient is more Able? • Patient B has the higher score 16 > 12 on all 8 items, BUT On the 4 Items A and B have in common • Patient A: 2 3 3 m m m m 4 = 12 • Patient B: 1 2 2 m m m m 3 = 8 • Patient A has the higher score of 12 > 8 • Are you sure you want to misuse missing-data-leaking raw scores for missing-data-impervious measures? COMET

4. The CRUMBLING CATEGORIES of Lumpy Ratings • Rating Forms Offer Equally Spaced Categories • 1 . 2 . 3 . 4 . 5 . 6 . 7 • But Raters Reply with Unequally Spaced Responses • 1 . . . 2 . . 3 4 . 5 . . 6 . . . . . 7 • The Measure Distance of One More Point from Category 1 to 2 can be FOUR times BIGGER than The Measure Distance of One More Point from Category 3 to 4 !! • Are you sure you want to mistake Lumpy Ratings for Equal Interval Measures? COMET

5. The DIRTY DATA of Unpredictable Responses • When Item Responses are Arranged from Easy Items to Hard Items you Expect Response Patterns like: • 7 7 6 6 6 5 5 4 = 46 and 4 4 3 3 2 1 1 1 = 19 • BUT suppose you get: • 7 7 6 6 {1}5 5 4 = 41 ? or 4 4 3 3 {7}1 1 1 = 24 ? • What then? • Raw Scores are Blind to Unpredictable Responses. • Only Quality Control of Well-Constructed Measures Tells you about Response Surprises • Are you sure you want to suffer raw-score-dirty-data blindness instead of enjoying data-vigilant measures? COMET

6. The PERVERSE PRECISION of Extreme Scores • The Statistical Precision of a Raw Score is MAXIMUM at exactly the place where the Information a Raw Score Provides is MINIMUM • When a person gets the lowest possible score, their raw score precision is perfect.We know exactly the score their low ability implies. But we have no idea how far Below that Score their ability might be! • It is the same with the highest possible score. We know exactly the score their high ability implies. But we have no idea how far Above that Score their ability might be! • They are Off-Our-Scale and Our Precision for their Unknown Measure is ZERO! • Are you sure you want to mistake imprecise raw scores for precise measures? COMET

7. The RUBBER RULER of Irregular Intervals and Squashed Extremes • When our items bunch in clumps of equally difficult items then a count of one more right answer within a clump implies only a little increase in our ability. • But when we leap ahead and our next right answer is in a distinctly harder clump, then we see that this one more right implies a large increase in our ability. • As for the ends of the test where one more right is from 0 to 1 or from all but one to all. • Then the implied change in our ability is infinite. COMET

8. COMET

9. WHAT ARE VARIABLES? • Length and weight may be real variables. But we construct their units of measure. • Inches and ounces are our creations - Our own imaginative constructions. • A variable is an amount of something which we can always picture as a distance • >From Less -------------------------> To More • We can arrange to experience evidence of this "something". • But its measurement line and its units of measurement are up to us to construct. COMET

10. EVIDENCE OF A VARIABLE? • The variable and its evidence could be: • length benchmarks exceeded • health symptoms absent • ability problems solved • skill tasks completed • attitude assertions condoned • We can arrange to provoke occurrences of evidence and count how many pieces occur. • But these counts are not measures. COMET

11. REQUIREMENTS FOR MEASUREMENT • Pieces of evidence must be concrete to be observed. • This necessary reality keeps them uneven in size. • To measure we need an even abstraction, a line marked out in abstractly equal units. • Pieces of evidence are unstable. They appear and disappear by accident. They are only probable signs of the variable which they are designed to manifest. • To measure, we must find a way to connect the pieces of evidence we can arrange to observe to the probabilities of the measures we want. COMET

12. WHAT IS MEASUREMENT? • DISTANCE (Length) was our First Variable • COUNTING Steps and Fingers was our First Measuring Operation • The Trouble with Counting is its UNEQUAL UNITS • How many apples fill a basket? How many oranges squeeze a glass? You may not believe it. But can we mix apples and oranges? • We do it all of the time, by WEIGHING them! COMET

13. CONSTRUCTING MEASURES • But weighing is a constructed abstraction.There are no tangible equal units. We have to invent them. • Equal feet are abstracted from real feet. Equal pounds are abstract real weights • We construct our instrumentation of the variables: length and weight to approximate units equal enough to serve our practical purposes • We measure so that we can use the past to plan and navigate the future. But the future is by definition UNCERTAIN COMET

14. HANDLING UNCERTAINTY • Imagine two batters: Smith bats 400 and Jones bats 200 • So which one will hit at their next batter-up? No way to know ahead of time. • Even Smith has only a 4 out of 10 record. We can’t wait to find out. • So which one shall we send to the plate? Smith's odds for a hit are 2/3; Jones' odds for a hit are only 1/4 • Smith odds for a hit are 8/3 times better than Jones'. • Even though we know nothing for sure, Does any doubt remain as to who to send to bat? • That's how we handle uncertainty. We use past experience to estimate PROBABILITIES and use these probabilities to forsee the future. COMET

15. COUNTING ABSTRACT UNITS • To finish this job we have to construct a reproducible transition from counting concrete events, like right answers, observed or reported symptoms, relative agreements, frequency or importance categories to counting abstract units of equal size and wide generality. • How can we do this? COMET

16. INVERSE PROBABILITY • To deal with the uncertainty we ask Bernoulli, Bayes and Laplace and interpret our observation X as evidence of its occurence probability Px • To construct unit equality we ask Campbell, Thurstone, Rasch and Luce & Tukey and define Px to satisfy the equation: log[Pnix/(1-Pnix)] = Bn - Di • Pnix is the probability of a successful response Xni being produced by person n to item i • Bn is the ability of person n • Di is the difficulty of item I • The construction of equal size and hence additive units is called CONJOINT ADDITIVITY COMET

17. CONJOINT ADDITIVITY • For situations where Xni occurs in incremental steps such as Xni = 0,1,2,3,,,M • This simple solution generalizes to • log[Pnix/Pnix-1] = Bn - Di - Fix • For situations where Xnijk = 0,M occurs as the result of • a Rater j rating the performance of • a Person n on • a Task k • This MEASUREMENT MODEL becomes • log[Pnijkx/Pnijkx-1] = Bn - Di - Cj - Ak - Fix COMET

18. CLOSING THE DEAL • Raw scores cause problems with: • missing data • lumpy ratings • unpredictable responses • extreme scores • irregular intervals and squashed extremes • Rasch measurement provides a solution to these problems by: • abstracting units of measurement • using probabilities to predict futures • constructing equal-sized intervals COMET