Measurement and Scaling

Measurement and Scaling Farzin Madjidi, Ed.D. Pepperdine University Graduate School of Education and Psychology

Variables • Independent • Precedes, influences or predicts results • Dependent • Affected by or predicted by the Independent Variable • Extraneous • Affected by the D.V., but not controlled or measured. Causes error

Variables • Confounding • An extraneous variable that varies systematically (has a relationship) with the I.V. • Intervening • Unobservable trait that influences behavior (e.g., effect of new intervention on self-esteem may be affected by the motivation level of subjects)

Variables • Control • Used to eliminate the effect of extraneous variables • Organismic • Aka, measured, or assigned • Characteristics of the subjects that cannot be manipulated

Levels of Measurements Four levels of Measurements • Nominal • Measures categories • Ordinal • Categories + rank and order • Interval • Equal distance between any two consecutive measures • Ratio • Intervals + meaningful zeros

Categories of Scales • Categorical (ratings) • Score without comparison - 1 to 5 scales • Comparative (ranking) • Score by comparing - Smartest • Preference • Subjective - which do you prefer • Non-preference • Objective - which solution is less costly

Categories of Scales • Unidimensional • Involves only one aspect of the measurement • Measurement by one construct • Multi-dimensional • Involves several aspects of a measurement • Uses several dimensions to measure a single construct

Types of Scales • Likert/Summated Rating Scales • Semantic Differential Scales • Magnitude Scaling • Thruston Scales • Guttman Scales

Likert Scales • A very popular rating scale • Measures the feelings/degree of agreement of the respondents • Ideally, 4 to 7 points • Examples of 5-point surveys • Agreement SD D ND/NA A SA • Satisfaction SD D ND/NS S SS • Quality VP P Average G VG

Summative Ratings • A number of items collectively measure one construct (Job Satisfaction) • A number of items collectively measure a dimension of a construct and a collection of dimensions will measure the construct (Self-esteem)

Summative Likert Scales • Must contain multiple items • Each individual item must measure something that has an underlying, quantitative measurement continuum • There can be no right/wrong answers as opposed to multiple-choice questions • Items must be statements to which the respondent assigns a rating • Cannot be used to measure knowledge or ability, but familiarity

Semantic Differential Scales • Uses a set of scale anchored by their extreme responses using words of opposite meaning. • Example: Dark ___ ___ ___ ___ ___ Light Short ___ ___ ___ ___ ___ Tall Evil ___ ___ ___ ___ ___ Good • Four to seven categories are ideal

Magnitude Scaling • Attempts to measure constructs along a numerical, ratio level scale • Respondent is given an item with a pre-assigned numerical value attached to it to establish a “norm” • The respondent is asked to rate other items with numerical values as a proportion of the “norm” • Very powerful if reliability is established

Thurston Scales • Thurston Scales • Items are formed • Panel of experts assigns values from 1 to 11 to each item • Mean or median scores are calculated for each item • Select statements evenly spread across the scale

Thurston Scales • Example: Please check the item that best describes your level of willingness to try new tasks • I seldom feel willing to take on new tasks (1.7) • I will occasionally try new tasks (3.6) • I look forward to new tasks (6.9) • I am excited to try new tasks (9.8)

Guttman Scales • Also known as Scalograms • Both the respondents and items are ranked • Cutting points are determined (Goodenough-Edwards technique) • Coefficient of Reproducibility (CReg) - a measure of goodness of fit between the observed and predicted ideal response patterns • Keep items with CReg of 0.90 or higher

Scale Construction • Define Constructs • Conceptual/theoretical basis from the literature • Are their sub-scales (dimensions) to the scale • Multiple item sub-scales • Principle of Parsimony • Simplest explanation among a number of equally valid explanations must be used

Item Construction • Agreement items • Write declarative statements • Death penalty should be abolished • I like to listen to classical music • Frequency items (how often) • I like to read • Evaluation items • How well did your team play • How well does the police serve your community

Item Writing • Mutually exclusive and collectively exhaustive items • Use positively and negatively phrased questions • Avoid colloquialism, expressions and jargon • Avoid the use of negatives to reverse the wording of an item • Don’t use: I am not satisfied with my job • Use: I hate my job! • Be brief, focused, and clear • Use simple, unbiased questions

Sources of Error • Social desirability • Giving politically correct answers • Response sets • All yes, or all no responses • Acquiescence • Telling you what you want to hear • Personal bias • Wants to send a message

Sources of Error • Response order • Recency - Respondent stops reading once s/he gets to the response s/he likes • Primacy - Remember better the initial choices • Fatigue • Item order • Answers to later items may be affected by earlier items (simple, factual items first) • Respondent may not know how to answer earlier questions

Assessing Instruments • Three issues to consider • Validity: Does the instrument measure what its supposed to measure • Reliability: Does it consistently repeat the same measurement • Practicality: Is this a practical instrument

Types of Validity • Face validity • Does the instrument, on its face, appear to measure what it is supposed to measure • Content validity • Degree to which the content of the items adequately represent the universe of all relevant items under study • Generally arrived at through a panel of experts

Types of Validity • Criterion related • Degree to which the predictor is adequate in capturing the relevant aspects of criterion • Uses Correlation analysis • Concurrent validity • Criterion data is available at the same time as predictor score- requires high correlation between the two • Predictive validity • Criterion is measured after the passage of time • Retrospective look at the validity of the measurement • Known-groups

Types of Validity • Construct Validity • Measures what accounts for the variance • Attempts to identify the underlying constructs • Techniques used: • Correlation of proposed test with other existing tests • Factor analysis • Multi-trait-multimethod analysis • Convergent validity - Calls for high correlation between the different measures of the same construct • Discriminant validity - Calls for low correlation between sub-scales within a construct

Types of Reliability • Stability • Test-retest: Same test is administered twice to the same subjects over a short interval (3 weeks to 6 months) • Look for high correlation between the test and retest • Situational factors must be minimized

Types of Reliability • Equivalence • Degree to which alternative forms of the same measure produce same or similar results • Give parallel forms of the same test to the same group with a short delay to avoid fatigue • Look for high correlation between the scores of the two forms of the test • Inter-rater reliability

Types of Reliability • Internal Consistency • Degree to which instrument items are homogeneous and reflect the same underlying constructs • Split-half testing where the test is split into two halves that contain the same types of questions • Uses Cronbach’s alpha to determine internal consistency. Only one administration of the test is required • Kuder-Richardson (KR20) for items with right and wrong answers

Practicality • Is the survey economical • Cost of producing and administering the survey • Time requirement • Common sense! • Convenience • Adequacy of instructions • Easy to administer • Can the measurement be interpreted by others • Scoring keys • Evidence of validity and reliability • Established norms

Measurement and Scaling