Chapter 8

1. Chapter 8 Measuring Concepts

2. Learning Objectives • Explain the processes of measuring concepts and collecting data • Distinguish between different types of measurement techniques • Measure concepts and collect data using different data collection techniques

3. Measurement Instrument • Tool used to measure the concepts of the study • Tool used to collect the data to be analyzed • Design of the instrument is driven by the research question • Types • survey, questionnaire, diagnosis, lab test, etc.

4. Process to Create the Instrument • Conceptualization – define the target population and concepts necessary to answer the research question • Construction – develop the individual measures • Operationalization – translate perceived concepts into measurable constructs • Pre-test– “dress rehearsal” of the final instrument to identify problem areas • Administration – use the instrument to collect the data

5. Operationalization • Example hypothesis: “Eating fruits and vegetables provides adequate fiber intake that can help lower blood cholesterol levels among males” • Need an appropriate and measurable definition of adequate intake • Ways to measure intake of fruits and vegetables • How much? How many? What type? How often? • Should food be weighed or measured? Should servings at different meals be counted separately? Are self reports valid? Are food diaries reliable? Is a prescribed diet feasible?

6. Levels of Measurement • The way that variables or concepts are measured determines the type of analysis that is possible • The “levels” of measurement can be arranged from least to most precise, least to most complex, and most to least restricted as far as analysis options

7. Categorical or Nominal Measures • Least precise and complex • Most limited in analysis options • Measurement options are categories that cannot be meaningfully ordered • For example, marital status and eye color • Dichotomous measures are categorical measures with only two categories • For example, yes/no; male/female

8. Ordinal Measures • More precise and complex • Still limited in terms of analysis • Categorical measures for which categories have a meaningful order • For example, low/medium/high; strongly disagree/disagree/neutral/agree/strongly agree • However, they are still categories with no inherently meaningful numeric value (although one can be assigned)

9. Continuous Measures • Most precise and complex • Least limited in terms of analytic options • Measure options have inherently meaningful numeric values • For example, temperature in Fahrenheit; weight in pounds; systolic blood pressure • Two general types • Interval – meaningful quantitative distance between measure options (e.g., temperature) • Ratio – meaningful intervals plus fixed origin or zero point (e.g., weight in pounds) • Can conclude that one category is X times greater than another

10. Validity and Reliability of Measures • Validity – the variable measures what it is intended to measure; validity is truth • Reliability – the variable consistently measures the same construct; reliability is consistency Invalid and Unreliable Invalid but Reliable Valid and Reliable

11. Evaluating the Validity of Measures • Is the measure measuring what we intend it to measure? • Minimum and somewhat subjective criteria: • Face validity - If the measure or question “appears” to measure the correct concept, then it is said to have good “face” validity • On the face of it, it appears to be appropriate

12. Really Evaluating the Validity of Measures • Predictive validity – the ability of the measure to predict something it should, theoretically, be able to predict (i.e., a measure of depression should predict suicide ideation) • Concurrent validity – ability of the measure to distinguish between groups that it theoretically should be able to distinguish between (i.e., a measure of depression should distinguish depressed persons from anxious persons)

13. Validity (cont.) • Convergent validity- degree to which the measure is similar to other measures it theoretically should be similar (i.e., a measure of depression should be able to identify depressed persons as well as the DSM-IV criteria diagnoses depressed persons) • Discriminant validity - degree to which the measure is dissimilar to other measures to which it theoretically should not be similar(i.e., a measure of depression should be substantively different from a measure of anxiety)

14. Evaluating the Reliability of Measures • Inter-rater reliability – degree to which different raters give consistent estimates of the same phenomena (i.e., two trained interviewers should be able to identify the same subject as being depressed) • Test-retest reliability – degree to which a measure performs consistently from one time to another (i.e., the same instrument should identify the same subject as being depressed over two administrations separated by short period of time)

15. Reliability (cont.) • Parallel-forms reliability – degree to which similar measures of the same construct provide consistent results (i.e., two similar measures of depression—perhaps with different wording or response options--should both identify depressed subjects) • Internal consistency – degree to which responses are consistent across items representing components of an overall construct (i.e., components of the measure of depression such as changes in appetite and lack of interest in previous activities should elicit consistent responses)

16. Chronbach’s alpha • Quantitative method to evaluate reliability • Measure represents how often ratings between interviewers or across measurement periods match • Measure that is completely reliable will have a coefficient of 1.0 (ratings always agree), and one that is totally unreliable will have a coefficient of 0.0 (ratings never agree) • Typically, a coefficient of 0.75 or higher (75 percent agreement) is considered acceptable

17. Content of the Measures • How well does the question address what we are trying to measure? • Is the question necessary/useful? • Amount of detail needed • Do other items get at it • Is more than one question needed to measure this? • Two parts • Context • Intensity • Guttman scale - multiple items added together • Do respondents have the necessary information?

18. Pilot Test Measures • Use the measures on a small group of people with characteristics similar to the study population • Evaluate the validity and reliability • Conduct focus group with the pilot group to determine how well the measures worked • What did the respondents think a question meant to ask about? • How do the respondents define key terms? • Was the wording confusing in any way? • Were all relevant response options included? • Additionally, cognitive tests can be implemented to explore in detail the perceived meaning and interpretation of survey questions.

19. Survey Data Collection • One of many types of data collection methods • Participants provide data in response to written/verbal questions • Provides quantitative data – data has numeric values, inherently or by assignment • Two general methods • Questionnaire • Paper-pencil • Computer • Interview • Personal • Telephone/IM

20. Advantages and Disadvantages of Survey Methods • Advantage – can collect a great deal of data at relatively little cost in time and resources • Disadvantages – limited control over response and completion rates (especially if the survey is self-administered)

21. Survey Measures • Efficient and often effective vehicle for measuring perceptions, opinions, and ideas • Measuring behaviors may be less effective • Prevarication bias – survey respondents are not completely honest about their behaviors • Especially an issue for illegal or socially undesirable behaviors • Verify if possible (e.g., drug tests, court records) • Maximize the perception of confidentiality • Self-administered are better for this than interviewer-administered surveys • Computer-assisted self-administered is the best method for this reason

22. General Types of Survey Measures • Structured - predetermined range of responses that the respondent can select • Advantage – precision • Disadvantage – may miss important components of the measure How confident are you to exercise when your exercise partner decides not to exercise that day?

23. Types of Survey Measures (cont.) • Semi-structured - include questions with response options but leave some or most of the questions open-ended so that respondents can answer in their own words, reflecting their own conceptualizations of their responses • Appropriate for concepts about which little is known Based on your lifestyle intervention classes, what are the 2 major benefits of participating in regular physical activity? 1) __________2)_____________

24. Survey Construction • Three important aspects of survey instruments are: 1) types of questions 2) question wording 3) question placement

25. Types of Questions • Closed-ended – structured response options • Must be mutually exclusive – no overlap between category choices • 0, 1-5, 5-10, 10-15, … are not mutually exclusive • Must be exhaustive – all possible choices should be included • Often satisfied by including “Other” category • Open-ended – respondents not give a choice of options; left to “fill in the blank”

26. Types of Questions (cont.) • Level of Measurement – response options • Dichotomous • Categorical • Ordinal • Example, Likert Scale – range from one end to the other continuum, usually with a neutral category in the middle (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree) • Continuous • Interval • Ratio

27. Types of Questions (cont.) • Serve a function • Filter or contingency – allow the respondent to skip a group of questions that are not relevant for them Have you ever used marijuana? No Yes If yes, about how many times? Once 2 to 5 times 6 to 10 times More than 10 times

28. Question Wording • Is the question specific enough? • Liked something very well - what does that really mean? • Is the question general enough? • Time frame too limited • Is the question biased or loaded? • Ask about both sides of an issue • Will respondents answer truthfully? • Group responses • Hypothetical projective respondent

29. Wordings to be Avoided • Pejorative terms (i.e., racial slurs) • Judgments (i.e., expressions of opinion) • Insensitive wording about sensitive topics (e.g., how wealthy are you?)

30. Level of Wording • Level of language (i.e., reading level) used should be appropriate for the audience • If not certain that respondents would clearly understand terms such as cholesterol or tumor, they should be defined clearly • If the survey is self-administered, the lowest appropriate reading level (e.g., sixth grade level) should be used

31. Placement of Questions • Logic in ordering the questions • First group of questions should be easy to answer, interesting and non-threatening • Sensitive questions should be embedded in the middle • Demographic questions should be at the end of the survey • Questions vital to the purpose of the study (i.e., dimensions of the exposure) should NOT be placed at the end of the survey • A clear numbering system for questions helps respondents navigate correctly through all the questions

32. Grouping of Questions • Group questions by theme or topic • Insert transition sentences between topics • Introduce the new topic • Opportunity to define terms and concepts • Indicate any specific information about the topic (i.e., remind respondents that they can skip sections or questions; indicate limitations about responses – only answer questions about alcohol if 21 or older)

33. Grouping of Questions (cont.) • If possible, group questions with the same response options (e.g., strongly agree to strongly disagree) • When possible, keep time frames and time groupings consistent • In the past year, the past month, the past week • Once a week, 2-4 times per week, 5-6 times per week, …

34. Scales • Guttman scales – combining (adding) measures that represent various components of a concept to form a larger and more comprehensive measure • Likert scales – response options range from one end to the other of a continuum; continuum is conceptual rather than numeric • Rating scales – items are rated on a numeric scale in terms of priority, interest, etc. • Semantic differentials scales – visual numeric scale (usually 7-point) for respondents to choose the “degree” of an attitude or belief from low to high or high to low

35. Clinical Data Collection • Important components • Data collection schedule – timing of the measurements • Collection forms – instruments, types of measurements

36. Example Clinical Data Collection • Clinical trial to reduce blood glucose and lipid profiles X= denotes data collection times

37. Secondary Data • Analyzing measures used and data collected by other investigators • Data that has been made available to the public • For example, National Health and Nutrition Examination Survey (NHANES) • Ideally, documentation is included with the data to describe the details of the data collection and measurement • Sometimes, the documentation is absent or incomplete – important to try to get as much information as possible about the data before analysis is initiated

38. Advantages and Disadvantages of Using Secondary Data • Advantages • Major time and cost saving • Often large and complex data sets • Nationally representative • Multiple waves of data collection • Complex sampling designs • Measures have been verified • Disadvantages • Data may not have been designed to answer the particular research question • Secondary researcher may not have a good understanding of the details of the data

39. Caution about Using Secondary Data • Remember to acknowledge the original investigators • May have to adapt the research question to what is and is not available in the data set • Do not overreach in the interpretation of results – they are limited by what they are • Be careful not to replicate much or any of the analyses conducted by the original investigators