Chapter 5.

1. Chapter 5. The Research Consumer Reviews the Measures

2. Data Collection and the Research Consumer • Evaluation researchers collect information on: • Study participants (who they are and how the program effects them) • Program implementation and “processes” • Effects of interventions and programs (outcomes, impact, and costs) • Consumers should ask: Does evidence about program effectiveness come from appropriate and valid measures? • Consumers should understand the characteristics of the data collected (e.g., “Yes I read magazines” versus “I read 4 magazines on average each month”). • Evaluation information comes from surveys, observations, tests, physical exams, and reviews of printed, oral, and electronic reports and databases.

3. Surveys Self-administered surveys ask participants to answer questions or respond to items in writing (“paper-and-pencil”) or online. ADVANTAGE • Many people are accustomed to completing surveys regardless of where or how they are administered. • Familiarity with a data collection process saves staff time.

4. Surveys (Continued) DISADVANTAGES • If asked to recall events (e.g., “In the past four weeks, how often did you….?), participants may not remember. • The people who respond to surveys may be the ones who feel most strongly (really pleased or really angry). • The self-administered survey’s format is not suitable (and not really designed) for obtaining explanations of behavior or sensitive information. • Without supervision, some participants may fail to answer some or even all questions because they do not understand them. • Some people may have difficulty completing surveys especially if the reading level is too high for them, they are not interested, or they are too busy or too ill.

5. Multiple-Choice Achievement Tests • Achievement tests are the most commonly used method of collecting information on educational accomplishment. • Tests are used to measure knowledge, understanding, and application of theories, principles, and facts. Advantages • Almost all students are used to multiple-choice achievement tests and have similar expectations of them regardless of their age or other demographic characteristics. • Many standardized tests are available enabling researchers to compare findings across students who have differing backgrounds and experience. • Tests are the often the most efficient method of measuring the knowledge that is supposed to result from a particular lesson or course of instruction. • Valid multiple-choice tests are relatively easy to score and interpret.

6. Multiple-Choice Achievement Tests (Continued) Disadvantages • Multiple-choice tests may not be the most appropriate method to measure “high” levels of knowledge or for assessing attitudes and values. • Each test question must be carefully studied to ensure that it measures the concept it is supposed to measure. Many questions do not meet this standard.

7. Record Reviews Record reviews are analyses of documented behavior. • Documentation may be in print, online, on or audio or video • Records come in two forms. • The first type consists of existing documents (e.g., participants’ medical, school, employment record). • The second is developed specifically for a given program (e.g., a food diary that is kept especially for a nutrition program).

8. Record Reviews (Continued) Advantages • Obtaining data from existing records can be relatively unobtrusive in that study participants’ daily activities need not be disturbed. • Records are a relatively reliable storehouse of actual practice or behavior. • If data are needed on many demographic characteristics (e.g., age, sex, insurance status), records are often the best source. • Data obtained from records like medical and school records may reduce participants’ research burden.

9. Record Reviews (Continued) Disadvantages • Finding information in records is often time-consuming. • The record review process must be demonstrably reliable and accurate. This may mean extensive training staff and monitoring the quality of the review. • Certain types of information are rarely recorded (e.g., functional or mental status; time spent with clients “after hours”). • Records do not provide data on the appropriateness of a practice or on the relationship between what was done by the practitioner (the process of care) and results (the outcomes of care).

10. Observations Observations produce descriptions such as: • the size of a classroom or the number, types, and dates of magazines in the office waiting room • a portrait of the dynamics of a situation (e.g., a meeting between parents and teachers or children at play) • Observations take three basic forms. • Participant observation in which the researcher actually becomes part of the community being observed • In-person observation • Unobtrusive observations (e.g. placing a camera in a room or clinic or yard)

11. Observations (Continued) Advantages • Observations provide an opportunity to collect firsthand information. • Observations can provide information that cannot be anticipated because the evaluator is present when the unforeseen occurs. Disadvantages • A very structured format and extensive training are required for dependable observations. • Observations are labor-intensive and time-consuming. • The observer (camera or human) can influence the environment being studied.

12. Interviews Interviews are conducted in person (with our without the assistance of a computer) and on the telephone and using electronic methods Advantages • Interviews allow the researcher to ask about the meaning of questions. • Interviews can be useful in collecting information from people who may have difficulty reading or seeing. Disadvantages • Time-consuming and labor intensive. • Require extensive training and monitoring if they are to elicit accurate information in a timely manner. • Special skills may be required to interpret responses that are off the record.

13. Databases and Data Sets Governments, statisticians, and researchers compile data into databases to keep track of individuals and communities so as to describe and monitor health, education, and need for and use of social services. • Researchers typically use large databases and data sets to help program planners understand the extent of need for public programs and monitor progress and outcomes over time.

14. Databases and Datasets (Continued) Advantages • Using existing data can be economical. Often a major cost in research is collecting new data. Disadvantages • The selection of data to collect, the choice of persons from whom to collect it, the quality of the data, and how data were recorded are all predetermined.

15. Vignettes (Hypothetical Scenarios) A vignette is a short scenario that is used in collecting data in “what if” situations. Advantages • Vignettes can be fun for participants to complete. • Vignettes can be efficient. They enable the researcher to vary important factors (e.g. age, gender) one at a time across vignettes. • Not every participant has to review a scenario with every factor as long as all participants review some factors and all factors are reviewed. Disadvantages • Producing vignettes requires technical and artistic (writing) skill • Researcher cannot be certain that the responses accurately reflect the participant’s true feelings or behavior. • Sampling can get complicated.

16. Response Choices • Response choices are variously referred to as rating or measurement scales or levels of measurement. • There are three basic measurement levels. • Categorical responses. Categorical responses force respondents to put answers into categories such as:. Are you male or female? • When only two choices are possible (e.g., male and female), the results are termed dichotomous. • Ordinal Responses. Questions using ordinal response options offer responses with a built-in order (e.g., poor, fair, good, very good, excellent) . • Numerical Responses: Continuous and Discrete • Continuous responses can have an infinite number of values (e.g., weight). Numerical data may be discrete, and ask for an actual number in response (number of calories in one cookie).

17. Questions Asked by Evaluators when Selecting Measures • Which variables are to be measured? • Can I borrow or adapt a currently available measure for each variable or must one or more measures be created? • If appropriate measures are available, do I have the technical and financial resources to acquire and administer them? • If none is available, do I have the technical skills, financial resources, and time to create them? • Are participants likely to be able to fill out forms, answer questions, and provide information etc. called for by the measure

18. Questions Asked by Evaluators when Selecting Measures • In studies that involve direct services and use of information from medical, school, prison and other confidential records, can I obtain permission to collect data in an ethical way? • To what extent will users of the evaluation’s results (e.g., practitioners, students, patients, program developers, policy-makers and sponsors) have confidence in the measures?