Research Methods

Research Methods Survey Research

Survey Research • Survey research is one of the most important areas of measurement in applied social research. • The broad area of survey research encompasses any measurement procedures that involve asking questions of respondents. • A "survey" can be anything from a short paper-and-pencil feedback form to an intensive one-on-one in-depth interview.

Roadmap • We'll begin by looking at the different types of surveys that are possible. These are roughly divided into two broad areas: Questionnaires and Interviews. • Next, we'll look at how you select the survey method that is best for your situation. • Once you've selected the survey method, you have to construct the survey itself. Here, we will be address a number of issues including: the different types of questions; decisions about question content; decisions about question wording; decisions about response format; and, question placement and sequence in your instrument. • We turn next to some of the special issues involved in administering a personal interview. Finally, we'll consider some of the advantages and disadvantages of survey methods.

Questionnaires vs. Interviews • Surveys can be divided into two broad categories: the questionnaire and the interview. • Questionnaires are usually paper-and-pencil instruments that the respondent completes. • Interviews are completed by the interviewer based on the respondent says. • Sometimes, it's hard to tell the difference between a questionnaire and an interview. • For instance, some people think that questionnaires always ask short closed-ended questions while interviews always ask broad open-ended ones. • But you will see questionnaires with open-ended questions (although they do tend to be shorter than in interviews) and there will often be a series of closed-ended questions asked in an interview.

Questionnaire • Mail Survey • + Wide Coverage • + Inexpensive • - Response rates low • - biased samples • Group-Centered Questionnaire: A sample of respondents is brought together and asked to respond to a structured sequence of questions. • + questions can be explained • + response rate high • - Narrow Coverage

Questionnaire • Household drop-off s: In this approach, a researcher goes to the respondent's home or business and hands the respondent the instrument. In some cases, the respondent is asked to mail it back or the interview returns to pick it up. • Like the mail survey, the respondent can work on the instrument in private, when it's convenient. • Like the group administered questionnaire, the interviewer makes personal contact with the respondent -- they don't just send an impersonal survey instrument. • And, the respondent can ask questions about the study and get clarification on what is to be done. Generally, this would be expected to increase the percent of people who are willing to respond.

Interviews • Interviews are a far more personal form of research than questionnaires. In the personal interview, the interviewer works directly with the respondent. Unlike with mail surveys, the interviewer has the opportunity to probe or ask follow-up questions. • General types include “face-to-face” and “elite” interviews • Interviews can be very time consuming and they are resource intensive. • The interviewer is considered a part of the measurement instrument and interviewers have to be well trained in how to respond to any contingency.

Interviews • Telephone interviews enable a researcher to gather information rapidly. Most of the major public opinion polls that are reported were based on telephone interviews. • Like personal interviews, they allow for some personal contact between the interviewer and the respondent. • And, they allow the interviewer to ask follow-up questions. • But they also have some major disadvantages: • Many people don't have publicly-listed telephone numbers. • Some don't have telephones. • People often don't like the intrusion of a call to their homes. • And, telephone interviews have to be relatively short or people will feel imposed upon.

Selecting the Survey Method Checklist of Selection Issues: Population & Accessibility • Can the population be enumerated? • Is the population literate? • Are there language issues? • Will the population cooperate? • What are the geographic restrictions?

Selecting the Survey Method Sampling Issues • What data is available? • Can respondents be found? • Who is the respondent? • Can all members of population be sampled? • Are response rates likely to be a problem?

Selecting the Survey Method Question Issues • What types of questions can be asked? • How complex will the questions be? • Will screening questions be needed? • Can question sequence be controlled? • Will lengthy questions be asked? • Will long response scales be used?

Selecting the Survey Method Content Issues • Can the respondents be expected to know about the issue? • Will respondent need to consult records? Bias Issues • Can social desirability be avoided? • Can interviewer distortion and subversion be controlled? • Can false respondents be avoided?

Selecting the Survey Method Administrative Issues • Costs • Facilities • Time • Personnel / Training

Comparative Evaluation of Survey Methods

Conducting the Survey • Constructing a survey instrument is an art in itself. There are numerous small decisions that must be made -- about content, wording, format, placement -- that can have important consequences for your entire study. • While there's no one perfect way to accomplish this job, we do have lots of advice to offer that might increase your chances of developing a better final product. • There are three areas involved in writing a question: • determining the question content, scope and purpose • choosing the response format that you use for collecting information from the respondent • figuring out how to word the question to get at the issue of interest

When a question has two possible responses, we consider it dichotomous. Surveys often use dichotomous questions that ask for a Yes/No, True/False or Agree/Disagree response. There are a variety of ways to lay these questions out on a questionnaire: Dichotomous Questions

We can also classify questions in terms of their level of measurement. For instance, we might measure occupation using a nominal question. Here, the number next to each response has no meaning except as a placeholder for that response. The choice of a "2" for a lawyer and a "1" for a truck driver is arbitrary -- from the numbering system used we can't infer that a lawyer is "twice" something that a truck driver is. Nominal Questions

We might ask respondents to rank order their preferences for presidential candidates using an ordinal question. We want the respondent to put a 1, 2, 3 or 4 next to the candidate, where 1 is the respondent's first choice. Note that this could get confusing. We might want to state the prompt more explicitly so the respondent knows we want a number from one to 4 (the respondent might check their favorite candidate, or assign higher numbers to candidates they prefer more instead of understanding that we want rank ordering). Ordinal Questions

We can also construct survey questions that attempt to measure on an interval level. One of the most common of these types is the traditional 1-to-5 rating (or 1-to-7, or 1-to-9, etc.). his is sometimes referred to as a Likert response scale. Here, we see how we might ask an opinion question on a 1-to-5 bipolar scale (it's called bipolar because there is a neutral point and the two ends of the scale are at opposite positions of the opinion): Interval Questions

Finally, we can also get at interval measures by using what is called a cumulative or Guttman scale Here, the respondent checks each item with which they agree. The items themselves are constructed so that they are cumulative -- if you agree to one, you probably agree to all of the ones above it in the list: Cumulative Questions

Sometimes you have to ask the respondent one question in order to determine if they are qualified or experienced enough to answer a subsequent one. T his requires using a filter or contingency question. For instance, you may want to ask one question if the respondent has ever smoked marijuana and a different question if they have not. in this case, you would have to construct a filter question to determine whether they've ever smoked marijuana: Contingency Questions

Contingency Questions • Filter questions can get very complex. Sometimes, you have to have multiple filter questions in order to direct your respondents to the correct subsequent questions. There are a few conventions you should keep in mind when using filters: • try to avoid having more than three levels (two jumps) for any question • Too many jumps will confuse the respondent and may discourage them from continuing with the survey. • if only two levels, use graphic to jump (e.g., arrow and box) • The example above shows how you can make effective use of an arrow and box to help direct the respondent to the correct subsequent question. • if possible, jump to a new page

Open vs. Closed • Open-ended Questions “What is the most important problem facing the nation today?” • Closed-ended Questions “Do you feel you are asked to pay more than you should in federal income taxes, about the right amount, or less than you should?”

Survey Designs • Cross-sectional Surveys • Longitudinal Surveys - Trend Studies - Cohort Studies - Panel Studies

Question Wording • One of the major difficulty in writing good survey questions is getting the wording right. Even slight wording differences can confuse the respondent or lead to incorrect interpretations of the question. Here, I outline some questions you can ask about how you worded each of your survey questions.

Questions a/b Question Wording • Can the Question be Misunderstood? • The survey author has to always be on the lookout for questions that could be misunderstood or confusing. • For instance, if you ask a person for their nationality, it might not be clear what you want (Do you want someone from Malaysia to say Malaysian, Asian, or Pacific Islander?). Or, if you ask for marital status, do you want someone to say simply that they are either married or no married? Or, do you want more detail (like divorced, widow/widower, etc.)? • Some terms are just to vague to be useful. For instance, if you ask a question about the "mass media," what do you mean? The newspapers? Radio? Television?

Misunderstandings • Here's one of my favorites. Let's say you want to know the following: What kind of headache remedy do you use? • Do you want to know what brand name medicine they take? Do you want to know about "home" remedies? Are you asking whether they prefer a pill, capsule or caplet?

Assumptions What Assumptions Does the Question Make? • Sometimes we don't stop to consider how a question will appear from the respondent's point-of-view. • We don't think about the assumptions behind our questions. • For instance, if you ask what social class someone's in, you assume that they know what social class is and that they think of themselves as being in one. • In this kind of case, you may need to use a filter question first to determine whether either of these assumptions is true.

Time • Is the time frame specified? • Whenever you use the words "will", "could", "might", or "may" in a question, you might suspect that the question asks a time-related question. • Be sure that, if it does, you have specified the time frame precisely. For instance, you might ask: Do you think Congress will cut taxes? or something like Do you think Congress could successfully resist tax cuts? • Neither of these questions specifies a time frame.

Personal Wording • How personal is the wording? • With a change of just a few words, a question can go from being relatively impersonal to probing into your private perspectives. • Consider the following three questions, each of which asks about the respondent's satisfaction with working conditions: • Are working conditions satisfactory or not satisfactory in the plant where you work? • Do you feel that working conditions satisfactory or not satisfactory in the plant where you work? • Are you personally satisfied with working conditions in the plant where you work? • The first question is stated from a fairly detached, objective viewpoint. The second asks how you "feel." The last asks whether you are "personally satisfied." Be sure the questions in your survey are at an appropriate level for your context. And, be sure there is consistency in this across questions in your survey.

Etc. Other Wording Issues • The nuances of language guarantee that the task of the question writer will be endlessly complex. Without trying to generate an exhaustive list, here are a few other questions to keep in mind: • Does the question contain difficult or unclear terminology? • Does the question make each alternative explicit? • Is the wording objectionable? • Is the wording loaded or slanted?

Question Placement Decisions About Placement • One of the most difficult tasks facing the survey designer involves the ordering of questions. Which topics should be introduced early in the survey, and which later? If you leave your most important questions until the end, you may find that your respondents are too tired to give them the kind of attention you would like. If you introduce them too early, they may not yet be ready to address the topic, especially if it is a difficult or disturbing one. • There are no easy answers to these problems - you have to use your judgment. Whenever you think about question placement, consider the following questions: • Is the answer influenced by prior questions? • Does question come too early or too late to arouse interest? • Does the question receive sufficient attention?

Placement A Checklist of Considerations • There are lots of conventions or rules-of-thumb in the survey design business. Here's a checklist of some of the most important items. You can use this checklist to review your instrument: • start with easy, nonthreatening questions • put more difficult, threatening questions near end • never start a mail survey with an open-ended question • for historical demographics, follow chronological order • ask about one topic at a time • when switching topics, use a transition • reduce response set (the tendency of respondent to just keep checking the same response) • for filter or contingency questions, make a flowchart

Bad Questions 1 “Suppose the budgets of your state and local governments have to be curtailed, which of these parts would you limit most severely?” (ACIR 1982) • Aid to the Needy 8% • Public Welfare 39%

Bad Questions 2 • “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” (Roper 1994a) 22% Possible, 12% Not sure • “Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?” (Roper 1994b) 1% Possible

Bad Questions 3 • “As you may know, in 1974, Jerry Springer, who had gotten married six months earlier, was arrested on a morals charge with three women in a hotel room. He also used a bad check to pay for the women’s services, and subsequently resigned as mayor of his city. Does this make you feel much more likely, somewhat more likely, somewhat less likely, or much less likely to support Jerry Springer for governor this year?” (Cambridge Survey Research 1974)

Bad Questions 4 • “Does your employer or his representative resort to trickery in order to defraud you of a part of your earnings?” • “If you are paid piece rates, is the quality of the article a pretext for fraudulent deductions from your wages?” • Marx 1880: 208

Question Wording & Response Set • Do you approve or disapprove of the job President Clinton is doing?” (Morin 1993) Approve 53%, Disapprove 38% • “Do you strongly approve, somewhat approve, somewhat disapprove, strongly disapprove of the job President Clinton is doing?” Approve (strongly and somewhat) 62%, Disapprove (strongly and somewhat) 32%

Question Wording and Order 1 • “Would you say that traffic contributes more or less to air pollution than industry?” (Wanke et al. 1995) Traffic 45%; Industry 32% • “Would you say that industry contributes more or less to air pollution than traffic?” Industry 57%; Traffic 24%

Question Order 2a • “Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?” • “Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?” • 36% Yes (Hyman and Sheatsley 1950)

Question Order 2B • “Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?” • “Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?” • 73% Yes

Response Validity:Over- & Under-reporting Behavior Face-to-Face Phone Self-Administered Low Threat/Normative Registered to Vote +15 +17 +12 Voted in Primary +39 +31 +36 Have own library card +19 +21 +18 High Threat Bankruptcy -32 -29 -32 Drunk driving -47 -46 -54 (Bradburn and Sudman 1980; Neuman 2006)

Research Methods