Survey Research

1. Survey Research Frankfort-Nachmias & Nachmias (Chapter 10 – Survey Research, Chapter 11 – Questionnaire Construction) Applications: James H. Kuklinski; Michael D. Cobb; Martin Gilens, “Racial Attitudes and the "New South." The Journal of Politics Vol. 59, No. 2 (May, 1997), pp. 323-349. Canache, Damarys, Mondak, Jeffery J., & Seligson, Mitchell A. (2001). “Meaning and measurement in cross-national research on satisfaction with democracy.” Public Opinion Quarterly, 65, 506-528.

2. Survey Research • Two types of error • Sampling error • Response error

3. Error in Surveys • Survey experts agree – BY FAR the largest amount of error in survey research comes from response error

4. Sources of Response Error • Survey Administration • Question Order • Question Type • Question Wording

5. Survey Administration • There are three typical methods used to administer surveys • Interview • Telephone • Mail

6. Personal/Telephone Interviewing • Advantages • High response rate • Flexibility/control in questioning • Observation (personal interviewing) • Supervision of interviewers (telephone)

7. Personal/Telephone Interviewing • Disadvantages • Expensive • Lack of anonymity • Characteristics of interviewer may introduce bias

11. Mail Surveys • Advantages • Cheap!!! • Often avoid bias due to interviewer characteristics / social desirability • Allows respondent time to answer

12. Mail Surveys • Disadvantages • Low response rates • Cannot probe • Cannot clarify • Cannot be sure who completes questionnaire

17. Dillman’s Total Design Method for Mail Surveys • First Mailing • Typically, when you send a survey out by mail, about 24% of those you sent it to will respond. • Why do we care about the response rate so much???

18. Dillman Method • After you send the first survey, wait about two weeks and then send a follow-up postcard • The postcard should be sent only to non-respondents to save money • Disadvantage: Loss of anonymity

19. Dillman Method • The postcard should say something like, “In the past week or so you received a survey from me about XYZ. I know you are a very busy person, but it is important for me to include your opinions in my research. If you still have the survey, could you please complete it and send it to me in the stamped addressed envelope I included? If it got misplaced, I am happy to send you a replacement survey. Please call or email me at ABC and I will send a replacement to you right away. Thank you so much for helping me with this important project. Sincerely, Jane Doe.” • The postcard follow-up typically increases response rate to about 42%.

20. Dillman Method • About one to two weeks after you send the postcard, send the first replacement questionnaire • Attribute the lack of response to the first survey getting lost in the mail, accidentally misplaced or discarded, or lack of time, etc. Stress the importance of that person’s response to your project. • The first replacement survey will usually get your response rate up to about 59%.

21. Dillman Method • Send a second replacement survey by certified mail about two weeks after the first replacement survey was sent. This will increase the response rate to about 72%

22. Dillman Method • Other pieces of advice to maximize response rate • Keep it simple • Make it look like personal mail • Get a sponsor • Offer inducements

23. The New Frontier in Survey Administration: The Internet • Email surveys • Web surveys

27. Question Order • The order in which you place your questions can cause the error of a response set. • A response set may occur when a series of questions have the same answer choices. This is most likely to happen if questions are on related subjects.

28. Question OrderExample: Response Set (from the GSS) • “Listed below are various areas of government spending. Please indicate whether you would like to see more or less government spending in each area. Remember that if you say ‘much more,’ it might require a tax increase to pay for it. • A. The environment. • 1. SPEND MUCH MORE • 2. SPEND MORE • 3. SPEND SAME • 4. SPEND LESS • 5. SPEND MUCH LESS • 6. CANT CHOOSE

29. Question OrderExample: Response Set (from the GSS) • B. Health Care. • 1. SPEND MUCH MORE • 2. SPEND MORE • 3. SPEND SAME • 4. SPEND LESS • 5. SPEND MUCH LESS • 6. CANT CHOOSE

30. Question OrderExample: Response Set (from the GSS) • C. Education. • 1. SPEND MUCH MORE • 2. SPEND MORE • 3. SPEND SAME • 4. SPEND LESS • 5. SPEND MUCH LESS • 6. CANT CHOOSE

31. Question OrderExample: Response Set (from the GSS) • D. Unemployment Benefits. • 1. SPEND MUCH MORE • 2. SPEND MORE • 3. SPEND SAME • 4. SPEND LESS • 5. SPEND MUCH LESS • 6. CANT CHOOSE

32. Question Order • Etc. • What is the danger in a response set and what can we do about it???

33. Acquiescence Response-set • A type of response-set pattern in which the respondent tends to “agree” with a series of statements due to their desire to please the interviewer (or respond with the “right” answer)

34. Acquiescence Response-set and the California F-scale • Authoritarianism personality – “implicit antidemocratic tendencies and fascist potential” • F-scale: http://www.anesi.com/fscale.htm

35. THE GENERAL ATTITUDE TOWARDS INSTITUTIONAL AUTHORITY SCALE For each of the following 16 opinions, please state your personal reaction by crossing (x) the appropriate answer for each question as: "agree strongly", "uncertain", "disagree strongly", "agree but not strongly", "disagree but not strongly" 1. The Police are pretty trustworthy 2. A Person should obey only those laws that seem reasonable 3. The Army develops initiative 4. It is unreasonable to say that as a rule teachers work in the best interests of their students 5. The Police are quite unfair in their treatment of certain groups of society 6. The law is the embodiment of justice and equality 7. I disagree with what the Army stands for 8. The Police have a hard job which they carry out well 9. Laws are so often made for the benefit of small, selfish groups that one cannot respect the law 10. The Army reduces men to robots 11. Teachers do not respect the individual personalities of the students 12. Policemen like to bully people 13. Teachers are usually ready to take seriously whatever students feel earnest about 14. Obedience to the law constitutes a value indicative of the highest citizenship 15. In this day and age students should not be expected to call a teacher "Sir" 16. People should feel proud to serve in the Army

36. Question Order • The order in which you place your survey questions can cause the error of a saliency effect • Specific mention of an issue in a survey may cause a respondent to associate the issue in connection with a later question.

40. Fox News and Public Opinion on Health Care Reform • “Based on what you know about the health care reform legislation being considered right now, do you favor or oppose the plan?” • Fox – 35% support, 49% oppose • Other polls – 43% support, 45% oppose • Why the difference?

41. Fox News and Public Opinion on Health Care Reform • 3. Do you think Barack Obama's travel and speaking schedule makes him look more like he is a candidate on the campaign trail or more like he is the president of the United States?4. Do you think President Obama apologizes too much to the rest of the world for past U.S. policies?5. Do you think the Obama administration is proposing more government spending than American taxpayers can afford, or not?6. Do you think the size of the national debt is so large it is hurting the future of the country?7. Would you rather: [ROTATE OPTIONS 1 and 2]Cut spending now so future generations don't have to payKeep spending at current levels and let future generations pay20. When Barack Obama was a candidate campaigning for the presidency, he spoke of the urgent need to finish the fight in Afghanistan, which he called the central front on the war on terrorism. Do you think that, as president, Obama is doing what it takes to win in Afghanistan?

42. Question Order • Consistency is another problem we might encounter due to question order. • Respondents may strive to appear consistent in their answers: an answer to a question may be constrained by an answer given earlier.

43. Question OrderExample: Consistency • “And what about the federal government, does it have too much power or too little power?” • FAR TOO MUCH • TOO MUCH • RIGHT AMNT OF POWER • TOO LITTLE POWER • FAR TOO LITTLE POWER • CANT CHOOSE • What if this question preceded the government spending questions?

44. Question Order • A final concern about question order is fatigue • Respondents may give perfunctory answers to questions late in the survey especially if it is long • In addition, response sets are often a problem in lengthy surveys toward the end

45. Question Order • Some final thoughts on question order • The first few questions should “break the ice” – don’t jump in with a big question. • Move from general questions to more specific questions. • Questions on personal or sensitive topics (such as income, race, age) should be asked at the end with a big disclaimer about why you are asking those questions

46. Question Type • Closed-ended questions are ones that have response options provided by the researcher. • What are some of the advantages and disadvantages to closed-ended questions?

47. Question Type • Advantages of Closed-Ended: • Easy to answer • Answers are easy to compare • Respondents willing to provide more personal information • May help clarify the question

48. Question Type • Disadvantages of Closed-Ended: • Forces people to accept the categories or puts too many responses into the “other” category. • Choices listed may communicate the kind of response wanted.

49. Question Type • Open-ended questions allow the respondent to write in an answer. • What are some of the advantages and disadvantages to open-ended questions?

50. Question Type • Advantages of Open-ended: • May better indicate respondents thoughts/beliefs/opinions. • Better when the list of responses might be excessive. • Might get unanticipated answers.