Survey Design and Implementation

1. Survey Design and Implementation Elizabeth C Wells Doctoral Candidate Ag & Extension Education http://www.msu.edu/~wellselz/

2. Why do people fill out surveys? What do you want to ask/measure? Accuracy/Survey Error Random Sampling Writing Questions Questionnaire Design Survey Implementation Telephone, Interview and Internet Surveys UCRIHS

3. Why People Fill Out Surveys • Social Exchange Theory Theory of human behavior in which actions of individuals are motivated by the return these actions are expected to bring. Elements of reward, cost and trust

4. Social Exchange Theory • Rewards show positive regard say thank you ask for advice support group values give tangible rewards make the questionnaire interesting

5. Social Exchange Theory • Reduce Social Costs avoid subordinating language avoid inconvenience avoid embarrassment make it short and easy minimize requests for personal information keep requests consistent

6. Social Exchange Theory • Establish Trust provide token of appreciation in advance sponsorship by legitimate authority make task appear important invoke other exchange

7. Constructs and Measurement • What do you want to know? • What are the variables? • How will you measure it? behavior, attitudes, feelings, demographics

8. Survey Error • Sampling Error – Who are you sampling? • Coverage Error – Does your list include everyone? • Measurement Error – Does everyone answer a question the same way? • Non-response Error – Why did respondent not answer: Instrument (whole questionnaire not returned) Item (question not answered)

9. Non Response Error • Only an issue if non responders would answer differently and change results compare early to late respondents double dip

10. Random Sampling • How much sampling error can you tolerate? • How large is the population? • How varied is the population? • How confident do you wish to be about estimates made from the sample?

11. Random Sampling - Chart Pop Size 50/50 split 80/20 split • 80 71 • 132 111 • 196 153 • 234 175 • 260 188 1000 278 198 Sample size for a 95% confidence level And a + 5% sampling error

12. Random Sampling • http://www.randomizer.org A www site that will generate random numbers which you can use for random selection of participants on a mailing list.

13. Writing Questions • Everyone will interpret the same way • Respondents are able to respond accurately • Respondents are willing to answer

14. Writing Questions • Use simple words • Keep it short • Be specific • Do not talk down • Avoid bias • Avoid hypothetical questions • Use complete sentences

15. Question Criteria • Does the question require an answer? If you fixed dinner last night, did you eat meat as part of that meal? • To what extent does the respondent have a ready made and accurate answer? Are tall people more likely to be elected President? • Can respondent accurately recall and report past behavior? How much television did you watch last month?

16. Question Principles • Use simple words and phrases tired……………………………exhausted honest…………………………candid work……………………………employment most important………………..top priority free time……………………….leisure doctor………………………….physician your answers………………….your responses to this questionnaire

17. Question Principles • Avoid vague quantifiers How often did you attend religious services last year?  Never  Rarely  Occasionally  Regularly

18. Question Principles How often did you attend religious services during the past year?  Not at all  A few times  About once a month  Two or three times a month  About once a week  More than once a week

19. Question Principles • Avoid bias by using both positive and negative sides in the question stem “To what extent do you agree….” vs “To what extent do you agree or disagree with this statement…. “

20. Question Principles • Avoid double-barreled questions 16. Should the new school have a swimming pool that includes lanes for swimming laps and is not enclosed for winter use?

21. Question Principles • Avoid asking respondent to say yes in order to mean no. 5. Do you favor or oppose not allowing the state to approve a casino without approval from 60% of the voters?  Favor  Oppose

22. 5. Do you favor or oppose requiring 60% approval by voters in order for a casino to be built.  Favor requiring 60% approval  Oppose requiring 60% approval

23. Question Principles • Use equal numbers of positive and negative categories for scalar questions AND distinguish undecided from neutral by placing at the end of the scale.

24. To what extent do you agree or disagree with this statement: “Listening to classical music is good for a persons emotional health.”  Strongly agree  Somewhat agree  Neither agree nor disagree  Somewhat disagree  Strongly disagree  No opinion

25. Question Principles • Soften impact of potentially objectionable questions: Have you ever shoplifted something from a store? vs Have you ever taken something from a store without paying for it?

26. 22. What was your total income from all sources in 2003? ______________ Total income for 2003 22. Which category best describes your total income from all sources in 2003?  $10,000. or less  $10,001. to $20,000.  $20,001. to $35,000.  $35,001. to $50,000.  $50,001. or above

27. Question Principles • Don’t combine numbers and words Doing so may impact respondents perception of the scale. 1= strongly agree, 2 = agree, 3 = disagree, 4 = strongly disagree

28. Question Principles • Eliminate check-all-that-apply questions to reduce primacy and sufficiency effect. People tend to check the first ones more than the last ones and sometimes check a few and feel that is enough to satisfy the researcher.

29. Extension Agent Other Cattlemen Fertilizer and other salesmen 59% selected Extension Agent Other Cattlemen Fertilizer and other salesmen Extension Agent 41% selected Extension Agent Question Principles

30. Questionnaire Design • Paper and layout choice booklet format preferred one column for legal size two columns for full size avoid upper left staple design avoid unusual size, shape or fold

31. Order of Questions • A questionnaire is like a conversation. It evolves according to social norms. Constantly switching topics makes it appear as though the researcher is not listening to the respondents answers. • Doing otherwise results in answers that are less thought out when you want them to focus!

32. Order of Questions • What was your family income in 2003? • Do you like to play golf? • What is your opinion on Global Warming? • Are you married? • How often have you gone fishing this year? • How adequate is your current health care? • How old are you? • What is your political party preference? • What is your occupation?

33. Question Order • Group alike questions together • Begin asking questions in the upper left corner • Ask one question at a time • Use item-in-a-series format

34. The First Question • Should clearly apply to everyone • Question should be easy • Question needs to be interesting • Connectedness between the question and the purpose of the survey.

35. Unacceptable First Questions 1. Please think about the things that make MSU Extension a pleasant place to work and write down the five most important aspects and rate them one to five in importance to you. .

36. Unacceptable First Questions 1. Please describe in your own words what you consider good about working for MSU Extension. 1. What year were you born? _________year born

37. Good First Question 1. Thinking about MSU Extension, how would you rate it as a place to work?  Excellent  Good  Fair  Poor

38. Graphic Design • Stimuli on page consists of two languages: written words graphic symbols and arrangements

39. Graphic Design • Desired navigational path • Create visual navigational guides • Develop additional guides for redirecting during skip patterns • Be consistent through out your instrument

40. Questionnaire Design • Instructions need to be exactly where needed not at the beginning of the whole questionnaire. • Use italics or bolding or underline to emphasize instructions and directions

41. Questionnaire Design • Pre-attentive processing is 210 degrees • Focused reading is 2 degrees or about 8 to 9 characters • Use font, size, contrast, underline, spacing to lead the respondent through the questionnaire

42. Start here: • Which of the following is your main work activity? Research  Teaching  Administration  Other (Please Specify) _________________

43. START HERE: Which of the following is your main work activity?  Research  Teaching  Administration  Other (Please specify) ____________________

44. Questionnaire Format • For paper questionnaires the standard font is 12 pt Times New Roman – also consider a larger font for senior audiences • For on-line questionnaires the standard font is 10 pt Ariel

45. Questionnaire Design • Use a square for your answer and ask for an “X” in the square. [ ]  1

46. Questionnaire Design • Place instructions exactly where needed within the question • Differentiate with font, bolding, underlining etc

47. 5. How many months have you worked at your current job? _________Number of months Please be as specific as possible in answering the next question, including any area of specialization. Example: Extension Agent – Agriculture. If you had more than one job, answer for the job for which you worked the most hours. 6. What kind of work do you do in your current job? _________________________________ Kind of work

48. 5. How many months have you worked in your current job? _____________________ Number of months 6. What kind of work do you do in your current job? Please be as specific as possible in answering. Include any are of specialization, for example: “Extension Agent – Agriculture”. If you had more than one job, answer for the job which you worked the most hours. ______________________________Kind of work

49. Questionnaire Design • Vertical alignment eases response task • Don’t double or triple bank answers

50. 1. Which one of the following best describes the 4-H project with the largest enrollment within your club? • market beef • lambs • hogs • dogs • cats • rabbits • goats • poultry • dairy