Surveys 101: Methodology & Good Questions Part I of II

1. Surveys 101: Methodology & Good QuestionsPart I of II Joyce Chapman, Consultant for Communications & Data Analysis State Library of North Carolina, 4-16-2014

2. Agenda • Planning a survey • Distributing a survey • Best practices for survey design This webinar will be recorded and made available here: http://statelibrary.ncdcr.gov/ld/webinars.html Much info for this webinar comes from the book "Designing & Conducting Survey Research: A Comprehensive Guide." By Louis Rea & Richard Parker.

3. What is a survey and when do we use it ?

4. Before you begin • What problems are you trying to solve? • What questions are you trying to answer? • What data will help you answer the questions? • Does any of this data already exist? • For colleges and universities: do you need IRB approval?

5. Clearly define the target population for the survey

6. How do we determine sample size? This is where it gets tricky for us non experts! Try something simple like a sample size calculator: http://www.surveysystem.com/sscalc.htm

7. How do I select participants who accurately reflect my population? Ideally you want a random sample, not a convenience sample

8. Types of survey distribution • Mail-out surveys • Web-based surveys • In-library paper-based surveys • Telephone surveys • In-person interviews • Intercept surveys

9. Types of survey distribution • Mail-out surveys • Web-based surveys • In-library paper-based surveys • Telephone surveys • In-person interviews • Intercept surveys Most common types used in libraries

10. Web-based surveys: advantages • Convenience • Rapid data collection • Cost-effective • Ample time • Confidentiality and security

11. Web-based surveys Web-based surveys: disadvantages • Limited respondent base • Self-selection • Lack of interviewer involvement

12. In-person surveys: advantages • Ability to clarify and probe • Ability to contact hard-to-reach populations • Assurance the instructions are followed • Observed data (watch out!)

13. In-person surveys In-person surveys: disadvantages • Interviewer-induced bias • Less anonymity

14. Survey design

15. The introductory statement

16. The introductory statement • Mention the organization conducting the survey, state the relationship between the sponsoring institution and the respondent. • "The Duplin County Public Library is conducting a survey of residents in order to assess community opinions about services provided by the library." • Make a general statement establishing the objectives and goals of the survey and the significance of results to the respondents. • "The purpose of this study is to identify those needs that undergraduate students feel should be addressed in order to maintain a competitive 21st academic library."

17. The introductory statement • Clarify the basis of the sample selection. • "The library is particularly interested in the opinion of patrons who live on the west side of town, and as such you have been selected at random." • State that participation is valued and answers are neither correct nor incorrect. Participation is confidential. • “Your opinions are very important to us. You should know that there are no right or wrong answers and that your responses will be treated confidentially. Survey results will in no way be traceable to individual respondents." • If you are conducting an in-person or phone survey, identify yourself by name and job title.

18. Question sequence

19. Question sequence • Begin with "introductory questions." • Sensitive questions should be placed near the end. • Consecutive questions that tend to evoke reflexive responses should be minimized or respondents should be sensitized to the differences.

20. Question format

21. Question format Advantages of closed-response • Answer sets are uniform; facilitates comparison. • Makes the question clearer to the respondent. • Categories might remind the respondent of alternatives that otherwise would have been forgotten. • Sensitive questions are often better addressed this way because you are pre-establishing an implicitly "acceptable" range of answers. • They are easier on the respondent, increasing the likelihood that the response rate will be higher.

22. Question format Disadvantages of closed-response • Participants may select a response randomly rather than in a thoughtful way. • Increased possibility that simplicity of format leads to entry errors. • Compel respondents to choose a "closest representation" of their actual response. Subtle distinctions among respondents cannot be detected within the pre-established categories.

23. Question format Open-response • Recommended to use sparingly / only when needed. • Analysis will requires time-consuming categorization of open-ended response data by the researchers • Requires greater communicative skills for respondent (spelling, writing, computer usage) • Good compromise: use a close-ended question with a follow-up that's open-ended. • Do you want X?  Yes__ No__. If no, why not? ____________

24. Question phrasing

25. Jargon, colloquialisms, word choice • Think of your audience. • Wording should be simple. • Avoid technical words. X"How many times in a typical week do you use the library's Internet-enabledpublic access computers?" O"How many times in a typical week do you use the library's computers [to access the Internet]?"

26. Abbreviations and acronyms • Assume your audience does not know any of these! X“CCPL is interested in replacing its OPAC." O“Cleveland County Public Library is interested in replacing its online catalog.” X“How frequently do you use our ILL services?” O“How frequently do you use our Interlibrary Loan services? [This service allows us to request material from another library for you if we do not have it here].”

27. Avoid ambiguity • Look at the survey from every angle: are there ways that someone could interpret a question to have two meanings? X"With which ethnic group do you most closely identify?" [someone might interpret this as "which group do I best get along with?" instead of "which ethnic group am I a member of?"] O"Please indicate your race or ethnicity."

28. Avoid ambiguity • Look at the survey from every angle: are there ways that someone could interpret a question to have two meanings? X"What is your income?" O"What is your income before taxes? Include salary as well as other sources of income." X"How many people are there in your household?" O"Including yourself, how many people are there in your household?"

29. Confusingly phrased questions • Make sure your wording is not unnecessarily confusing. • The respondent should not have to spend time re-reading/interpreting the question. X"Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?" It seems likely __ It seems unlikely __ O"If the library had a Maker Space, would you use it?" Yes__ No__ Unsure__

30. Avoid “double barreled" questions • A double barreled question introduces two or more issues with the expectation of a single response X"Are you satisfied with opening hours at the West branch and at the main library? O"Are you satisfied with opening hours at the West branch?" "Are you satisfied with opening hours at the main library?" O"Are you satisfied with opening hours at the West branch?" "Are you satisfied with opening hours at the main library?"

31. Avoid “double barreled" questions • A double barreled question introduces two or more issues with the expectation of a single response X"Is our staff friendly and professional?" O"Is our staff friendly?" "Is our staff professional?"

32. Do not use manipulative information • Certain questions require some explanation/background. Be careful that explanatory statements do not unduly influence responses. XThe county government spends approximately $10 per resident on landscaping public areas. Do you believe that the county government is adequately allocating funds for our library by designating only $1.15 per resident?" O"Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per resident?"

33. Do not use manipulative information • We often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info. O"Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per resident?" O"If you were to learn that the county government spends approximately $10 per resident on landscaping public areas, would that change your opinion about the adequacy of allocating $1.15 per resident to the library?"

34. Response formatting

35. Order of response options • Often there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer. XGroup study rooms O Digital media lab Digital media lab E-books Laptop lending Group study rooms E-books Laptop lending Printing/copying Printing/copying

36. Interval scale categories • Should be as equal as possible in size. You also want each category to have an equitable number of respondents. Do not allow to overlap. Provide an unbounded final category. XAge 0-10 O Age 0-9 Age 10-15 Age 10-19 Age 15-20 Age 20-29 Age 20-50 Age 30-39 Age 50-75 Age 40-49 Age 50+

37. Multiple response • Sometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear that you are allowing multiple! Otherwise results are unclear. X For which of the following reasons do you use the library? X__ Y__ Z__ OFor which of the following reasons do you use the library? Choose all that apply. X__ Y__ Z__

38. Appropriate response choices • Surveys can be very frustrating when the questions are fixed response but do not have all the appropriate answer choices. • Make sure you provide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate. X Does the laptop lending program meet your needs? Yes__ No__ ODoes the laptop lending program meet your needs? Yes__ No__ N/A__ [or “I’ve never used this program__”]

39. You've drafted the survey!

40. Review each question against a checklist • Do you have an introduction with all the parts we talked about? • Are the first questions easy/factual? Are sensitive questions near the end? • Are reflexive response questions minimized or is respondent sensitized? • Have you clarified multiple v. single choice responses? • Are you using closed- v. open-ended responses appropriately? Do you have "other" open-ended responses where needed? • Have you done away with all acronyms/abbreviations? • Have you avoided jargon, ambiguously worded, confusing, and double barreled questions? Do you have any manipulative language?

42. Questions? Contact: joyce.chapman@ncdcr.gov 919-807-7421 Find this Powerpoint/recording: http://statelibrary.ncdcr.gov/ld/webinars.html Sign up for part II of the Survey 101 webinar series here: http://statelibrary.ncdcr.libcal.com/event.php?id=399040