Gist Qualtrics : Using the UNT Online Survey System

1. Gist Qualtrics: Using the UNT Online Survey System Dr. Russell Ruffu, Office of Institutional Research and Effectiveness

2. Today’s Agenda • Electronic surveys at UNT • Qualtrics at UNT: Brief History • Resources for Problem-Solving • Ethical Use of Data • Qualtrics Demo

3. Electronic Surveys at UNT • 2000: IRE purchased Inquisite (not open across campus) – hard to learn and very complex • 2005: Division of Student Affairs purchased Survey Monkey (monthly subscription) • 2007: DSA moved to annual Survey Monkey subscription • 2009: DSA purchased second, private Survey Monkey account to lock down sensitive data (limited access) • 2010: Colleges of Education and Business purchased Qualtrics (limited license) • 2010/2011: IRE approached campus users to purchase university-wide license for Qualtrics • April 2011: Qualtrics added to MyUNT

4. StepsforProblem-Solving • Start off at Qualtrics University www.qualtrics.com/universitywww.qualtrics.com/university/learn-qualtrics/ • Access the Qualtricsself-help module at http://institutionalresearch.unt.edu/sites/default/files/Qualtrics%20101Training%20Series%2010_1_12.pdf • Contact Institutional Research and Effectiveness Qualtrics CoordinatorDr. Russell Ruffurussell.ruff@unt.edux 7894

5. Help for specific types of questions/items: Go toHttp://www.qualtrics.com/university/researchsuite/basic-building/basic-survey-options-settings/survey-experience

6. Ethical Use of Data • All data collected should be treated as security-sensitive • You should only access data for which you have a legitimate work purpose • Use care when sharing data with others, whether formally or casually • Be aware of collaborating with external parties, you may unwittingly be compromising FERPA • National Institutes of Health training (phrp.nihtraining.com/users/login.php) – useful even if not IRB study • Avoid downloading individual response data to a laptop or portable file/device

7. HOW TO USE THIS GUIDE: • The following slides are not designed to show you every detail of Qualtrics – just the most frequently used. • This presentation has been organized to mirror the actual process of creating a survey, and distributing the instrument • Where appropriate, screen shots and notes in the margin are designed to help first-time users.

8. Qualtrics Training SeriesPart I Survey Creation

9. Opening page: use MyUNTFaculty  Qualtrics Click on Create Survey tab or the Create Survey icon Page shows all of your surveys as well as surveys in collaboration (separated by the gray divider bars).

10. Create Survey: Quick Survey Builder Click Quick Survey Builder. Create from Copy provides a feature that copies a survey from your Qualtrics file. The Survey Library links to templates that you have created or borrowed that exist in your Qualtrics account, or are available from the Qualtrics library.

11. Enter Your Survey’s Name and Identify the Folder it will be Kept. Then click “Create Survey.”

12. Block and block options A block can contain specific categories of questions which can be manipulated for navigation and analyses purposes. For example, blocks might be labeled Track 1 and Track 2 for respondents who ‘branch’ off within the survey; or Blocks might be labeled as Demographics, Activities, and Perceptions to segregate the type of information being sought. (If you simply want one question to appear at a time, just add a Page Break.)

13. Create a New Item: Click Create New Item. Write the question text. Bottom Right corner, click “Add Block.” Write second question. What if you decide you want both questions displayed in same block? Then remove the last block. But… Blockdeletionis a little funky BEST: Plan your blocks/question groups before you start developing the survey. FUNKY: Check box for each question below the block marker that you want to removemove all questions up by clicking up-arrow at far left of questions. Remove Block option now appears in bottom right corner of last block. FUNKIER: Block OptionsDelete Block, questions go to trash at bottom of surveyclick on question in trash, then click RestoreMove question to desired location.

14. Most Commonly Used Question Types • Descriptive Text – allows you to communicate with respondent (no active data collection) • Graphic – allows you to upload your logo or other image • Multiple Choice – allows respondent to pick from a list (either one item or multiple items) • Matrix Table – Series of statements being measured on a common scale (e.g. strongly disagree, disagree, neutral, agree, strongly agree) • Text Entry – allows for write in (qualitative) responses in either single lines, short essay or long essay formats • Slider – Graphically interesting scale item • Rank Order – Users can re-arrange answers in order of importance • Side by Side – Lets users choose answers in a forced choice model

15. Item Types Let’s start with Text/Graphic items

16. Text/graphic item: A bit about buttons & graphics FYI, code is available from the Qualtric site to serve many functions. This is a sample. Button functions, like Next and Back, are automatically embedded. The Back button is inserted/selected by going to the Survey Options icon at the top of the page.

17. Survey options Items selected are inserted automatically.

18. Labels can be added to the Next and Back buttons. Go to Look and Feel, locate Next Button Text and type in your label.

19. Display Logic: If answer item selected, then display another question If Question Q5 “In what type of h… do you reside” tent is Selected (then display this question) Force Response is activated

20. Skip Logic: If choice selected, then Skip to question X or End of Survey Skip Logic must be applied for each answer selection. That is, there is no condition for applying “all” to all of the answer choices.

21. Branching: A Quick Look so You Know It Exists Branching will send participants who choose certain specified answers down a different road to answer a set of questions that others will not see. Choose Survey Flow from the tool bar, then click Add Below from the options provided on the right in the blocks. Choose Branch from the yellow menu. Define the conditions. Notice that you can select And…And, or you can select Or, Or, or a combination of these. Drag the defined conditions to “Add a New Element Here. Save Flow

22. Elements In Branching Operation Block= new section of questions Branch=new road based on previous answer(s) Embedded Data=information about respondents long before you survey them Randomizer=Changes order of response choices Web Service=bring additional information from a website into your survey Authenticator=The Authenticator makes it possible to have respondents “log in” to a survey, along with other features to authenticate the respondent. Conjoint=A conjoint study usually involves showing respondents a set of features and asking them to indicate how much they like or prefer the different attributes of that feature. End of Survey=takes participant to end of survey

23. Qualtrics Training SeriesPart II Survey Distribution

24. Distribute Survey Sub-Tabs There are a couple of ways to distribute your survey. The first is called Survey Link. The other one we will focus on is the use of Panels, which seems to be the most popular here at UNT. FIRST – you have to activate the survey using the link above! • Survey Link: According to Qualtrics University, • “It’s the easiest way to distribute a survey: a simple, anonymous link that can be taken by anyone. You can: • Send it out with your personal email. • Post it on a web page. • Use social media to distribute it.”

25. http://www.qualtrics.com/university/researchsuite/distributing/basic-distribution/anonymous-survey-linkhttp://www.qualtrics.com/university/researchsuite/distributing/basic-distribution/anonymous-survey-link

26. The Survey Link Highlight and Copy it. Go to the Email Survey sub-tab to an email. Paste the link here.

27. Mailing Lists/Panels

28. Upload a mailing list from a csv Excel sheet The FirstName LastName are NOT mandatory, but are commonly included in spreadsheets with email addresses. They are not necessary for uploading survey mailing addresses. 1. 2. 3. 4.

29. Import from a File: choose this one

30. Only Excel CSV can be imported

32. -OR- 1. 2. 3. 4.

33. Email Survey You can type in, copy & paste, select everyone or individuals from a previous panel.

34. Create conditional sample from existing panel Set the Panel conditions

35. -or- Enter by hand. Note expansion of panel to include embedded data. You can use this to test the integrity of emailing your survey. Be aware that... Using this email choice produces…

36. …not all features are activated in the SEND TEST EMAIL Not so good (using “Send Test Email”) Much better (using “Schedule Mailing”)

37. Authentication: Avoid Ballot Box Stuffing; Assure Correct Audience Only is taking your survey. Authentication Usually occurs Before a Participant is allowed to take the survey, and may rely on embedded data typically included in a panel. The panel is often associated with a csv file that includes a password, euid, or other unique embedded data. It can be the same panel used for distribution. Start with the Edit Survey tab  Survey Flow  click Add a New Element Here  select Authenticator click Move and drag Authenticate Using Panel to the top of the Survey Flow. Place blocks and other element beneath the Authentication box. Using the drop-down menus, select the fields used for authentication process. Optional labels can be typed in if you want the field names to be different from those used in the Panel. The minus and plus icons are used to add or remove fields from the authenticator. Options bottom right corner = set parameters and text messages for authentication. Click OK and save changes. Go to http://www.qualtrics.com/university/researchsuie/advanced-building/survey-flow select Authenticator for a 2.5 minute video describing the authentication procedures.

38. Qualtrics Training SeriesPart III Downloading and Analyzing Results

39. Overall Tips: Analyzing Data • In general, look for trends in the responses • Evaluate blanks as a possible clue that the respondent didn’t understand the question • Be aware of “Christmas Treeing” where respondents answer all the same way throughout every item – consider deleting • Look at your response rates and compute a confidence interval – go to: http://www.surveysystem.com/sscalc.htm for a great resource • Share data internally and externally to drive changes in practice • Look at your demographics, do they match your overall population? • Are you missing certain members of your community? How does this non-response affect your findings? • When in doubt, contact IR&E for a consultation on your data and findings.

40. View Results Click here first!

41. Recorded Responses: Individual

42. Download Data: to CSV, SPSS (commonly used)

43. CSV OUTPUT Answers as Values Answers as Labels

44. SPSS DATA DOWNLOAD

45. Thank you for your attendance today. This PowerPoint is available on the IR&E web page at: http://institutionalresearch.unt.edu/institutional-assessment GIST QUALTRICS: Survey Builder Training Module

46. CONTACT INFORMATION: Dr. Russell Ruffu Office of Institutional Research and Effectiveness Hurley 121E Phone: 940.369.7894 Email: Russell.ruffu@unt.edu Web:http://institutionalresearch.unt.edu/institutional-assessment