Preparing and Running User Experiments

1. Preparing and Running User Experiments By Mei Li, Pearl Ho, and Deepika Gandhi

2. How to Prepare and Run Usability Testing • How to gather participants and prepare a proposal. • Preparing the environment, test materials, and test team. • How to measure usability and deal with participants.

3. Preparing Materials, Environment, and Test Team for Usability Tests Presenter: Pearl Ho From Dumas and Redding Chapters 11, 12, 14, 15, 16

4. Overview • Selecting and organizing tasks to test • Creating task scenarios • Preparing test materials • Preparing the test environment • Preparing the test team

5. I. Selecting and Organizing Tasks to Test • How do you select which tasks to test? • What resources do you need for each task?

6. Selecting Tasks • Tasks that probe potential usability problems • Use tasks suggested from concerns and experience • Tasks that users will do with the product

7. Selecting Tasks • Tasks that probe potential usability problems • The more problems you find, the more successful your test will be • Tasks suggested from concerns and experience • Tasks that users will do with the product

8. Selecting Tasks • Tasks that probe potential usability problems • Tasks suggested from concerns and experience • People who develop the product will have some sense of what will be difficult to use based on which parts were difficult to design or which parts where they disagreed about the best approach • Tasks that users will do with the product

9. Selecting Tasks • Tasks that probe potential usability problems • Tasks suggested from concerns and experience • Tasks that users will do with the product • There are other tasks that users can do with a product over and above those that relate to usability problems, concerns, or goals. For example: • New or modified tasks • Tasks that are critical to the operation of the product • Frequently done tasks • Things done under pressure

10. Determining the Resources You Need for Each Task • Estimating task times • Listing the resources you need for tasks

11. Determining the Resources You Need for Each Task • Estimating task times • Time it will take to run the task during the test. • Time that users will feel is acceptable for completing a task. • Consider characteristics of participants who will be in the test • Time for experienced person. This forms baseline • Problems for typical participant and time to recover from them • Time to set up tasks and time between tasks • Estimate will probably be a range, not a single value • Listing the resources you need for tasks

12. Determining the Resources You Need for Each Task • Estimating task times • Listing the resources you need for tasks • Hardware – participants needs to do the task AND what you need to set up and run the task • Software and data files – again for both participants and you • Instructions and procedures you will need

13. Determining the Resources You Need for Each Task • In addition to task times, for most tests, it takes about an hour to • Start the test • Give the prebriefing • Give the participant a break halfway through the test • Have the participant fill out the posttest questionnaire • Give the debriefing

14. Creating the Final List of Resources • Here’s an example of what a list of resources might look like for testing the task of sending a reply on an email program

15. II. Creating Task Scenarios • What is a scenario? • What makes a good scenario? • Do you give participants written scenarios? • How do you divide up tasks and scenarios? • How do you make participants stop between tasks for timing purposes?

16. What is a scenario? • You use scenarios to tell participants what you want them to do during the test. • Scenarios describe the tasks in a way that takes some of the artificiality out of the test. • It makes the task more realistic.

17. What makes a good scenario? • Short • User’s words, not product’s words • Unambiguous • Enough information to do the task • Directly linked to your tasks and concerns

18. Do you always give participants written scenarios? Not always. You may: • Have test team members pretend to be customers, supervisors, or colleagues and walk into the test room to deliver the scenarios in person • Use the product to deliver the scenarios • Mix modes, give them writing, but then add visits or calls

19. How do you divide up the tasks and scenarios for participants? • It depends… • In deciding how to match up scenarios and tasks, the major issue you have to consider is whether you want to separate measurements of time, errors, or other codes for particular tasks.

20. How do you make participants stop between tasks? • If you want to time tasks separately, you need to get participants to stop between each task. • You can give them one task at a time. • Or you can just say after each task: “Please tell us when you have finished this task.” or “Please wait for us to tell you to turn the page.”

21. III. Preparing Test Materials • The Legal Form - minimal risk, informed consent • Questionnaires • Participant Training

22. The Legal Form As a tester it is your responsibility to: • Create a form that correctly states each party’s rights • Ensure that test participants have read and understood what the form says • Observe or witness participants signing the form

23. Concept of minimal risk • Minimal risk means that “the probability and magnitude of harm or discomfort anticipated in the test are not greater, in and of themselves, than those ordinarily encountered in daily life or during the performance of routine physical or psychological examinations or tests.” • If, in your judgment, any usability test could put participants at risk, you have three options: • Eliminate the risk • Follow the federal policies for such tests • Don’t do the test

24. Informed consent • Key to protecting both parties rights • Participants should be given the opportunity to choose what will or will not happen to them

25. Three elements of any informed consent: • Information • Comprehension • Voluntariness

26. Three elements of any informed consent: • Information: • Procedures you will follow • The purpose of the test • Any risks to the participant • Provide the opportunity to ask questions • Provide the opportunity to withdraw at any time • Comprehension • Voluntariness

27. Three elements of any informed consent: • Information • Comprehension • You must convey the information clearly and completely. • Do not rush through consent procedure, make it seem unimportant, or not allow questions. • Voluntariness

28. Three elements of any informed consent: • Information • Comprehension • Voluntariness • Participants must be free from coercion and undue influence • If participants are considering withdrawing from a test, remain calm and neutral in your manner so that you do not unduly influence them

29. Other Major legal issues: • Nondisclosure, if product is still under development • Waivers, permission to use tapes and questionnaires • Participant’s rights, ex, right to be informed

31. Questionnaires • Pretest questionnaire - gather info about the background of participant • Posttask questionnaire - gather judgments and rating after each task • Posttest questionnaire - gather judgments after completion of all tasks

35. Training Participants Reasons to train • To ensure that all participants have the same level of skill or knowledge before they begin the tasks • To provide some groups of participants with training that others do not get

36. Training Participants To provide effective training, you need to decide: • The purpose of the training • What skills and knowledge you want participants to learn • How you will conduct the training • What criteria of success you will use to measure the effectiveness of training

37. IV. Preparing the Test Environment • Preparing the Product • Data-logging material preparation

38. Preparing the Product • Create the sample data you need. • Plan for disaster. Save time and frustration. • Preparing for manuals. • Legible type and graphics • Numbered pages • High quality, professional looking binding or binder • Put in table of contents, cross-references, and an index

39. Preparing Data-Logging Materials or Software • The key to having quantitative measures of performance is recording the duration and frequency of events during the test. • You can buy data-logging software or at least have stopwatch and sheet of paper that lists the task numbers. • If you decide to videotape the test, keep in mind that it takes twice as long to run test and then watch the tape.

40. V. Preparing the Test Team • How many people do you need to run a test? • Should people switch roles during the test? • Who should be on the test team • What roles are there for test team members

41. How many people do you need to run a test? • Three people make a good team. Some have five, including a technical person. • Two can make up team if both are skilled usability specialists and participants don’t have to move around much.

42. Should people switch roles during the test? No. • You don’t want differences in data based on differences in test team.

43. Who should be on the test team? • Everyone on test team should be on the planning team, but not everyone planning has to be on test team. • Product developers or usability specialists? Realize obvious tradeoffs and choose one that makes sense for you. • Other people, such as managers, developers, writers, can participate in a test as observers.

44. What roles are there for test team members? • Test administrator • Briefer • Camera operator • Data operator • Help desk operator • Product expert • Narrator

45. See the text book if you want specific tasks and a checklist for each role

46. QUESTIONS? On to Deepika…. During your User Requirements Activity