Asking Users and Experts

Asking Users and Experts by Xianghua Ding Hoang Minh Ho Dac

Outline • Asking Users: • Interviews • Questionnaires • Asking Experts: • Heuristic evaluation • Walkthroughs • Cognitive • Pluralistic

Asking Users • Interviews • Developing questions • Planning interviews • 4 Types of interviews • Data analysis and interpretation • Questionnaires

Developing Questions • Avoid long questions • Avoid compound sentences • “How do you like this cell phone compared with previous ones that you have owned?” • “How do you like this cell phone? Have you owned other cell phones? If so, how do you like it?” • Avoid using jargon • Avoid leading questions • “Why do you like this style of cell phone?” • Be alert to unconscious bias

Planning an Interview - Steps • Introduction • To introduce himself, explain the purpose and get consent • Warmup session • Using easy, non-threatening questions • Main session • To ask prepared questions from easy to difficult. • Cool-off period • a few easy questions • Closing session • To thank the interviewee and to clear up the scene • Make the interview as pleasantas possible • The golden rule is to be professional

Planning an Interview - Advice • Dressing • In a similar way to the interviewee if possible • Prepare a consent form and ask the interviewee to sign in • Make the equipment work • Make sure your recorder works and know how to use it in advance. • Record answers exactly

4 Types of Interviews • Unstructured interviews • Structured interviews • Semi-structured interviews • Group interviews

Unstructured Interviews • Both interviewer and interviewee have control and can steer the topic • More like a conversation • Focus on particular topic but go into depth • Questions are open • No predetermined content and format • Interviewee is free to answer questions as fully or as briefly as she wished • Need a plan to make sure the main things to be covered

Unstructured Interviews- Advantage & Disadvantage Disadvantages • Time consuming • Ethical issues • Impossible to replicate the process • Difficult to analyze all data Advantages • Generate rich data

Structured Interviews • The interviewer has the most control • Pose predetermined , closed questions • The study is standardized • The same questions are used with each participants • Useful when study goal’s are clear and specific questions can be identified

Semi-structured Interviews • Combines features of structured and unstructured interview • Use both closed and open questions • Has a basic script for guidance • so the same topics are covered • Probe are device to get more information • “Do you want to tell me anything else?” • Be aware not to preempt an answer • “You seem to like this color” • Accommodate silence • Prompt the person to help her along

Semi-structured Interviews -An example • Which websites do you visit most frequently? • <Answer several but stresses that se prefers uci.com> • And why do you like it? • <Answer> • Tell me more about X? • <Answer> • Anything else? • <Answer> • Thanks. Are there any reasons that you haven’t mentioned?

Group Interviews • Involve a small group guided by a interviewer to facilitate discussion • Focus group • Normally 3-10 people are involved • Participants are representative of a certain type of users; they normally share certain kind of characteristics • Allows diverse and sensitive issues to be raised

Group Interviews –Advantage & Disadvantage Advantages • Method is readily understood • Findings appear believable • Low-cost • Quick results • Easily be scaled Disadvantages • Facilitators need to be skillful • Difficult to get people together in a suitable location and time

Data Analysis & Interpretation • Quantitatively • For structured interviews • Qualitatively • For unstructured Interviews • A coding form may be developed • Comments may be clustered along themes and anonymous quotes used to illustrate points of interest • Tools such as NUDIST

Asking Users: Questionnaires • An alternative technique for getting users’ opinions • Can have closed and open questions Strengths: • Distributed to a large number of people • Provide evidence of a wide general opinion

Guidelines for Designing Questionnaires • Make questions and instructions clear • If possible, ask closed questions and offer a range of answers • Include a “no-opinion” for questions that seek opinions • General questions should precede specific ones • Group related items • Specify age as a range • Different versions for different population • Balance between white space and compactness • If scales are used, the range should not overlap • The ordering of scales should be intuitive and consistent

Example of Poorly Designed Questionnaires 1. State your age in years 2. How long have you worked here? (check one only) 3. How long have you use the Internet? (Check one only) 4. Do you use the Web to: 5. How useful is the Internet to you? Purchase goods Send email Visit chatrooms Find information 1 yr 2 yrs 3 yrs > 3 yrs < 1 yr 1-3 yrs 3-5 yrs > 5 yrs

Question and Response Format • Checkbox • Used for demographic or background data • Respondents check an appropriate box or circle a response • Ranges • used for getting opinions • 2 types 1. Likert Scales 2. Semantic Differential Scales

Example of Likert Scales Instruction: In the following questions, 1 represents strongly agree and 5 represents strongly disagree. Please check only one. • The company website is helpful: 1 2 3 4 5 • The website color is annoying: 1 2 3 4 5  Should we mix positive questions with negative questions?  What is the best rating scale ? (e.g. odd like 1-3, 1-5,… or even)

Example of Semantic Differential Scales Instruction: for each pair of adjectives, place a cross at the point between them that reflects the extent to which you believe the adjectives describe the home page. You should place only one cross between the marks on each line. • Attractive |_|_|_|_|_|_|_| Ugly • Clear |_|_|_|_|_|_|_| Confusing • Helpful |_|_|_|_|_|_|_| Unhelpful  How to calculate the total score?

Administering Questionnaires Two important issues 1. How to reach a representative sample of people 2. How to ensure a reasonable response rate • With small number fewer than 20, 100% is often achieved • With larger populations, 40% return is generally acceptable • Solutions • Tell people it is OK to complete just a part • Include stamped, self-addressed envelope • Explained why you need the questionnaires • Assure anonymity • Contact users • Offer incentives

Online questionnaires • Email • Can be targeted to specific users • Quick response • Limited to text • Web-based • Flexible graphical design • Errors could be corrected easily • Immediate data validation • Less time for data analysis • Low cost for copying and postage • Have random samples of respondents • Response rate may be lower than paper form

Steps to develop web-based questionnaires • Devise the questionnaire on paper, following the guidelines • Identify a random sample of population. Avoid biased or convenience sampling • Turning the paper questionnaire into a web-based version • error-free interactive • Accessible and readable from all online users • Identification information handled confidentially • User-test before distributing

Examples of Web-based Questionnaires • What do you think about the questionnaires at this website? Are they good or bad? Why? • http://www.perseusdevelopment.com/surveytips/samplesurveys.html

Analyzing Questionnaire Data • Display data graphically (e.g. bar charts) • Often simple statistics are needed (number of participants, percentage of responses…) • Identify any trends, patterns or relationship between responses

Example – Statistics Table

Example – Pie Chart

Example (cont.)

Asking Expert • Heuristic evaluation • Introduction • Core heuristics • Doing heuristic evaluation • Heuristic evaluation for web sites • An example • Guidelines of website • Walk through • Cognitive Walkthrough • Pluralistic walkthrough

Heuristic Evaluation –Introduction • An inspection technique in which experts evaluate whether user-interface elements conform to a set of heuristics • Closely related to design guidelines • Different sets of heuristics for different products

Core Heuristics • Nielsen’s • Visibility of system status • Match between the system and the real world • User control and freedon • Consistency and standards • Help user recognized, diagnose, and recover from errors • Error prevention • Recognition rather than recall • Flexibility and efficiency to use • Aesthetic and minimalist design • Help and documentation

Doing Heuristic Evaluation • Briefing session • Experts are told what to do • A prepared script is useful as a guide • Ensure each person receives the same briefing • Evaluation Period • Experts independently inspecting the product, using heuristics for guidance. • At lease 2 passes • Give a feel of the flow of the interaction and the product scope • Focus on specific interface elements • Debriefing session • Discuss their findings, prioritize the problems, and suggest the solutions

Heuristic Evaluation for Websites – an Example • http://www.lib.uci.edu/

Heuristic Evaluation for Websites – an Example • Heuristics tailored from Nielsen’s original set • Internal consistency • Is the logo, format, text , font or usage of terms consistent? • Minimizing the user’s memory load • Layout • Is it compact? Is the page layout meaningful? Is there too much text on the page? • ……..

Heuristic Evaluation for Websites – an Example • Findings about the websites • The formatting of pages and presentation of logos are consistent of website • Some forms require users to recall instead of recognition • http://www.lib.uci.edu/services/workshops/isform.html • The layout is kind of complicated, kind of too much text • http://www.lib.uci.edu/online/ebooks.html

Heuristic Evaluation for Websites - Guidelines • Navigation • Avoid orphan pages that are not connected to the home page, which lead users into dead ends • Avoid long pages with excessive white space that force scrolling • Provide navigation support • Avoid non-standard link colors • Provide consistent look and feel • Access • Avoid complex URLs • Avoid long download time that annoys users • Information design

Walkthroughs • An alternative to heuristic evaluation • To predict users’ problems without doing user testing • How ? - Walk through a task with the system and recognize usability problems • 2 types: 1. Cognitive walkthroughs • No user participation 2. Pluralistic walkthroughs • Users, experts, specialists, designers, developers… are involved

Steps in Cognitive Walkthroughs • Users’characteristics are identified • The walkthrough team come together for task analysis • The walkthrough team go through each task, answering the three questions: • Will users know what to do? • Will users see how to do it? • Will users understand from the feedback whether their action are correct or not? 4.Record what cause problems, why and how serious they are to users 5. Revise the design to fix the problems

An Example of Cognitive Walkthroughs What to Evaluate: Java Home Page Task: to find the tutorial for Java 3D Typical users: students, developers Go through each step to complete the task • At Java Home Page • Will users know what to do ? – yes, to search the Tutorial Section • Will users see how to do it ? - yes, click on the left submenu Tutorial • Will users understand the feedback ? – yes, it leads to the List of Tutorials • At The List of Tutorials …

Steps in Pluralistic Walkthroughs • Choose a task to evaluate. Take a series of screenshots to complete that task. 2. Each member of the team looks at the screen pictures and writes down the sequence of actions they would take to move from one screen to another 3. First, users present their suggest of actions Next, experts present their findings Last, developers comment 4. Go back to step 1 with another task

An Example of Pluralistic Walkthroughs Suppose you are a member in the walkthrough team. Purpose: to evaluate the web site of Sea World Step 1 • Chosen task: to find the location and open hours • Scenarios: There are 2 screens in the path to find Sea World’s location and open hours

1 2 • Step 2: • You are shown the above 2 screens in the path to complete the task • Can you say what action will lead you from the first screen to the second screen? (don’t consult with other members in the team) • Step 3: • Do other members in the team agree with you? • If yes, what do you think about the interface? If no?

Cognitive vs. Pluralistic Pluralistic Cognitive • Focus on users’ problems in detail • No user involved in evaluation. • Do not need a working prototype • Time-consuming • Has narrow focus, only useful for certain types of systems • Strong focus on users’ tasks • Multidisciplinary evaluators with user participation in evaluation • Difficult to arrange time and location for the walkthrough team • Time-consuming • Explore only a limited number of tasks

Thank you!

Asking Users and Experts