OBSERVING USERS

1. OBSERVING USERS Shubha Tandon Nandita Kodali

2. Outline • Introduction • Goals , Questions &Paradigms • How to Observe • Data Collection • Indirect Observation (tracking user’s activities) • Analyzing, Interpreting and Presenting Data

3. Introduction • Watching and Listening • User can be observed in 1) Controlled environment as in Usability Testing 2) Natural environments i.e the feild

4. Goals, Questions and Paradigms • Goals and Questions are necessary to help the observers stay focused • Goals and Questions should guide all evaluation studies • Studies should also be open to modification or refocusing as observers learn more about the situation

5. What and when to observe • Observing is done at all stages of Product Development • Observers can be : a) on-lookers b) participant observers c) ethnographers The degree of immersion that evaluators adapt varies across a broad outsider-insider spectrum.

6. Outsider-Insider - illustration Aim : To observe WAP phones Scenario 1:The observer joins a group which goes to DC .The group uses it find restaurant in the area ,and while waiting for a taxi call the restaurant and book tables. Observes that there are some problems with the interface ,but on balance the device was useful and the group is pleased

7. Outsider-Insider - illustration Scenario 2 :User need to do a preplanned task in a usability laboratory ,which is to search for the phone number of a particular restaurant . The video recording and interaction log suggest that the screen is to small for the amount of information they need to access

8. Outsider-Insider - illustration • Which situation does the observer have more control ? • What are advantages and disadvatages of the two methods? • When might each type be more useful ?

9. Outsider-Insider - illustration • Which situation does the observer have more control ? -Second study as the task is predetermined , The participant is told what to do and they are in a controlled environment.

10. Outsider-Insider - illustration • What are advantages and disadvantages of the two methods? Field Study : Advantages- Real situations,Real Problems. Experienced the delight expressed at the over all concept and also frustration with the interface . Understood what the users like and need in real life situations

11. Outsider-Insider - illustration Disadvantages: Since the observer is an “insider” how objective can she/he be? The observer might be having a good time and might not notice some peoples annoyance Another study could be done to find out ,but it is hard to replicate the exact situation.

12. Outsider-Insider - illustration Laboratory Study: Advantages- Several users perform same task can compare performance and take averages. Easy for the observer to be objective . Disadvantage-Study is artificial ,says nothing about how to use the device in real environment.

13. Outsider-Insider - illustration When might each type be more useful ? • Depends on the goals of the study • Laboratory study is used for examining details of the interaction style and correcting usability problems e.g. button design etc. • Field study is used to see how the phone is used in real world ,how it integrate or changes users behavior .

14. Approaches to Observation • “Quick and dirty “ observation -Can occur anywhere anytime, to find out what's happening quickly and informally. • Observation in Usability testing -controlled environment, video and interaction logs used . • Observation in field studies -observer can be anywhere in the outsider-insider spectrum. Complete participants ,more marginal participants ,observers who also participate ,people who observe from the outside and do not participate

15. How to observe • In Controlled environments • In the Field

16. How to observe • Basic Data collection tools: direct observation ,taking notes, collecting videos etc. • Laboratory-what individuals do? • Field –in what context they do it , how they interact with other people, technology etc .

17. Observation in Controlled Environments • Observer collects data and then tries to make sense of the data . • Practical issues have to be taken care of in advance eg :where the users are located, test the equipment, get consent from user etc

18. Observation in Controlled Environments cont. Q) How does the observer know what the user is thinking? Sol) Thinking aloud Technique: This technique requires users to think aloud everything they are thinking or trying to do

19. Observation in Controlled Environments cont. • A better solution: Two people working together and talking to each other .It is proven to be more successful , as its more natural and revealing for people to talk and help each other out .

20. Observation in the Field • Events in the field are complex and change frequently. • Evaluators have Frameworks to structure and focus their observations .

21. Observation in the Field cont. Eg of a basic framework • The Person. Who is using the technology at any particular time ? • The Place . Where are they using it ? • The thing . What are they doing with it? Experts prefer a more elaborate frame work with greater attention to detail

22. Checklist for Field observation • Have a study goal and questions • Select a Framework • Decide how to record events • Routine revisions of notes and records • Highlight and separate personal opinion • Refocus if necessary • Try to gain acceptance • Prepare on how to handle sensitive issues

23. Checklist for Field observation cont. 9. Try Working as a Team 10.Consider different perspectives

24. Data Collection • Methods available: • Notes and still camera • Audio recording plus still camera • Video • Can be used individually or in conjunction • Which to use - decide based on context, time available and subject sensitivity.

25. Notes Plus still Camera • Least technical and cheep • Taking notes is flexible and unobtrusive • Transcribing handwritten notes, can help organize and analyze data • Downside: • writing is tedious, slow and boring • maybe biased - only what note-taker thinks is important gets recorded. • feedback to design team depends on the note-takers authority • Photos can supplement written notes.

26. Audio recording plus Camera • Inexpensive • Provides mobility • Relatively unobtrusive • Provided extensively detailed audio information, • Permanent original record - can be revisited • More convincing than notes - incontrovertible • Drawbacks: • Requires a lot of transcription - but depending on detail maybe only parts are required. • High external noise • Changing cassettes and microphone position maybe a problematic.

27. Video • Captures audio + visual information- Complete • Reliable, unbiased, permanent data • Critical incidents can be identified and tagged for analysis • Downside: • More expensive - requires mixing, analysis equipment • Obstructive - requires focusing and positioning • Attention becomes focused on what is seen - may miss important details outside the focus span. • Detailed analysis may be very time consuming -but may not be needed

28. Indirect Observation - Tracing Users • When Direct observation is not possible • Totally unobtrusive • Two techniques: • Diaries • Interaction Logging • From these records evaluators reconstruct what happened and look for usability problems

29. Diaries • Provide records of: • what users did • when they did it • what they though about interaction with technology • Very useful when users are scattered and unreachable in person. E.g.. Internet and web evaluations • Templates (like open questionnaires) can be created for standardization

30. Advantages helpful scattered users Inexpensive No special equipment or expertise suitable for long term studies Standardized online templates can be read directly into a database Disadvantages study needs the participants to be committed to remembering and completing the dairies Needs incentive Have to be very straightforward user observations tend to be subjective (better or worse/ longer or shorter) than they actually are Diaries - good and bad

31. Interaction logging • Can be done by recording • key presses, mouse or other device movements • these logs can be synchronized with video and audio to understand how users go about the tasks • Logs are time stamped to calculate how long users spend on a particular task or part of software. • For websites: • Explicit counters to record number of visitor • counters to how long people stayed at a site and which areas they visited, where they came from etc.

32. Advantages Unobtrusive large volumes of data collected automatically can help collect useful information about number of visitors used to maintain and upgrade a website or subsites Disadvantages Is it ethical? powerful tools needed to analyze the vast amount of data collected. (e.g.: WebLog) Interaction Logging - good and bad

33. Interaction logging - Is it ethical? Tradeoff: • With technology data can be collected without users’ knowledge. • If users are told they are being observed they may change their behavior. • Where do you draw the line ?

34. Analysis, Interpreting and presenting the data • Studies generate large amounts of data • So, it is important to • first identify goals and questions • based on these determine which data is collected and how it is analyzed. • For analysis: • eyeball data to see what stands out • Are there patterns or significant events? • Is there evidence that answers a question or supports a theory? • Analysis according to goals and questions

35. Data Categories • Three types of data: • Qualitative data which is interpreted - tells a ‘story’ about what was observed • Qualitative data which is categorized - using techniques like content analysis • Quantitative data : treated statistically

36. Qualitative analysis to tell a story • Objective: To construct convincing story illustrated with powerful example from data. • Steps involved: • Review data after each observation session, identify key themes and make collections • Record themes in a coherent, flexible form, with examples • Record the date and time of each data analysis session. • As themes emerge, check understanding with other observers and informants. • Iterate this process till a faithful ‘story ‘ emerges • Report findings to development team, preferably with an oral presentation as well as report.

37. Qualitative analysis for categorization • Three main techniques: • Looking for incidents or patterns • Analyzing data into categories • Analyzing discourse

38. Looking for incidents/ patterns • Useful when extremely fine-grained analysis not needed • Look for critical incidents when users were obviously struck • marked by comment, silence, puzzled looks • review these in detail, treat rest of video as context • Another approach: Use theory to focus on relevant incidents. • Need tolls for handling data and recording analysi • NUDIST • Video-PRO, etc. • Report from analysis is feed back to development team with video clips

39. Analyzing Data into categories • Content analysis: Fine grained way of analysis video • Challenges: • Determining meaningful orthogonal (mutually exclusive) categories to code the content under. • Deciding appropriate granularity for categorization • Content categories have to be reliable so that analysis can be replicated. - can be accomplished by training two researchers in using categories and having both analyze the data till high ‘inter-research reliability rating’ is obtainedobtained.

40. Analyzing discourse • Assumption: ‘There is no objective Scientific truth’ • Focus analysis on the meaning of what is said, not the content: • strongly interpretive - different people may have different perspectives • pays attention to context • Conversation analysis (fine grained discourse analysis) in which semantics is analyzed is detail is used for analyzing discourse on Internet (chartrooms, bulletin boars, etc.)

41. Quantitative data analysis • Steps involved: • Video data collected in usability laboratories is annotated by hidden observers • Errors or unusual behavior is usually marked and remarks added • Evaluators use the annotations to calculate performance times so as to compare performance of various prototypes • This data is also subjected to simple statistical analysis - mean, SD etc..

42. Feeding findings back into design • How to convey evaluation results to design team: • Well written report with overview at beginning and detailed content list • Include anecdotes, quotations, pictures, and video clips • Quantitative data may be helpful depending on type of study and goals • Verbal presentation including video clips is very powerful • having both qualitative and quantitative analysis is good - provided different perspectives.

43. Summary • Observation in usability testing depends tends to be objective – from the outside • In participant observation, observers works with the user to understand their activities and problems • Observational data collection and analysis depends on paradigm – quick and dirty, user testing, field study • For data collection a combination of video, audio, paper records, diaries and data logs can be used.

44. Summary • Evaluators should discuss and summaries their findings soon after the observation session. • Analyzing video and data logs – tedious – important to have access to appropriate tools and specific questions and goals to guide the process • Helpful to flag events in real time and subject these key events to detailed analysis, using the rest as context.

45. Questions ???