Interactive Systems Technical Design

1. Interactive Systems Technical Design Lecture #4 Context-aware interaction with mobile devices ISTD 2003

2. Context • Lenat et al.: ”There are a dozen mostly-independent dimensions along which context vary (Absolute time, Type of time, Absolute place, Type of place, Culture, Sophistication/Security, Granularity, Epistemology, Argument-preference, Topic, Justification and Let’s) Conversely, each region of that 12-dimensional space implicitly defines a context” • Abowd et al.: ”Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computation object” • Schmidt et al.:” Interrelated conditions where something exists or occurs” • Simplified: ”Description of the situation”, ”Snapshot of the world” ISTD 2003

3. Context Space Dimensions (Lenat et al.) • Absolute time :a certain time interval in which events occur • Type of time:a non-absolute type of time period, e.g. ”after breakfast” • Absolute place:a particular location, where events occur, e.g. ”Oulu” • Type of place:a non-absolute type of place, e.g. ”in bed” • Culture:linguistic, religious, ethnic, age-group, wealth etc. of typical actors • Sophistication/Security:who already knows this, who could learn it, etc. • Granularity: phenomena and details which are (not) ignored • Epistemology (Modality/Disposition): who wants/believes this context to be true? • Argument-Preference: local rules for resolving pro-con argument disputes • Topic/Usage: aspects and applications • Justification: are things in this context generally proven, observed, on faith, … • Let’s: local bindings of variables etc. that are true in that context ISTD 2003

4. Context Categories (Dey) • Categorization helps to uncover the relevant pieces of context wrt. a given application • Who’s, Where’s, When’s and What’s of entities • Incoming context is used to encode some action in the application • Most common contexts • Location • Time • Identity • Activity ISTD 2003

5. Context-Awareness • Abowd et al.: ”Context-awareness or context aware computing can be defined as the use of context to provide task-relevant information and/or services to a user” • Pascoe et al.: ”The ability of a device or program to sense, react or adapt to its environment of use” • Simplified: ”Use of context” ISTD 2003

6. Context-Awareness Categories (Dey) • Attempt to describe how context is used • Categories • Presentation of information and services to a user • Automatic execution of a service • Tagging of context information for later retrieval • Show primary types of features to consider wrt.a particular context-aware application ISTD 2003

7. Context (Engineer’s View :) • Time (absolute, relative, …) • Location (absolute, relative, symbolic, …) • People (profile, presence status, location, …) • Physical environment (objects, weather, …) • Virtual environment (data, …) • Operation/activity (task, mission, objectives, …) • Device context • Note! • Absolute vs relative context information • Physical vs symbolic/logical context information ISTD 2003

8. Mobile Device Context • Environment • Illumination, temperature, humidity, noisiness, … • People, machines, networks, services, … • Location & position • Coordinates, places, indoors, outdoors, … • Near/far wrt. user, tilted, orientation, … • Activity • Touched, held, moving, shaking, looked at, talked to, power on/off … • Note! A mixture of absolute/physical and relative/symbolic/logical context information ISTD 2003

9. Context-Awareness and Mobile Devices • Goal: Improved interaction • Adaptive services and interfaces • Automated context recognition preferred over manua • Adaptive interfaces • Modality (how the user uses the device) • Visual appearance • Functions (different functionalities in different contexts) • Presentation of context-relevant information • Decrease information overload • Selective context sharing ISTD 2003

10. Context-Awareness and Mobile Devices (cont.) • Challenges for multidiciplinary research • Data (context) acquisition & hardware issues • Context recognition • Distributed computing & protocols to handle mobility • Human Computer Interaction issues for creating usable applications • Knowledge engineering issues for representing and manipulating context information and utilizing background information ISTD 2003

11. Operation sequence Example (proximity sensor) Context recognition IR emitter & receiver Sensors 1. IR emitter emits light External information 2. IR receiver measures the amount of reflection, which depends on the surface type, size of object and distance Context representation Proximity representation (a) Direct (sensor) values (a) Distance to the user in cm (b) Inferred states (b) Proximity state (Close, InRange, OutOfRange) Context utilization Adapt UI according to distance (loudspeaker volume, font size, …) Context-Awareness for Improved Interaction ISTD 2003

12. Case Study ”Sensitive mobile phones” Microsoft Research, USA Hinckley K, Pierce J, Sinclair M & Horvitz E (2000) Sensing techniques for mobile interaction. Proc. UIST'00, 91-100. (Best Paper Award) Hinckley K & Horvitz (2001) Toward more sensitive mobile phones. Proc. UIST’01, 191-192. http://research.microsoft.com/users/kenh/ ISTD 2003

13. Overview • Context of interaction is recognized with simple sensors • Particularly, information on how the device is being held • Context is used to improve interaction techniques • Holding the device in certain ways indicates what action is being performed (or a small set of possible / likely actions) • Complete automation of particular functions • Speedups and improvements of particular functions ISTD 2003

14. Mobile Device: Casio Cassiopeia E105Augmented with Sensors ISTD 2003

15. Sensor #1: Proximity Range Sensor • IR emitter (light-emitting diode) • IR light modulated at 40Khz • IR receiver (tuned to sense 40Khz only) • Measures the amount of reflection • Different readings for different surfaces and sizes of object • Pretty consistent, since we are mostly looking at the same thing (skin) • Estimates the distance to an object • Range 5 – 40 cm ISTD 2003

16. Sensor #2: Touch Sensors • Two separate sensors (“plates”) • “Holding” sensor on back and sides • “Bezel” sensor at the left edge of screen • Sensor ”plates” • Two regions: ”active” area and ”ground” • Conductive paint • Sensor “plates” detect capacitance changes • Capacity changes when the user’s hand touches the “plate“ vs. not touching ISTD 2003

17. Sensor #3: Tilt Sensor • Two-axis linear accelerometer (in plane of device) • Gravity provides constant acceleration (1g) • Direction of acceleration vector  x,y tilt • Direction of gravity vector can’t be recognized • Screen up vs. screen down can’t be recognized • Could be solved with a gravity-activated switch • Can also detect patterns of movement • Realized with a single chip (ADXL05) • Cheap and relatively easy ISTD 2003

18. Sensor Architecture • PIC micro-controller (Microchip 16C73A) • Constantly reads sensors (~400 samples / sec) • Reports values to PDA via serial port • Context server (“Broker”) application on PDA • Receives and processes values • Makes information available to other applications via API • Polled values • Subscribed events ISTD 2003

19. Processed Values From Sensors • Things taken directly from processed sensor data without recognition / inference • Tilt sensor • TiltAngleLR, TiltAngleFB • HzLR, MagnitudeLR, HzFB, MagnitudeFB • Dominant frequency and magnitude of movements (separate x,y measures) • Obtained via FFT over short window ISTD 2003

20. Processed Values From Sensors (cont.) • Proximity sensor • Proximity • Estimate in cm • ProximityState • Close (< ~7cm) • InRange (~7-25cm) • OutOfRange • Duration inthat state ISTD 2003

21. Contexts Inferred From Touch Sensors • Holding (duration) • From back / side touch sensors • Is user holding the device? • If so how long he has been holding it? • TouchingBezel (duration) • Similar for bezel touch sensor • Not considered to be touching it until duration>0.2 sec • Thresholds like this is used in a number of places ISTD 2003

22. Contexts Inferred From Tilt Sensor • LookingAt (duration) • Small range of angles appropriate for typical viewing • Moving (duration) • Any movement and how long since last still period • Walking (duration) • Detected by repetitive motion in 1.4-3Hz range • Shaking • Device is being shaken vigorously ISTD 2003

23. Contexts Inferred From Tilt Sensor (cont.) Operation sequence: ”entering an elevator, looking at display, holding the device down at side, walking to a meeting” Angle Looking at display Walking Y (FB) Tilt Holding at side X (LR) Tilt ISTD 2003

24. Context-Aware Interaction Techniques • Automatic power on • Voice memo recording • Portrait / Landscape display mode selection • Tilt Scrolling • LCD contrast compensation • Display changes for viewing while walking ISTD 2003

25. Automatic Power On • PIC processor is on even when PDA is off • PDA is powered up when you pick it up to use it • Holding in orientation for use • Specifically • Holding + LookingAt • In portrait orientation (but not Flat) • For 0.5 seconds • Extremely natural interaction • Pick it up and its ready to use • Avoids most false positives ISTD 2003

26. Voice Memo Recording • Interaction: Pick it up like a cell phone and talk into it” • Holding + Proximity + proper orientation (tilt) • Audio feedback (critical) • Click when pickup gesture recognized • Beep to start recording • Double beep at stop • Stop via loss of preconditions • Again, very natural interaction: pick it up as necessary to do a particular thing and it does it ISTD 2003

27. Display Mode Selection • DisplayOrientation event / data • Flat, Portrait, PortraitUpsideDown,LandscapeLeft, LandscapeRight • Refresh event when changed • Dead bands importantto remove jitter • Must cross all the way through zone to count • Flat is important • Don’t change orientation • Interacts/interfereswith scrolling • Input controls remapped tocurrent orientation ISTD 2003

28. Tilt Scrolling • Must explicitly touch bezel touch area to enable • Not quite as nice, but still pretty easy • Then tilt up, down, left, right to scroll • Rate controlled (exponential) • Dead band of 4 degrees to prevent jitter • Application menu and Start bar hidden during scrolling • LCD contrast compensation • Contrast adjusted according to the viewing angle ISTD 2003

29. Tilt Scrolling (cont.) • Interference: Tilt scrolling interferes with display orientation selection • Hence explicit clutch • “I’m scrolling now” event for explicit disabling across applications • BUT… what about when you let go • Can’t tilt back prior to releasing clutch • Not scrolling anymore once released • Handled by basically disallowing display change in direction of scroll tilt immediately after a scroll ISTD 2003

30. Future Work: Display Changesfor Viewing while Walking • If you hold the device in reading orientation (Holding + LookingAt) and walk (Walking), it will increase font size, etc. • Walking detected by repetitive motion in 1.4–3 Hz range ISTD 2003

31. Lessons and Issues • Sensor fusion • Typically need multiple evidence for a situation in order to avoid false positives • E.g., “looking at” via both angle and touching instead of just angle • Need recognition error tolerant designs • False positive or negative is not a catastrophe • In general recognition errors can easily ruin a good interaction • Recovery costs can easily destroy benefits ISTD 2003

32. Lessons and Issues (cont.) • Interaction/interference between techniques • Handled by explicit disabling • Event saying “We’re using that now” • Used to disable other effect • Somewhat ad hoc • General issue that needs work • Probably needs some sort of general conflict resolution mechanism • But not clear what that is, or how to do it ISTD 2003

33. Lessons and Issues (cont.) • Big take away point • Knowing context of operation (e.g. how held)allows space of possible current interactions to be constrained • Less explicit actions required to invoke an action or provide a input • E.g., typically avoid button presses or other mode entry • Can result in simpler and more natural interaction • Best case: Picking it up to do something causes the something to “just happen” ISTD 2003

34. Video Video ISTD 2003