Ubiquitous Computing Toolkits

1. Ubiquitous Computing Toolkits Tara Matthews Advance User Interface Software Fall 2004

2. Goals of Ubicomp • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity

3. Toolkits for Goals • Scalable interfaces • multiple devices Ubicomp Infrastructure • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity

4. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple usersCSCW (not covered) • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity

5. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop Physical UIs / Mobile • augmenting surrounding environment (other lectures) • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity

6. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services Context-Aware • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity

7. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users Peripheral Displays • aware of user’s interruptiblity

8. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity Interruptibility(not covered)

9. Toolkits for Goals • Scalable interfaces • multiple devices • inter-device communication • multiple users • Natural interaction • getting away from the desktop • augmenting surrounding environment • provide context-appropriate services • Calmness • more info w/o overwhelming users • aware of user’s interruptiblity • Goals accomplished? Evaluation

10. Toolkit Categories • Ubicomp infrastructure • Context-aware • Peripheral displays • Evaluation • Not covering: • CSCW (separate field) • Interruptibility (no toolkits yet) • Physical UIs (covered in Jack’s lecture) • Mobile Devices (covered in an unclaimed lecture)

11. Ubicomp Infrastructure • iROS • Stanford Interactive Room Operating System • iStuff • Physical application toolkit build on iROS • Aura • CMU “Distraction-Free Ubiquitous Computing” project • Distributed services infrastructure • Gaia • University Of Illinois at Urbana-Champaign • Infrastructure for Active Spaces

12. iROS • Interactive Room (iRoom) Operating System • Definition: A ubiquitous computing environment where groups come together to collaborate on solving problems (not a toolkit) • Contains: • Embedded devices: large display screens, table-top display, & other output devices • Mobile devices: mobile devices brought into space can be used with embedded facilities

13. iROS Goals • Facilitate common Interactive Workspace usages observed in iRoom prototype workspace: • Move data between devices • Control devices & apps from other devices • Coordinate apps, including ones not written to work together • Additional goals: • Provide for heterogeneity of devices in workspace, & new devices being added over time • Allow easy integration of legacy devices • Robust to transient failures • Portable across installations • Actions appear instantaneous to users

14. iROS • 3 sub-systems: • Event Heap: stores and forwards events • Data Heap: facilitates data movement in an interactive workspace • iCrafter: provides a system for service advertisement and invocation, along with a user interface generator for services.

15. Event Heap • Tuple space model • Associated memory: content is addressable using a pattern • Distributed systems based on blackboard model are easily created in tuple space • Decouples apps in space & time • Added semantics: • Apps not designed to work together, so need • Self-describing tuples, Flexible typing, Typed tuples • Apps can snoop or interpose • Tuple sequencing • Data shared & can build up • Tuple expiration • Benefits • Anonymous coordination • Interposability • Snooping

16. Event Heap Figure: Example operation showing placed events, and using template events for matching

17. Data Heap • Provides type & location independent storage • Machine independent: • Don’t need explicit location of data • Type independent: • If a set of type converters are available, system automatically transforms the data Figure: Shows automatic retrieval of a Word document in PostScript form

18. iCrafter • Universal control system: control anything from anywhere • Services advertise how they can be controlled • System returns appropriate interface per device • Interfaces can range from fully custom to dynamically generated

19. 1. USER REQUEST The user requests an interface for a service. The appliance sends this request to the Interface Manager, along with appliance description information. iCrafter

20. 2. GENERATOR SELECTION Upon receiving the request, the Generator Selector selects an appropriate generator based on the service(s) selected and the appliance description. iCrafter

21. 3. GENERATOR PROCESSING The Generator Processor runs the selected generator with access to all context (appliance, service, environment), which outputs a UI description. iCrafter

22. 4. USER INTERFACE RENDERING The UI description is sent to the UI Renderer on the appliance, which renders the interface and handles user interactions, including service invocations over the EH. iCrafter

23. iStuff • Toolkit to prototype physical UIs • Flexible, lightweight components • Protocol independence • Platform independence with cross-platform capabilities • Ease of integration with existing applications • Support for multiple simultaneous users • Built on iROS • iStuff wireless devices (e.g., iButton, iSlider, iLight) send/receive events to/from Event Heap via a proxy • PatchPanel translates events (e.g., iButton: light on | light off) • Apps get events from devices via PatchPanel + Event Heap

24. iStuff Device Wireless connection iStuff component Transceiver Application PatchPanel Proxy TCP/IP connection Event Heap iStuff Architecture

25. iSlider iButtons iMike iStylus X10 RF iDog iMouse Anoto Pen iBuzzer iLight iSpeaker iStuff Devices Input Output

26. Aura • CMU “Distraction-Free Ubiquitous Computing” theme: • Mobility: allow users to move computational tasks from one place to another • Maximize use of available resources • Minimize user distraction & drains on attention • Personal information aura: spans wearable, handheld, desktop, & infrastructure devices • Requires new system design: hardware, network, OS, middleware, & UI support

27. Aura Research Framework • Users • Tasks • Services • OS / Network • Physical Devices Research Examples • UI / Agents / Speech • Task-driven computing • Rapid service configuration • Energy-aware adaptation • Multi-fidelity computation • Nomadic data access • Intelligent networking • Power-aware devices • Wearable computers

28. Aura Example Scenario • Fred prepares presentation & is late • Fred gets PDA & Aura transfers slides to it • Finishes slides on PDA walking to meeting • Aura finds location of meeting from calendar & uploads slides to projection machine • Aura recognizes unfamiliar meeting attendees faces & skips budget slides

29. Aura Architecture • User tasks are represented as set of distributed services • Database query interface to access distributed services • Easily search mixed environments for needed services • A key service is People Location Service • Environments self-monitor & re-negotiate task support given runtime variation of capabilities & resources • Can detect when task requirements (e.g., min response time) aren’t met, & search & deploy alternative configurations to support task • Uses software architecture models (graph of components & connectors) for runtime adaptability

30. Aura Architecture • Task Manager: tracks user context, environment, & task changes • Context Observer: provides info on physical context & reports relevant events Task & Environment Managers • Environment Manager: handles access to distributed services • Suppliers: provide services for tasks (text editing, video playing,...)

31. Privacy in Aura • Aura services depend on user location info • Introduces privacy concerns • Location policies: define who gets location info, where, when, and at what granularity • e.g., “Bob is allowed to know Alice’s location when she is in NSH” • Similar to Bell Labs’ Houdini system’s “rules” • Not based on how user’s actually disclose location

32. GAIA • Infrastructure for Active Spaces, UIUC • Like Aura: • Distributed services architecture accessed via structured queries • Main difference: • Focus on user customization of apps based on resources in current space

33. Toolkit Categories • Ubicomp infrastructure • Context-aware • Peripheral displays • Evaluation

34. Context • Context is key to ubicomp environment that reacts correctly to user’s current situation • Definition: Info used to characterize the situation of a person, place or object relevant to the interaction between a human & a computational service • Primary context types: • Identity (who), Location (where), Time (when), Activity (what) • Secondary context types: • e.g., for identity: email address, phone number, birthdate, etc

35. Context-Aware Applications • Definition: Uses context to provide relevant info and/or services to the user, where relevancy depends on the user’s task • 3 categories: • Present information & services to users • e.g., nearby wireless networks, directions • Automatically execute services • e.g., phone vibrates when in meeting • Tag info with context for later retrieval • e.g., class lectures and notes

36. Toolkits for Context • 3 main approaches: • Widgetmodel • Context Toolkit • Dey, Abowd, & Salber, 2001 • Based on architecture of GUIs • Distributed services model • Context Fabric • Hong & Landay, 2001 • Based on client-server dialog • Blackboardmodel • iRoom • Winograd, 2001 • Based on artificial intelligence apps (Engelmore, 1988)

37. Widget Model: Context Toolkit • Operating system view • Abstraction to hardware (sensors);data is secondary • Similar to GUI input handling: uses widget abstractions for handling context input • Uses 3 abstractions: • widgets, interpreters & aggregators

38. Benefits • Separates semantics from low-level input sensor handling • Apps can focus on context, not sensors or analysis of sensor input • Allows re-use of context widgets, interpreters & aggregators

39. Drawbacks • Tight coupling – less robust to component failure • Single-manager (Discoverer) control – less flexible

40. Context Toolkit Architecture

41. Widgets • Encapsulate info about a single piece of context (e.g., location or activity) • Hide details of underlying sensors from apps • Provide customizable & reusable building blocksof context sensing • Provide historical context info

42. Widgets • Include optional services, or actuation of output in environment • Like Interactors: responsible for input & output • e.g., light widget could sense light level & adjust lights accordingly • Get context by subscription or polling • Callback methods used to notify subscribers

43. Aggregators • Widgets + ability to aggregate context info • Collect sensory info about an entity (person, place, thing, etc.) from multiple sources into one widget • Hide more complexity about context-sensing mechanisms by combining multiple sensors • Enable maintainability and efficiency

44. Interpreters • Abstract sensor info • Use lower level sensory information to deduce higher level info • e.g., identity + location + sound sensors  “is there a meeting?”

45. Putting an App Together • Discoverer enables apps to locate and subscribe to context components • Distributed infrastructure uses p2p comm.

46. Distributed Services Model • Database view (vs. OS view of CTK) • Data is primary, hardware is secondary • Focus on how data is modeled, distributed, protected, & used • Middleware technologies that can be accessed through a network • Example: infrastructure=Internet; service=DNS • Any kind of device or application can use context gathering & processing services by adhering to predefined data formats & network protocols

47. Benefits • Interoperable: independent of hardware platform, OS, & programming language • Decoupling: Sensors, services, & apps can be changed w/o affecting each other • Simpler devices: sensors, processing power, data, & services w/in infrastructure shared

48. Drawbacks • Applications must poll for data (proactively) • Servers are specialized per sensor

49. Context Fabric • Includes 3 pieces • Distributed data model of people, places, things • each device has a space: private data goes to local space • multiple views of context (mine, yours, theirs) • Context Specification Language (like SQL) • apps specify what context info they need in uniform way • e.g., "What are the nearby movie theaters?” • Context service as API (per device) • interpret CSL queries and provide context info

50. Privacy in Context Fabric • Decentralized architecture allows it to capture, store, process, & share in privacy-sensitive manner • Capture, store, & process data on user’s device • Provides mechanisms for end-user control of sharing • Plausible deniability built-in • e.g., if request for context info denied, system can reply “unknown”