Huddle and Information Triggers: Helping Users Automate Complex and Temporally-Distant Tasks

1. Huddle and Information Triggers:Helping Users Automate Complex and Temporally-Distant Tasks Jeffrey Nichols IBM Almaden Research Center UC Berkeley June 14, 2007

2. Today – Two Mini Talks • Huddle • Automatic generation of interfaces for systems of connected appliances • Part of my dissertation work on the Personal Universal Controller • Information Triggers • Helps users recall important information or tasks at the moment they are important • New project at IBM Almaden Research Center • In collaboration with Jake Wobbrock at the University of Washington

3. The Multi-Appliance Problem

4. The Multi-Appliance Problem, cont. How can we provide improved interfaces for systems of connected appliances?

5. Control of Multiple Appliances Universal Remote Controls Built-in Interfaces Custom Installations

6. Huddle • Improved multi-appliance interfaces based on content flow Flow-Based Interface Aggregate User Interfaces

7. Personal Universal Controller (PUC) • Automatically generates interfaces for remotely controlling appliances [Nichols, UIST 2002] • Specification language for describing appliance functionality Dependency information that describes the availability of functions based on appliance state • Automatically generate new interfaces to be consistent with previous interfaces [Nichols, CHI 2006] Knowledge base that identifies and stores similarities between appliances

8. Related Work • Roadie [Lieberman, IUI 2006] Goal-oriented natural language interface for appliances Uses planner to configure appliances based on common sense database (EventNet) • ICrafter [Ponnekanti, Ubicomp 2002] Builds aggregate user interfaces based on “service interfaces” Hand-coded interface aggregators Limited automatic generation capabilities • Omojokun, et al. [Omojokun 2006] Examined how remote control interfaces might be generated from recordings of actual usage Recorded remote control usage with IR receiver • General Work on Interface Generation • Task-based work: TERESA [Mori 2004], XIML [Puerta, IUI 2002], others… • SUPPLE[Gajos, IUI 2004]

9. Huddle Overview • Introduction • Content Flow Modeling • Flow-Based Interface • Aggregate User Interfaces • Discussion and Future Work

10. Content Flow Model • Home Theater DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

11. Modeling Tasks • Home Theater: Watching Television DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

12. Modeling Tasks • Home Theater: Watching Television DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

13. Modeling Tasks • Home Theater: Watching Television DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

14. Modeling Tasks • Home Theater: Watch DVD DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

15. Modeling Tasks • Home Theater: Watch DVD + Copy Tape DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

16. Modeling Tasks • Presentation Room VCR PowerPoint Projector Laptop Task Manager External Video Control Media Player

17. Modeling Tasks • Presentation Room: Presenting PowerPoint VCR PowerPoint Projector Laptop Task Manager External Video Control Media Player

18. Where does the model come from? DVD Receiver BroadcastTelevision TV VCR #1 VCR #2

19. Where does the model come from? DVD Receiver TV VCR #1 VCR #2

20. Where does the model come from? DVD Receiver TV VCR #1 VCR #2

21. Flow-Based Interface (FBI) Flow-Based Interface • Allows user to manage high-level tasks • Quick specification of flows • AI Planner automatically configures appliances • Question/Answer interface to address conflicts

22. Flow-Based Interface

23. Flow-Based Interface

24. Flow-Based Interface

25. Flow-Based Interface

26. Flow-Based Interface

27. Flow-Based Interface

28. Flow-Based Interface

29. Flow-Based Interface

30. Flow-Based Interface

31. Aggregate User Interfaces (AUIs) Aggregate Controls for “Watch DVD” • Allows user to manage low-level details • Control or setup of active flows • General setup of all appliances • Merged functions

32. Aggregate User Interfaces • Three kinds • Control of Active Flows DVD Receiver BroadcastTelevision TV VCR #2 VCR #1

33. Aggregate User Interfaces • Three kinds • Control of Active Flows DVD Receiver BroadcastTelevision TV VCR #2 VCR #1

34. Aggregate User Interfaces • Three kinds • Control of Active Flows DVD Receiver BroadcastTelevision TV VCR #2 VCR #1

35. Aggregate User Interfaces • Three kinds • Control of Active Flows DVD Receiver BroadcastTelevision TV VCR #2 VCR #1

36. Aggregate User Interfaces • Three kinds • Control of Active Flows DVD Receiver BroadcastTelevision TV VCR #2 VCR #1

37. Aggregate User Interfaces Volume for TV Volume for Receiver • Three kinds • Control of Active Flows “Copy Tape” “Watch TV using TV speakers” “Watch TV using Receiver speakers”

38. Aggregate User Interfaces • Three kinds • Control of Active Flows • Setup of Active Flows Two screens of “Watch DVD” setup

39. Aggregate User Interfaces • Three kinds • Control of Active Flows • Setup of Active Flows • General Setup of all appliances Two screens of General Setup for a home theater config.

40. Aggregate User Interfaces • Three kinds • Control of Active Flows • Setup of Active Flows • General Setup of all appliances • Merged Functions • Most functions shouldn’t be merged • Some should: clock, language, sleep timer… • Future work… Baudisch’s flat volume control General merging Some merged functions

41. Discussion • How can our use of content flow be extended? • Lighting in a presentation room • Video-conferencing rooms • Outside the home: factories & offices • Kitchen?

42. Future Work • Extending to more detailed content types • Home theater: support for HD signals, encoded audio, etc. • Setup and Troubleshooting • Tool for specifying initial wiring • Helping with Problems • More aggregate interfaces • Usage-based aggregate? • Usability evaluations

43. Information Triggers

44. Information Triggers • Motivation It’s easy to find stuff on the web It’s even easy to re-find something you’ve previously found... ...if you remember to re-find it when you need it “Participants [in a re-finding study] complained that they often forgot to look for an information item until the period of its usefulness had passed.” (Jones et al. 2005)

45. Information Triggers • What we’ve built so far • A Firefox extension that allows people to associate web pages with future events at which that page will be particularly relevant • When the event occurs, the web page is automatically opened for the user • Goal is to make triggers as easy and lightweight to configure as possible • Supports three different types of triggers: • Date & time • Arrival of an e-mail • Visit to a web page

46. Beyond Current Reminder Systems • Existing reminder systems provide a short message about their topic, but info triggers provides the information itself • Context-based reminder systems (e.g. Cybreminder) have mainly used location for context

47. Scenarios • Waiting to login to a website to configure account information (date & time trigger)

48. Scenarios • Resuming a task that can’t be completed in one sitting, e.g. waiting for manager approval (new e-mail trigger)

49. Scenarios • Opening a note-taking site when visiting a page with specific keywords (page visit trigger) information retrieval web pages

50. Scenarios • Delaying a task until a leisure period (page visit trigger)