Unobtrusive Augmentation of Everyday Environments with Computing

1. Unobtrusive Augmentation of Everyday Environments with Computing Hans-W. Gellersen Lancaster University Department of Computing Ubiquitous Computing Research

2. Computer-Augmented Environments Background • Ubiquitous Computing: more computers embedded everywhere • Augmented Environments: primary attention to the environment Two (or Three) Projects • Pin&Play: A wall that is also a network • Load sensing: Table surfaces that are also sensor/input device • (Smart-Its: A prototyping platform for augmented artefacts)

3. Background Ubiquitous Computing • “The Computer of the 21st Century”, Mark Weiser, SciAm 1991 • Mass deployment of computing in everyday life • New paradigm for how computers are used: “each person is continually interacting with hundreds of nearby interconnected computers without explicitly attending to them” • also: Invisible Computing, Ambient Intelligence, etc Augmented Environments • “Back to the Real World”, CACM July 1993 • Primacy to interaction with accustomed physical/tangible environment • Leverage the meaning/purpose of our designed environments • Introduce computing in the background

4. Design Perspective Toward a hybrid design • Combine unique capabilities of computer technology with properties of physical environments • Focus on foreground activity: how people interact with their designed environment – preserve familiarity and accustomeduse Interaction with Physical Artefacts / Environments • Physical/Tangible Interaction • Physical affordances: suggesting and guiding action • Distributed interaction: actions across artefacts / space • Spatial/ambient interaction • Spatial organisation of action/communication • Ambient interaction: “spatial attention model”

5. Technical Perspective Computers in the Background • Computers as secondary artefacts (embedded/situated) • Embedded: (only a) part of some other artefact • Situated: meaningfully placed, designed for specific context(“context-made” rather than “context-aware”) The Environment is the Interface • Build upon affordances of the primary artefact or environments • Don’t break with accustomed uses and familiar concepts • Physical I/O (sensors/actuaors) to tie computers to entities in the physical environment • Networking to enable coherent interaction • Spatial interaction, proximate networking, etc • Allow for new interactions/relationships across parts of the environment

6. Bishop’s Marble Answering Machine • Physical interaction with digital information

7. Wellner’s Digital Desk • Why a desktop metaphor when you can have a real desktop • Seamless transitions: physical and digital interaction

8. Computer-Augmented Environments Background • Ubiquitous Computing: more computers embedded everywhere • Augmented Environments: primary attention to the environment Two (or Three) Projects • Pin&Play: A wall that is also a network • Load sensing: Table surfaces that are also sensor/input device • (Smart-Its: A prototyping platform for augmented artefacts)

9. The Pin&Play Network Concept • “The wall as network bus for the things attached to it” • A new type of network to connect everyday objects on common surfaces such as boards and walls Use of familiar interaction • “pinning objects to the wall” “pinning nodes to the network” Preserve original functionality • Embedded computing does not displace original uses

10. Pin&Play Components Network Surface • Common surface augmented with conductive material to create two-dimensional network medium Connectors • Pushpin-like physical connector for socket-less attachment of objects Pin&Play Objects • any type of device/object with embedded computing, mounted onto connector Network: • “Pin&Play” behaviour: discovery of objects when they become attached

11. Pin&Play Noticeboard • First demonstrator build for proof of concept

12. Pin&Play Noticeboard Pin&Play Surface • Corkboard augmented with two conductive sheets • Ground layer on top, data/power layer hidden, cork as insulator • Low cost, off-the-shelf,deployable at large-scale Pin&Play Connectors • Simple connector board with pushpin for two separate connection points

13. Pin&Play Noticeboard Network Technology • 1-wire bus, Dallas MicroLAN • 16300 bits/s Pin&Play Objects • “Smart Notification Pin”: iButton and switchable LED • Time-in-a-can iButton: memory, internal calendar and clock Switch Switch Time-in-a-Can Time-in-a-Can

14. Pin&Play Noticeboard User interaction • insert or remove pin • network detects change • protocol to determine pin with highest priority Network control • External laptop connected as 1-wire network node • runs network controller • used to pre-set pins with priority and deadlines

15. Scaling up … … bigger boards, more nodes

16. Testing versatility … … different objects, different applications Pin&Play Lightswitch • place it where you like!

17. Pin&Play Technology Research • Network surface development • Simple and robust protocol design, zero maintenance • Scalability and density (initial target: 25 nodes/sqm) Application Research • Augmented noticeboards and other interactive surfaces • Embedded home control buses • Networking and free placement of controls (light switches, appliance controls etc.) • Communication bus for wall-attached artefacts • Clocks, calendars, sensors, digital picture frames, …

18. Load-Sensing Surfaces Concept • Gravity is ubiquitous: no physical thing can escape it • Use this force to build interfaces between the physical and the virtual • Augment common surfaces (floors, tables, shelves): this is where gravitation pulls objects to • Technology: variety of load sensors and pressure gauges with different characteristics

19. Load-Sensing Surfaces Principle • Augment surface at the corners • Force applied (e.g. by weight of an object, or explicit pressure) is detected as load depending onposition of the pressure point … more than just weight • surface detects weight/pressure • But also position, traces, etc

20. Load Sensing Control Basic load sensor • e.g. your kitchen scale Load-sensing surface Force Load Scale Control F1 F2F3 F4 Load Surface Centre of Gravity Control DF1 DF2DF3 DF4 Load Change Surface Position

21. Load Sensing Control Basic event detection • Object placement • Object removal Further event processing • Detect movement • Detect specific events • Detect Object ID/Class Tracking movement • Detecting traces on surfaces Tracking objects • Tracking across surfaces • Correlation of events • Grouping events associated with the same object Surface Event Sensor Load Change Event Position Control SurfaceMovementTracker Load Change Trace onsurface Position Control Observing Event Trace acrosssurfaces Event E1 Event En

22. Load-Sensing Surfaces “Weight Lab” • Various augmented surfaces • Floor: 240 x 180cm, up to 800kg load • Larger table: up to 200kg • Coffee table: up to 8kg, highly sensitive • Shelves and trays Floor with embeddedS-load cell

23. Load-Sensing Surfaces Sensor board with wireless communication Augmented tables

24. Load-Sensing Surface Context Acquisition • Weight of objects • Detection depends on sensor range (i.e. small weights not detectable on heavy-load surface) • Application: object identification (classes/ instances) • Position of objects • cm-level accuracy (depend. on surface size) • Table can be pre-loaded • Multiple objects can be positioned if placed non-simultaneously

25. Load-Sensing Surface Context Acquistion • Beyond weight and position: basic events derived from signal analysis over short time (low-cost analysis in time domain) New object knocked over Object put down removed

26. Load-SensingSurfaces Context Acquistion • Tracking of people/objects • Prediction ofactivities

27. Load-sensing surfaces Surfaces as Context-sensor • Object position and movement • Human activity modelling • Many applications: track objects, detect breakdown of routines • Implicit human-computer interaction ‘Everyday’ Surfaces for explicit interaction • Challenge: when do we interact with the table, and when with the computer (compare speech input problem)

28. Load-Sensing Surface Surfaces as Interaction Device

29. Load-Sensing Surface Surfaces as Interaction Device moveleft moveleft touch and move right cup book click andrelease click click

30. Load-Sensing Surface Surfaces as Interaction Device

31. Conclusion Unobtrusive infrastructure • Pin&Play and Sensor Surfaces are example for infrastructure integrated with everyday use • Comparatively low-cost, very deployable Added functionality • New forms of interaction • New applications • Not compromising existing use and familiarity

32. Future Work More interdisciplinary work with designers • e.g. Equator project with RCA • series of product design concepts for sensor-tables A different story: how to investigate such systems ? • Inevitably involves hardware prototyping (awkward as the aim is not to advance hardware) • Hardware for demonstrators is hard to reproduce • Need support for physical prototyping Smart-Its • Small wireless sensor devices designed for rapid prototyping