Gordon Bell www.MyLifeBits.com www.TotalRecallBook.com 8 March 2010

1. Context Aware: TNBT, … Thing?The Big Bet of (CaC) is that The Next Big Thing is:Dramatic Increase of Human Productivity… usingcomputational skills provided by computers as well as …sensing details system that will augment people experience in interaction with reality Gordon Bell www.MyLifeBits.comwww.TotalRecallBook.com 8 March 2010

2. CA: Context Aware Computing… Assume it is TNBT; then is it? • New capabilities based on these platforms or technologies? • All the bits in your e-Memory (Bell-Gemmell PoV) • “Personal_state(t)”: Loc/pt. x,y,z,ρ,θ,φ; HR,BP, stress; T, Δ’s • Social Networking • Cloud enabling new client platforms • All the above • New product(s)? New Service(s)? Features? Featurettes? • Revenue? Business model? • CA-thing is very specific as the intersection of ≥2 “dimensions” • CA-ness can enable a function or a “point” product/service • What problem(s) does having a/more context solve? • How? • Why?... More context, more opportunities for action? • When in doubt, create a taxonomy!

3. CA and CAO: GB thoughts re. definitionMS makes $s as an opportunity enabler • Context Awareness … the intersection of two or more “state dimensions” providing an opportunity for some kind of action e.g. meeting, insight, path to take/avoid, something to buy… • Time and place • Place and place (e.g.meet friends, shopping) • CA is the detection & creation of CAO’s… SEARCH! • More state => more opportunities for action • More public state => interpersonal opportunities • More mobility (physical or cyberspace) => opportunity • Availability of entire life => opportunity

4. What is Context? (ISO) Users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.* * As defined by ISO 9421-11

5. Scope: User 9/2008 • ContactIdentity Data • Name, addresses, emails, phones, etc. • Biometric • Goals & Tasks • Activities Life Experience & Personal Interests • Health • Health History • Current Health “Context combines users, their goals and tasks, infrastructure (hardware, software, materials), physical and social environment in which the product is used.” Psychological Type & Current Emotions • Traditions • Person • Family • Social Groups • Nation Social Relations Microsoft Confidential

6. My context = My e-Memory + my current state

7. Mail, dcs, books, web, journals, podcasts… Vibe • Social: iM, facebook, … calls, conversations… • Financial: $ transaction • Health: PHR, activity, diet • iLife {photo/movie TV music}, trip trajectories • Logs … for all “lives” • …for all activities Memory centric view: Context = eMemory + bioMemory (aka State of mind) + environment Memex aka MyLife Bits aka eMemory “Lives ...” Web eMemory =ground truth bioMemory = metadata... “Personal_state(t) Loc/pt. x,y,z,ρ,θ,φ HR, BP, stress Temp, Δ’s… ???s

9. Realizing Memex… Digital Capture, Storage, and Utilization of All Personal Informationwww.MyLifeBits.com

10. Room Capture Import files MyLifeBits Shell GPS import & Map display Voice annotation tool VIBE logging Text annotation tool SenseCam Screen saver Legacy applications Internet Radio capture & EPG PocketPC transfer tool MAPI interface TV capture tool Browser tool files Outlook interface Legacy email client Telephone capture tool PocketRadio player IM capture TV EPG download tool MyLifeBits Platform c2005 MyLifeBits store database

11. Using an e-Memory to augment b-memory(how did we get the bits for context?) • 1998-2001 > Capturing the high order past & present bits • Instigated by Raj Reddy’s Million Book Project • Stimulated by telepresence, Billg “someday you can recall everything…”, idea of tidiness, design for CyberMuseum (Computer History Museum) • “MyLifeBits is about converting everything to bits” • 2001-2005 > Getting bits from e-Memory… as many bits as possible • Stimulated by Bush, Gray (database) … • MyLifeBits Project moves to SQL. Acquire “context” from everything we can • “MyLifeBits is a transaction processing system for a life” • 2005: Memex Project. Redmond := Gemmell and Lueder; Project := 0 • 2009: Total Recall := Book_write_&_publish (Bell_and_Gemmell) • “Mylifebits is an e-memory… to aid bio-memory for life &afterlife” • Right Period to create “Memex” based on technologies • We can encode it: Scanning and recognition problem (worked on e-Paper) • We can store it: Terabyte (we thought that would store a digital life) • We can find it: Search came into its prime

12. We think life-Blogging is nuts! not life-bloggiNG… life-bloggiNG=> Greater CAO

14. Capturing every step. If I know where I am, what good does it do me or anyone?

15. Health: Context matters! When you are about to … Hospital, GP, specialist, lab, dentist, chiropractor, pharmacist… You need to have control of your records Health Recordsby the individual, for the individual

16. Cell Phone as a Platform for HealthcareTechFest 2009 SCS Where was I, what was I doing, who was I with, (what meeting was I in) when I had a high HR or high stress indicator ?

17. The killer app. Health“those spots look a bit better, don’t they?”

18. NavigenicsReport

19. 1942 Mayo Clinic visit

20. Bio-Metric Life-logging

21. EKG just before pacemaker 040629 A flaky connection between chambers!

22. Capturing every heartbeat • 72.6 beats/min; 38.16 Million beats/year • 3.13 billion beats year • The important number is 4-4.5 years, or ETS • Battery life: the expected trip to surgery!

23. Bypass Feb 83

24. Tracking Test Results……. It’s all context… effecting mental state

25. Philips in-home devices

26. Bodybugg

27. Smart materials

28. In-body health sensing Nanobot in the bloodstream pillcam EndoSure Wireless Pressure Sensor in an aneurysm sac

29. Health Monitoring:“Your husband just died, … here’s his black box”

30. Context awareness actions are based on f(which lives you are in right now,what activity are you engaged in,your accessible digital life)

31. Family Parents, Siblings, Children Friends Clubs and Organizations Educational Institutions Work organizations … Projects “Lives” aka “Lifebrowsing” “Personal Role Management” “Autobiographical Memory” for structuring life into organizational groups; general events; and event specific knowledge time periods Who: which Life What: activity Where: you are Who: Lives… Lifetime periods Which lives Contacts Office Documents Communication email Profession specific Finance & legal Health records On body recording Books Instructional Mat. Music Photos Videos Location Everything I see Everything I hear What: Degree “being digital”. Your “digital life” Office Health Learning Ambience Person Home Job Where: “Location” of your digital life PC … Home … Cellphone … Cloud… Public Cloud Degree of privacy

33. Personal LifeLog Applications Diary/Journal Self Tutor Mentor Advisor Meeting Prep Personal Assistant Babysitter Financial Manager Companion Caretaker Medical Manager Parole Officer Photo Album Application used by: Assistant for Elderly Personal Flight Recorder Autobiography Captain’s Log Conservator Biography Baby Book Trustee Obituary Personal Proxy Executor Others Application controlled by: Others Self Doug Gage, ARPA LifeLog

34. My Bits… then (c2000) and now (c2010) • Single system… • Where are they? • How owns them? • Who can access them? • How do I keep themprivate? • … Bush’s Memex c1945

35. My Bits 2010 Dist’d state & Context Memex Store From Susan Dumais c2005 (I think)

36. Where are all of your bits and meta-data?Islands of content & pointers • Folder/file hierarchy • Outlook (msg, contact, calendar, task); Journal & VIBE • OneNote; EverNote • Social/professional: iM, Skype, Connection Manager, LinkedIn, Facebook, Twitter, etc. • Money… financial institutions (bank, broker, insurance) • HealthVault … BodyBugg, … medical providers • Music: iTunes, Media Player, Media Center, …Zune • Photos: Live photo gallery, Picassa, Flicker • Media Center • iLife={iPhoto, iMovie, iTunes, iWeb, iBook}

40. MyLifeBits Project aka MemexInfluencing products and research • Products & Research Efforts • DARPA LifeLog; DARPA Assist; DARPA CALO and SRI spinoff SIRI • EverNote as a OneNote competitor but focused on e-Memory, • reQall a voice reminder and memory (knows about time and space) • http://www.lifebio.com/ Capture your story for immortality; www.famento.com - your family history • Stanford SALT project to Archive the faculty • IBM Pensieve (IBM Research Personal Memory Organizer) • Korean and Japanese Research Insitutes both Dedicated to MyLifeBits • UK Memories for Life Project… • Caprica TV prequel to Battleship Galactica • Aimee Baldridge, Organize Your Digital Life: How to Store Your Photographs, Music, Videos, and Personal Documents in a Digital World • MSR Global Heritage Project • Conferences: • ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE) • Society for Imaging Science &Technology Conference on Archiving, April 2004 • First Digital Lives Conference, British Libraries, Feb 2009 • National Conference on Archive, West Midlands, UK, November 2009 • Saving Our Present for the Future: Personal Archiving 2010, archive.org, Feb 2010 • Cultural Heritage Track for ACM Multimedia

41. A couple of recent Google slides…Comment by Marc Benioff “Enterprise software had traditionally been built by engineers who worked in corporate office parks with little or no interaction with the customers until a pre-determined “five-year plan” was complete. This isolationist existence has seriously stymied innovation in the technology industry. Companies were not moving fast enough for customers.” -Marc Benioff, Behind The Cloud

42. Cloud-client computing: research role? • The PC is a less relevant environment in the face of the cloud and especially IF/When office apps go there. • Many users would be happy to have apps run in the cloud and have their data there permanently… • CS Research needs to be working and training students for the cloud environments • Bing Product cycle: 4 months; Win/Office: >36 mos. • Idea to use is 9X faster; a published idea ships immediately • R > AD …. > Product groups > Product (3-5 years at best) • Transfer is fundamentally flawed. Product groups can just look at the competitors

43. Memex: A Supported and Active Research Platform for MSR’s Research*Ken Wood, Roger Lueder, Jim Gemmell, Mary Czerwinski, Gordon Bell • Supported Platform created and based on: MyLifeBits, VIBE, SenseCam, Digital Geographics, and MLB University Research Program • The narcissistic effect: “By having a common platform in use and built on, we can expect exponential increases in functionality through cross-app use and common schema, plus incremental improvements to performance and reliability. This is the nature of all such systems.” • The Memex program has four components or goals: • continued research on the platform that includes validating WinFS efficacy --especially for the PC for personal data, real time, and a person’s transactions; • research coming from Redmond, Cambridge, product efforts and universities; • a pre-WinFS platform for building and testing new database-based apps in a real use environment that allows us to”eat our dogfood”; and • live use that creates stability, understanding, and new research ideas, problems, and attraction for more apps. *Proposal 6/15/2005

44. End

45. The Agenda for the Lifetime e-Memory • Guarantee that data will live forever! “dear appy” problem • Data Aggregation from the myriad of data places: cloud, work, home, social sites • More content: Cheap, easy, and data-rich (e.g. time, place) capture of: GPS and time everywhere Paper capture has to be as easy as discarding (scanner/shredder) Personal meeting capture... Telephony and audio capture with indexing– speech to text E-book…e-magazines & journals need to have critical mass! Media Center compatible for entertainment (photos, video, TV, radio) Connection to other sensor nets in home, auto, and elsewhere as they slowly come into existence • Annotations/meta-information add every-increasing valueEasy annotation for aiding search and it becomes the content • Automatic classification and organization into taxonomies, facets, etc. • Content analysis (critical for photo & video!) • Information control: privacy, security, expunge/deniability,… • Having to be schizophrenic or have a lobotomy when leaving a “life” • Onedbase for everything (articles, books, conversations, ... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? • The “killer apps”: Alzheimer, immortality, surrogate memory? • Clean living, cloaking, versioning, structure to contacts (family, org charts) • GUI’s to improve use (e.g. time to learn, use, retention)

46. HMI 2009: 1.3 Thr;3.6 Zby; 10.8 PwordsReport on US Consumers, Bohn and Short, UCSD

47. From HMI Report on Info Sources in the US