Prof. Marc Davis University of California at Berkeley School of Information Management and Systems

1. Prof. Marc Davis University of California at Berkeley School of Information Management and Systems Garage Cinema Research http://garage.sims.berkeley.edu Garage Cinema Research and the Future of Media Technology

2. The Four Questions • How many of you read text every day? • How many of you write text every day? • How many of you watch video (e.g., television, movies, DVDs, internet video, etc.) or look at photographs every day (be honest)? • How many of you make video or take photographs every day?

3. Today’s Agenda • Introductions • Garage Cinema Research Overview • Garage Cinema Research Projects • Media Streams Metadata Exchange • Active Capture • Adaptive Media • Mobile Media Metadata • Social Uses of Personal Media • Garage Cinema Research Future • Garage Cinema Research and Yahoo!

4. Today’s Agenda • Introductions • Garage Cinema Research Overview • Garage Cinema Research Projects • Media Streams Metadata Exchange • Active Capture • Adaptive Media • Mobile Media Metadata • Social Uses of Personal Media • Garage Cinema Research Future • Garage Cinema Research and Yahoo!

5. Who Am I? • Assistant Professor at SIMS (School of Information Management and Systems) • Background

6. What Do I Do? • Research • Director of Garage Cinema Research • Projects in Media Metadata, Active Capture, Adaptive Media, Mobile Media Metadata, and Social Uses of Personal Media • Executive Committee Member and Co-Founder of the Center for New Media • Affiliated Faculty Member of the Berkeley Institute of Design • Associate Editor of IEEE Multimedia and ACM Transactions on Multimedia Computing, Communications, and Applications • Teaching • Information Organization and Retrieval • Multimedia Information • Digital Media Design Studio • Foundations of New Media

7. Today’s Agenda • Introductions • Garage Cinema Research Overview • Garage Cinema Research Projects • Media Streams Metadata Exchange • Active Capture • Adaptive Media • Mobile Media Metadata • Social Uses of Personal Media • Garage Cinema Research Future • Garage Cinema Research and Yahoo!

8. Garage Cinema Research • Research and develop technology and applications that will enable daily media consumers to become daily media producers • Theory, design, and development of digital media systems that • Create descriptions of media content and structure (metadata) • Use metadata to automate media production and reuse

9. What is the Problem? • Today people cannot easily find, edit, share, and reuse media • Computers don’t understand media content • Digital media are opaque and data rich • We lack structured representations • Without metadata, manipulating digital media will remain like word-processing with bitmaps

10. Signal-to-Symbol Problems • Semantic Gap • Gap between low-level signal analysis and high-level semantic descriptions • “Vertical off-white rectangular blob on blue background” does not equal “Campanile at UC Berkeley”

11. Signal-to-Symbol Problems • Sensory Gap • Gap between how an object appears and what it is • Different images of same object can appear dissimilar • Images of different objects can appear similar

12. Computer Vision and Context • You go out drinking with your friends • You get drunk • Really drunk • You get hit over the head and pass out • You are flown to a city in a country you’ve never been to with a language you don’t understand and an alphabet you can’t read • You wake up face down in a gutter with a terrible hangover • You have no idea where you are or how you got there • This is what it’s like to be most computer vision systems—they have no context • Context is what enables us to understand what we see

13. METADATA Traditional Media Production Chain Metadata-Centric Production Chain M E T A D A T A PRE-PRODUCTION PRODUCTION POST-PRODUCTION DISTRIBUTION

14. Research Projects • Media Streams • A framework for creating metadata throughout the media production cycle to enable media reuse • Active Capture • Automates direction and cinematography using real-time audio-video analysis in an interactive control loop to create reusable media assets • Adaptive Media • Uses adaptive media templates and automatic editing functions to mass customize and personalize media • Mobile Media Metadata • Leverages the spatio-temporal context and social community of media capture to automate metadata creation for mobile media • Social Uses of Personal Media • Analysis of social uses of media to predict future uses and shape the design of next-generation personal media devices and applications

15. Automated Media Production Process Active Capture Automatic Editing Personalized/ Customized Delivery 1 3 4 Web Integration and Streaming Media Services Adaptive Media Engine Flash Generator XHTML MMS Annotation of Media Assets Annotation and Retrieval 2 Asset Retrieval and Reuse Print/Physical Media Reusable Online Asset Database

16. The Future of Media Technology • Media capture devices become programmable and networked • Metadata creation and use become integrated throughout media production and reuse • Media production changes from being a mechanical process to a computational process • Media become programmable and networked • Daily media consumers become daily media producers

17. Today’s Agenda • Introductions • Garage Cinema Research Overview • Garage Cinema Research Projects • Media Streams Metadata Exchange • Active Capture • Adaptive Media • Mobile Media Metadata • Social Uses of Personal Media • Garage Cinema Research Future • Garage Cinema Research and Yahoo!

18. Research Projects • Media Streams • A framework for creating metadata throughout the media production cycle to enable media reuse • Active Capture • Automates direction and cinematography using real-time audio-video analysis in an interactive control loop to create reusable media assets • Adaptive Media • Uses adaptive media templates and automatic editing functions to mass customize and personalize media • Mobile Media Metadata • Leverages the spatio-temporal context and social community of media capture to automate metadata creation for mobile media • Social Uses of Personal Media • Analysis of social uses of media to predict future uses and shape the design of next-generation personal media devices and applications

19. Media Metadata: Media Streams

20. Media Streams Features • Key features • Stream-based representation (better segmentation) • Semantic indexing (what things are similar to) • Relational indexing (who is doing what to whom) • Temporal indexing (when things happen) • Iconic interface (designed visual language) • Universal annotation (standardized markup schema) • Key benefits • More accurate annotation and retrieval • Global usability and standardization • Reuse of rich media according to content and structure

21. Research Projects • Media Streams • A framework for creating metadata throughout the media production cycle to enable media reuse • Active Capture • Automates direction and cinematography using real-time audio-video analysis in an interactive control loop to create reusable media assets • Adaptive Media • Uses adaptive media templates and automatic editing functions to mass customize and personalize media • Mobile Media Metadata • Leverages the spatio-temporal context and social community of media capture to automate metadata creation for mobile media • Social Uses of Personal Media • Analysis of social uses of media to predict future uses and shape the design of next-generation personal media devices and applications

22. Creating Metadata During Capture Current Capture Paradigm Multiple Captures To Get 1 Good Capture New Capture Paradigm 1 Good Capture Drives Multiple Uses

23. Active Capture Direction/ Cinematography Capture Interaction Active Capture Human Computer Interaction Computer Vision/ Audition Processing

24. Active Capture Setup

25. Active Capture

26. Active Capture: Reusable Shots

27. Research Projects • Media Streams • A framework for creating metadata throughout the media production cycle to enable media reuse • Active Capture • Automates direction and cinematography using real-time audio-video analysis in an interactive control loop to create reusable media assets • Adaptive Media • Uses adaptive media templates and automatic editing functions to mass customize and personalize media • Mobile Media Metadata • Leverages the spatio-temporal context and social community of media capture to automate metadata creation for mobile media • Social Uses of Personal Media • Analysis of social uses of media to predict future uses and shape the design of next-generation personal media devices and applications

28. Marc Davis in T2 Trailer

29. Evolution of Media Production • Customized production • Skilled creation of one media product • Mass production • Automatic replication of one media product • Mass customization • Skilled creation of adaptive media templates • Automatic production of customized media

30. Editing Paradigm Has Not Changed

31. Central Idea: Media as Programs Content Representation Producer Parser Media Media Parser Media Content Representation • Media change from being static data to programs • Shots are inputs to a program that computes new media based on content representation and functional dependency (US Patents 6,243,087 & 5,969,716)

32. Adaptive Media Design Space Author- Generated Compilation Movie Making Historical Documentary Movie Making Traditional Movie Making Structure Not Author- Generated Author- Generated Content

33. Adaptive Media Design Space Author- Generated Compilation Movie Making Historical Documentary Movie Making Traditional Movie Making Video Lego (structure is constrained) Structure Video MadLibs (structure is determined) Not Author- Generated Author- Generated Content

34. The Blank Page Approach

35. Captain Zoom IV MadLib™

36. Constructing With Lego™ Blocks

37. Video MadLibs Adaptive media template with open slots Structure is fixed Content can be varied Video Lego Reusable media components that know how to fit together Structure is constrained Content can be varied Video MadLibs and Video Lego

38. Research Projects • Media Streams • A framework for creating metadata throughout the media production cycle to enable media reuse • Active Capture • Automates direction and cinematography using real-time audio-video analysis in an interactive control loop to create reusable media assets • Adaptive Media • Uses adaptive media templates and automatic editing functions to mass customize and personalize media • Mobile Media Metadata • Leverages the spatio-temporal context and social community of media capture to automate metadata creation for mobile media • Social Uses of Personal Media • Analysis of social uses of media to predict future uses and shape the design of next-generation personal media devices and applications

39. Moore’s Law for Cameras 2000 2002 $400 Kodak DX4900 Kodak DC40 $ 40 SiPix StyleCam Blink Nintendo GameBoy Camera

40. Capture+Processing+Interaction+Network

41. Camera Phones as Platform • Media capture (images, video, audio) • Programmable processing using open standard operating systems, programming languages, and APIs • Wireless networking • Personal information management functions • Rich user interaction modalities • Time, location, and user contextual metadata

42. Camera Phones as Platform • In the first half of 2003, more camera phones were sold worldwide than digital cameras • By 2008, the average camera phone is predicted to have 5 megapixel resolution • Last month Casio and Samsung introduced 5 megapixel camera phones with optical zoom and photo flash • There are more cell phone users in China than people in the United States (300 million) • For 90% of the world their “computer” is their cell phone

43. Campanile Inspiration

44. Mobile Media Metadata Idea • Leverage the spatio-temporal context and social community of media capture in mobile devices • Gather all automatically available information at the point of capture (time, spatial location, phone user, etc.) • Use metadata similarity and media analysis algorithms to find similar media that has been annotated before • Take advantage of this previously annotated media to make educated guesses about the content of the newly captured media • Interact in a simple and intuitive way with the phone user to confirm and augment system-supplied metadata for captured media

45. MMM Demo Video

46. Context When Date and time Where CellID refined to semantic place Who Cellphone user What Activity as product of when, where, and who Content When was the photo taken? Where is the subject of the photo? Who is in the photo? What are the people doing? What objects are in the photo? From Context to Content

47. Space – Time – Social Space SPATIAL TEMPORAL SOCIAL

48. What is “Location”?

49. Camera Location = Golden Gate Bridge Subject Location = Golden Gate Bridge Camera Location = Albany Marina Subject Location = Golden Gate Bridge Camera Location vs. Subject Location

50. Kodak Picture Spot