drag-and-pop a technique for accessing remote screen content on touch- and pen-operated systems

1. drag-and-popa technique for accessing remote screencontent on touch- and pen-operated systems patrick baudisch & ed cutrell, dan robbins, mary czerwinski,peter tandler, ben bederson, and alex zierlinger microsoft research visualization and interaction research

2. goals • mixing touch screens/pen-input with large screens creates interesting new interface challenges • bringing target icons to the user (“drag-and-pop”) allows users to complete drag interactions faster • general theme: limiting interaction space to the display space at the user’s location can solve problems

3. touch and pen input renaissance PDAs Tablets Liveboards / Smartboards multi-display systems DynaWall, iRoom Smartboard wall connect tablet to external screen … touch/pen-input + multimon touch/pen input breaks

4. scenario 1: tables + screen tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse

5. scenario 2: dragging+bezels dragging across bezels in display wall is no problem for the mouse… …but a big problem when using pen/touch input

6. scenario 3: long distances dragging is designed for small screens…… but becomes time-consuming on large screens

7. drag-and-pop • users starts dragging icon towards a distant folder or application • icons of compatible type come towards mouse cursor • user drops icon with minimal motion • targets retract drag-and-pop works across bezels

8. demo… bringing target icons to the user allows users tocomplete drag interaction at the user’s current locations

9. scenario 1: tablet + screen filing icons into folder on external monitor

10. scenario 2: dragging+bezels dragging acrossbezels in display wall

11. scenario 3: long distances dragging on large screens

12. video

13. related work • techniques for transferring information • drag-and-drop avoids hidden clipboard (e.g. Xerox Star) • hyper­dragging (Rekimoto, 1999) • pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001) • overcome large distances • magic pointing (Zhai et al., 1999) requires an indirect input device • gesture input techniques (Rubine, 1991) • throwing (Geißler, 1998) for reliable target acquisition? • laser pointers to acquire targets on a Smartboard (Myers et al. 2002) • mouse-based interaction techniques • lodestones and lay lines (Jul, 2002) • flick (Dulberg et al., 1999) • sticky icons (Swaminathan and Sato 1997)

14. design

15. selecting candidates • initialize • all icons are candidates • filter • eliminate icons with non-matching file types • eliminate icons that are too close • eliminate icons outside target angle • if necessary, restrict to some hard limit

16. preserving layout • snap to grid • eliminate empty rows and columns • translate back • place center of bounding box in front of user • closer for experts

17. the rubber band • animationdid not work • “frozen”motion blur • narrow midriff • suggests elasticity • clue for distance • simplified version

18. getting it out of the way • to rearrange icons on the desktop (overloaded): • any mouse motion moving away from the “popped-up” icons de-activates drag-and-pop • introduce flick gesture into mouse motion

19. study

20. pre-study • 15 single, 6 dual, and4 triple monitor users • overall resolutions 800,000 pixels to 3,900,000 pixels • (= 66% more than the display wall used in the experiment). • 3 layouts for study: sparse (11), frame (28), cluttered (35)

21. user study • participants: 2 female, 5 male • dynaWall • 3 Smartboard • 15’ long (4.5m) • 3 x 1024x768 pixels • native code not stable enoughà Macromedia Flash version • task: drag icons into matching folder • highlighting disappeared when started • each desktop: 11-35 icons + 10 icons to be filed

22. Control Drag-and-pop results Control Drag-and-pop 3.7 timesspeedup • faster with drag-and-pop • error rate higher with drag-and-pop • most of the effect caused by the bezels

23. subjective satisfaction • > 6 (out of 7) • “I liked using drag-and-pop” • “I always understood what was happening when drag-and-pop was on”, • “I would use drag-and-pop for large displays.” • < 3 for • “It took a long time to get used to drag-and-pop” • “It was hard to control what the targets did when drag-and-pop was on.” • drag-and-pop interface causes less manual stress and fatigue than the control interface

24. lesson learned • the shortest connection between two points on a display wall is not a straight line • (we fixed this by opening target sector towards top of display)

25. the larger picture

26. general theme • WIMP metaphor can breakon large screens with pen/touch input • drag-and-pop generalizes direct manipulation • bring content to the user • let the user interact with it • send content back • à interaction space is not the same as display space anymore

27. drag-and-pick • problem • launch app or open file • drag-and-pick • user drags “background” • all icons in that direction move to the cursor • user drags % releases mouse over it • target is activated

28. what about dialog boxes?

29. inside applications… • drag-and-pop workseven if target is • occluded • clipped • closed (folder) • use the concept tofile emails?

30. goals revisited - conclusions • mixing touch screens/pen-input with large screens creates interesting new interface challenges • bringing target icons to the user (“drag-and-pop”) allows users to complete drag interactions faster • general theme: limiting interaction space to the display space at the user’s location can solve problems

31. thank you! try it out: Google drag-and-pop thanks to: the VIBE research group(mary czerwinski, george robertson…)diane kelly, dieter boecker,lance good, amanda williams,