Direct Manipulation

1. Direct Manipulation Shneiderman and Plaisant, Chapter 6 (from slides at http://wps.aw.com/aw_shneider_dtui_5/) Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct Manipulation interfaces. Human-Computer Interaction, 1, 311-338.

2. Overview • Introduction • Big picture first, details and a theoretical account later • Examples of Direct-Manipulation systems • Explanations of Direct Manipulation • Hutchins et al. paper • Virtual Environments, or Immersive Interfaces

3. Recall, Interaction Styles from Shneiderman • Styles for “giving commands” to the system • 5 main types (p. 67): • Direct Manipulation • More detail later • Menu selection • Form fillin • Command language • Natural language • May blend, especially when users are diverse

4. Paradigm Example: File System Direct Manipulation vs. Command Line • Direct Manipulation: • User operates on model world to change task world • Will sharpen this up with more detailed accounts • Dix et al. and Hutchins, Hollan, & Norman (reading)

5. “Direct Manipulation” Interface to CarNo Computer, but “Direct” – “an ultimate example” Task: Maneuver the car from Point A to Point B (within certain constraints) Interface: Steering wheel, pedals, etc. Geographic area through which user (driver) maneuvers car is displayed directly in front of user (and in high resolution, and in large field of view!) Manipulating steering wheel top to left (and right) causes car front to move left (and right) visual and kinetic feedback is immediate Pushing brake pedal results in car slowing arbitrary mapping of user action to car maneuver (automatized) visual and kinetic feedback is immediate Definition, again: User operates on a visually displayed “world” to perform tasks in a domain Here, the world on which the user operates, is the real world For computing tasks, there is the task world and an interface model world The user operates on the model world to change the task world

7. System “work on task” “commands” “performs” “gives” Task User “feedback” Recall (or not), from 1st classInteraction Frameworks (Dix) • Interaction: • Communication between user and system • Why have a framework? • Allows “precision” in accounting for differences • E.g., gulfs of execution and evaluation • Presents global view • All elements receive attention

8. System “work on task” “commands” “performs” “gives” Task User “feedback” and its representation Background: Models in UI DesignFrom Dix et al. • Models of UI design consider: • System • Interface • User • Will “rewrite” Dix’s account to incorporate standard account of models

9. System System “work on task” “work on task” “commands” “gives commands” “performs” “gives” feedback feedback Task Task User User “feedback” and its representation and its representation Background: Models in UI DesignFrom Dix et al. • Models of UI design consider: • System • Interface • User • Will “rewrite” Dix’s account to incorporate standard account of models • First, it is the System that directly provides feedback about the task

10. System “work on task” “gives commands” Task System User “work on task” “gives commands” feedback feedback Task User and its representation Models in UI Design • Next, just change notation a bit • And …

11. “work on task” “gives commands” Task System User Models in UI Design • For our purposes, Dix’s System (which also represents task) comprised of two components • 1. System model • How system actually works • Implementation model • 2. Interface model • Model system presents to user • Manifest model User Model Interface Model System Model • Finally, 3. User model • How the user thinks system works • Conceptual model

12. User/Conceptual/Mental … Model • How the user thinks system works • Conceptual model • May or may not correspond to the system! • Ideally, should match closely, … if (and only if) system reflects task • Mental model of the system (and task)

13. Example of DM: Word Processing • People used to use script files for formatting, until 1980’s • E.g., nroff • “word processors” allowed direct manipulation of document (text and images) output • Albeit with a fair amount of pointing, selecting, and menu use

14. Example of DM: VisiCalc and Later • People used to use paper and pencil for accounting, until 80’s • Shneiderman • The VisiCalc spreadsheet and its descendants • Imagine erasing a thousand numbers … • “VisiCalc users delighted in watching the program propagate changes across the screen.” • In some cases, spatial representations provide a better model of reality • Successful spatial data-management systems depend on choosing appropriate: • Icons • Graphical representations • Natural and comprehensible data layouts

15. Example of DM: WIMP File System • Copy, move, rename, delete • But, is it a file cabinet?

16. Example of DM: Drawings • Autocad, etc. • Spatial

17. Example of DM: Statistical Analysis • Specify statistical operations by manipulating icons • Similar to “visual programming”

18. Example of DM: Statistical Analysis

19. Example of DM: Games • D & D with voice chat • Doom, etc. • .. And better

20. System “work on task” “ gives commands” feedback feedback Task User “feedback” and its representation (possibly manipulable) Basic Interaction ParadigmIn context of direct manipulation and command language • Historically, “commands” differ by paradigm • Switches, cards • Command line • W-Icons-M-P • Relatively, direct or indirect manipulation of system • “giving commands to system” • Cmd lang: Conversational • User has a conversation with interface, telling it what to do • 3rd person • User tells system what to do in task • Then system does it • Direct Manipulation: Direct • No conversation • User interacts directly with (representation) of task • 1st person • Operates directly on model worlds

21. General Comments about DMSchneiderman’s Historical Account … and Definition • Shneiderman described in 1974 – will see sharper description • Positive feelings associated with “good” user interfaces often attributed to “direct manipulation” interfaces: • Mastery of the interface • Competence in performing tasks • Ease in learning the system originally and in assimilating advanced features • Confidence in the capacity to retain mastery over time • Enjoyment in using the system • “Eagerness to show the system off to novices” <- ! • Desire to explore more powerful aspects of the system • Definition: • User operates on a visually displayed “world” to perform tasks in a domain • All in all easier, to see examples, then analyze

22. Direct (and Indirect) Manipulation Interfaces • “Direct manipulation” user interfaces contrasted with, • well, “indirect manipulation” interface • Say, command line systems, or card systems, or switch systems, or … • Recall, discussion of history of computing … relative costs, technology available • But, that was a long time ago … • Paradigm example: File system manipulation • Bottom line, file system in a computer is just a collection of storage elements orientations • Even idea of “files” is an abstraction • Organization of files in a hierarchy is a further abstraction • Command line • cp /usr/fowler/presentations/colloquium_98/hypertextnodes.gif /usr/fowler/lectures/6362/dir_manip/ • Tell the system to take one file and make a copy of it and place that copy in the designated place • mv hypertextgraph.gif temp_hypertextgraph.gif • Tell the system to “move” one file to another file with a different name (rename)

23. System “work on task” “ gives commands” feedback feedback Task User “feedback” and its representation (possibly manipulable) Conversations and Model Worlds…In context of direct manipulation and command language • Interface as conversation: • Interface is a language medium - user and computer have a conversation about an assumed, but not explicitly represented, world • Interface is intermediary between user and assumed world about which things are said (and in which commands are issued) • Interface as model world: • Interface is itself a world where tuser can act and which changes state in response to user actions • World (domain) of interest is explicitly represented - there is no intermediary between user and world • Well done, this can create a sense of acting upon the object of the task domain themselves • feeling of direct engagement

24. Direct Manipulation “Characterized”Again, … history and fyi • An orientation • Really, direct manipulation is perhaps best characterized as an “orientation” to interface style • “Definition/orientation”, yet again: • User operates on a visually displayed “world” to perform tasks in a domain • Among the earliest characterizations (Schneiderman, 1982): 1. Continuous representation of the object of interest 2. Physical actions or labeled button presses instead of complex syntax 3. Rapid incremental reversible operations whose impact on the object of interest is immediately visible • Benefits (Schneiderman, 1982): 1. Novices can learn basic functionality quickly, usually through a demonstration by a more experienced user 2. Experts can work extremely rapidly to carry out a wide range of tasks, even defining new functions and features 3. Knowledgeable intermittent users can retain operational concepts 4. Error message are rarely needed (why?) 5. Users can see immediately if their actions are furthering their goals, and if not, they can simply change the direction of their activity

25. Direct Manipulation “Characterized”Hutchins et al. orientation • Typical characteristics of direct manipulation interfaces: 1. System (task) objects have visual representations 2. User interacts with a model world 3. No “conversation” about what to do the user directly manipulates the model world 4. Representations that match the user model facilitate feelings of directness 5. Output from one system action is input for another • Hutchins et al.’s account • Delineates range of interface phenomena that can contribute to feeling of directness • Assumption is that the feeling of directness results from the commitment of fewer cognitive resources • Or (put the other way around), the need to commit additional of the user’s cognitive resources leads to feelings of indirectness • And, so, account is in essence a cognitive account of direct manipulation interfaces

26. Hutchins et al. PaperOverview • Provides a detailed analysis of the elements of direct manipulation • Again, conversation metaphor and model world metaphor for interfaces • Two aspects of directness in direct manipulation: • Distance and Engagement • directness = f(distance, engagement) • Cognitive effort results from gulfs of execution and evaluation • Gulf = f (Mismatch of system elements and user’s task element) • gulf of execution • match of commands and mechanisms of system with thoughts and goals of user • gulf of evaluation • extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated • Model worlds and inter-referential input/output • Two forms of distance: Semantic and articulatory • A space of interfaces: • Engagement x Distance from user goals

27. Two Aspects of Directness in Direct ManipulationDistance and Engagement directness = f(distance, engagement) • Directness • The feeling that results from interaction with an interface (direct vs. indirect) • Distance • “Distance” between user’s thoughts and system’s requirements for using it to perform task • (e.g., g32, s/edtor/editor - large distance) • Short distance means translation between user’s thoughts and system is simple and straightforward • (e.g., insert ‘i’, or type over “edtor” with “editor”) • Describes factors which underlie the generation of the feeling of directness

28. Two Aspects of Directness in Direct ManipulationDistance and Engagement directness = f(distance, engagement) • Engagement • Feeling that the user is directly manipulating the objects of interest • e.g., dragging a subdirectory to another subdirectory vs. mv /usr/bob/*.* /usr/jo/ • Again, conversation metaphor and model world metaphor for interfaces • talking about and describing an operation vs. • performing operation in a (visual) model world (representing the task domain)

29. directness = f(distance, engagement)Gulfs of Execution and Evaluation • Distance: Relationship between user’s task and way task can be accomplished via interface • To enhance directness, • need to minimize cognitive effort required to map interface actions to task actions • i.e., “bridge the gulf of user’s goals and way they are specified in the system” • “gulfs” exist to the extent of the mismatch of user’s task intentions and how system used to carry out those task intentions • Gulfs are about 1. making things happen and 2. knowing if happened • 1. gulf of execution • match of commands and mechanisms of system with thoughts and goals of user • 2. gulf of evaluation • extent to which system output displays present good conceptual model of system that is readily perceived, interpreted, and evaluated

30. System “work on task” “commands” “performs” “gives” Task User “feedback” Directness and Model World Metaphor1 - Cognitive Effort • directness = (inversely related to, or 1/) amt of cognitive effort to manipulate system • Less effort, more directness • More effort, less directness • Leading to feelings of “direct engagement” with the task • Cognitive effortresults from gulf of execution and evaluation • Explains why systems are relatively easy or hard to use, and suggests how to minimize effort, or, make system easier to learn and use

31. Directness and Model World Metaphor2 – Controlling world vs. interface • Yet, “direct engagement” describes a qualitative feeling that user is directly engaged with control of objects (of task), rather than the computer or program which controls the task • e.g., really drawing the drawing vs. moving a cursor which, then, effects a line or fill • Characterized as feeling of “first-personess” vs. “third-personess” • User is operating on objects of task directly vs. telling someone (the system) to operate on objects • Latter is surely case in “conversation metaphor” model of user interfaces • User operates with linguistic structures (i.e., telling interface what to do) vs. • User him/herself operating directly on the task objects (with no intermediary) • *** Central metaphor of system is the model world itself - the key to direct manipulation interfaces

32. System “work on task” “commands” “performs” “gives” Task User “feedback” Model Worlds andInter-referential Input/Output • Operating in a model world, of course, requires that representations of task (model world) objects be manipulable in a direct, visual way • Additionally, if user is to carry out a series of operations (effect a series of instructions/commands which change state of system) • Then necessary that results of one operation be available for further modification • That is, the output of one operation serve as input for another operation • Term “inter-referential input/output” refers to this characteristic of interface objects • e.g., files and folders are moved and more moved again, etc. • words and paragraphs in editor undergo further modification (of course)

33. System “work on task” “commands” “performs” “gives” Task User “feedback” Two Forms of Distance: Semantic & Articulatorydirectness = f(distance, engagement) • Term “language” is quite general • Refers in general to mapping (often arbitrary) of sign to meaning • Interface language - elements which cause change in system state • Two “dialects” • Language for input (causing change to occur) • Language for output (evaluating ensuing state of system), • e.g., in dm are files in new loc vs. result of mv ... (in fact none unless cd and ls) • Semantic Distance • Relationship of an expression in interface language to what user wants to say • Articulatory Distance • Relationship between meaning of expression and its physical form

34. System “work on task” “commands” “performs” “gives” Task User “feedback” Two Forms of Distance: Semantic & Articulatorydirectness = f(distance, engagement) • Semantic Distance • Relationship of an expression in interface language to what user wants to say • Articulatory Distance • Relationship between meaning of expression and its physical form • E.g., dragging a file icon to another location in a file structure is small – has both small semantic (goals) and articulatory (expression (=visual form) distances • Both relatively large for command langs • Menu interface might have intermediate or less • Goals can be easy or hard to semantically map (good vs. not menu structure) • Likely, form of expression (~command) may have large dist.

35. Both Articulatory and Semantic Distances in Gulfs of Execution and Evaluation • Relationships among: • semantic distance • articulatory distance and • gulfs of execution and evaluation • Recall, menu example

36. Direct Engagement • Summing up, “direct engagement” occurs when user experiences direct interactions with the objects in a domain • Again, “first-personness” • no intermediary or conversation about what to do • Model world represented in interface is the task • Interface becomes “invisible” or transparent or disappears • Not clear-cut how to create • And these feeling in fact can come from automization (learning) for many (or any?) interface • Laurel (1986) suggests direct engagement requires: • Execution and evaluation should exhibit both semantic and articulatory directness • Input and output languages should be inter-referential • allows to feel as if manipulating objects of concern • Rapid system response (ideally, 0 or time appropriate for task) • “Unobtrusive” interface • interface is not to be noticed, if it is, leads to third-person relation

37. A Space of Interfaces • Direct manipulation interfaces minimize semantic and articulatory distances and maximize engagement

38. Problems with DM Interfaces • Graphical interfaces, generally, have universal usability challenges • E.g., for visually impaired • Uses much screen space for graphical representations of objects • Detail through text may be displaced • May still have to learn meaning of visual representations • Not all visual representations have “straightforward” meaning • Visual representation may be misleading • Even with analogy grasped, may still have to learn limitations • Requires switching between keyboard and mouse • Small screens with touch interfaces limit interaction

39. Challenges for DM Interfaces • Choosing “right”, or useful, interface metaphor (that has a natural visual representation) a challenge … • What is a good metaphor for air traffic control? • What is the “best” metaphor • … and for whom • E.g., file cabinet for secretaries, directed acyclic graph for computer scientists • “Rapid (<100 ms), incremental, reversable” actions using graphic representations can have high processing demands, even by current standards • E.g., high processing load db systems • Doing this web-based currently a challenge

40. Summary – Principles of DM(from Shneiderman) • Principles: • Continuous representations of the objects and actions of interest with meaningful visual metaphors • Physical actions or presses of labeled buttons, instead of complex syntax • Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

41. End • .

42. Fyi - Visual Thinking and Icons Exploiting visual nature of computers Can challenge the first generation of hackers Individual cognitive styles vary: Verbal vs. visual Linear vs. holistic “Left vs. right - brain” Icon defined An image, picture, or symbol representing a concept (or a small religious object) Guidelines for icon design • Represent the object or action in a familiar manner • Limit the number of different icons • Make icons stand out from the background • Consider three-dimensional icons • Ensure a selected icon is visible from unselected icons • Design the movement animation • Add detailed information • Explore combinations of icons to create new objects or actions Five levels of icon design: • Lexical qualities • Syntactics • Semantics • Pragmatics • Dynamics

43. Fyi - Direct-Manipulation Programming Visual representations of information make direct-manipulation programming possible in other domains Example Demonstrational programming is when users create macros by simply doing their tasks The five challenges of programming in the user interface: • Sufficient computational generality • Access to the appropriate data structures and operators • Ease in programming and editing programs • Simplicity in invocation and assignment of arguments • Low risk Cognitive dimensions framework may help analyzing design issues of visual programming environments.