Display Techniques in Information-Rich Virtual Environments

1. Display Techniques in Information-Rich Virtual Environments Nicholas F. Polys PhD Research Proposal August 18, 2004 Committee: Dr. Doug Bowman, VT Dr. Chris North, VT Dr. Scott McCrickard, VT Dr. Ken Livingston, Vassar College Dr. Don Brutzman, Naval Postgraduate School

2. Problem scope & Statement Background Research Questions & Goals Information Design Dimensions Approach, Method, Measures Experimental Program Significance Proposal Outline

3. Complex systems typically span multiple scales and involve heterogeneous data types (objects, spatial relations, attributes) Engineers, researchers, and analysts need to access, manage, and understand a wide variety of information and inter-relationships General Problem: Integrated Information Spaces

4. Spatial / perceptual data: geometry, colors, textures, lighting Abstract data / world & object attributes: nominal, ordinal, quantitative Temporal data / behaviors: states, dynamics Fundamental Data Types

5. Simulation and design applications require spatial/perceptual fidelity and information enhancement: Engineering / CAD Construction / architecture Medicine / biology Science / research Education / training … Information-Rich Domains

8. Multiple applications and fragmented views make it difficult to understand the relationships between information types Next generation information interfaces must unify display and interaction spaces for: Exploration Search Comparison and Pattern recognition User Problem: Integrated Information Spaces

9. We lack precise definitions, development tools, and systematic research as to how perception and cognition operate in information-rich interfaces and environments: Combining Virtual Environments and Information Visualizations is currently ad hoc and application-specific There are competing models of Vision and Working Memory that may apply An experimental methodology and theory is required to assess, design, and deliver ‘appropriate’ displays Problem Statement:

10. Edward Tufte, Envisioning Information (1983, 1990) Jaques Bertin, Semiology of Graphics (1983) Donald Norman, Cognitive Engineering (1986) Joseph Goguen, Semiotic Morphisms (2000) Colin Ware, Perception for Design (2003) Background:Information Psychophysics

11. Visual display of abstract information Visual Markers (Cleveland & McGill, 1984; Mackinlay, 1986; Card et al, 1999) Multiple Views (North, 2001; North et al, 2002; Convertino et al, 2003) Zoom-able Interfaces (Bederson et al, 1996, 2000; Woodruff et al, 1998a-c) Background:Information Visualization

12. Visual display of spatial and perceptual data Immersive and Desktop Platforms Conceptual Learning (Salzman et al, 1999) Navigating space (Darken et al, 1996, 2002) Naturalism & Performance (Bowman, 2002; Bowman et al, 2004) Image Plane (Pierce et al, 1997) Background:Virtual Environments

13. Enhancing perceptual scenes with additional abstract information Feiner et al. Windows on the World (1993) Bell et al. Dynamic Space Management, View Management (2000, 2001) Background:Augmented Reality

14. Co-references between text and images (Chandler & Sweller, 1990; Faraday & Sutcliffe, 1997, 1998) Task Knowledge Structure (Sutcliffe & Faraday, 1994; Sutcliffe 2003) Meaningful Learning: troubleshooting, redesigning, deriving principles (Mayer, 2002) Background:Multimedia & Comprehension

15. Componentized WM (Baddeley, 2003) phonological loop, visuospatial sketchpad, episodic buffer, central executive Individual differences in WM Capacity (Just & Carpenter: 1996) Short and Long term WM (Ericsson & Kintsch: 1995) Background:Architecture of WM

16. Central Executive Visuospatial sketchpad Episodic buffer Phonological Loop Fluid system Visual semantics Episodic LTM Language Crystallized system Working Memory (Baddeley, 2003)

17. Capacity 3-5 ‘items’ Functional units & chunking Objects & features (Vogel et al, 2001) Visual indices & dynamic feature binding (Saiki, 2003) Subsystems: form & color, space & movement (Logie 1995) Relation to Central Executive (Miyake et al. 2001) Background: Visuospatial WM

18. Computational model: Understanding Cognitive Information Engineering [UCIE] (Lohse, 1991) similar to GOMS (Card, Moran, & Newell, 1983), and ACT* (Anderson, 1983) Processing encoding visual patterns inferring and retrieving functional relations via graph schema associating / labeling referents Background: Interpreting Linegraphs

19. We need to understand: How spatial/perceptual information and abstract information can be combined and displayed What makes the combinations effective What makes them usable and How users think and act when using them Proposal: Information-Rich Virtual Environments (IRVEs)

20. Where and How should enhancing abstract information be displayed relative to its perceptual referent so that the respective information can be understood together and separately? Research Question for IRVE Information Design:

21. To understand how the respective design techniques of Virtual Environments and Information Visualization can be combined and balanced To enable stronger mental associations between spatial and abstract information while preserving the models of each type of information. Research Goals:

22. Define a theoretical framework for Information-Rich Virtual Environments (IRVEs) as the solution to the problem of integrated information spaces Results: Terminology Definitions Bowman et al, 2003 Research Goal 1:

23. Enumerate the design space for IRVE tasks and display techniques through Usability Engineering process and literature review Results: IRVE Task Space, Information Design Space, Interaction Design Space Polys & Bowman, 2004; Polys et al, 2004c Research Goal 2:

24. Prototype information-rich application interfaces to identify problems and generate hypotheses regarding optimal IRVE information design per task Results: IRVE Display Prototypes Claims Analysis 1 (heuristic evaluation, informal user studies, pluralistic walkthroughs) Polys, 2003; Polys et al, 2004a, 2004b Research Goal 3:

25. Describe IRVE display configurations in a concise XML DTD and Schema and use this display description to generate runtime components Results [pending]: IRVE Testbed Research Goal 4:

26. Provide a quantitative basis by which to characterize the density and distribution of information in an IRVE Results [pending]: Assessment method for IRVE information sets Research Goal 5:

27. Identify tradeoffs and guidelines for the IRVE display design space using Empirical usability evaluations and Metrics for individual cognitive differences Results [pending]: Empirical data relating design techniques, information sets, and user performance Research Goal 6:

28. For the combination of abstract and perceptual visualizations in IRVEs, association should be maximized and interference (such as occlusion and crowding by layouts) should be minimized… What are the best ways to manage layout space and association cues so that perceptual and abstract information can be understood together and separately? Research Question re-phrased:

29. Survey IRVE information design space Prototype IRVE display techniques and applications Understand and quantify how IRVE information sets vary Enumerate information design heuristics and guidelines for IRVEs Goals & Results Summary

30. Visibility Legibility Association Occlusion Aggregation IRVE Information Design Challenges

31. Abstract information design parameter Psychological process Usability impact Visual attributes:- color- fonts- size- background- transparency Perception - Legibility - Readability - Occlusion Layout attributes:- location- association - density Interpretation - Relating abstract and perceptual information - Conceptual categories & abstractions - Occlusion Aggregation: - level of informationencoding - type of visualization Making Sense - Comparison & Pattern Recognition - Effectiveness - Satisfaction IRVE Information Design Dimensions

32. The layout space of abstract information in IRVEs is described by the coordinate system it is resident in: Object World User Viewport Display Layout Space (Locations)

33. Object space is relative to an object’s location in the environment (e.g. Semantic Objects). Object Space

34. World space is relative to an area, region, or location in the environment. World Space

35. User space is relative to the user’s location but not their viewing angle. User Space

36. Viewport space-is the image plane where HUDs or overlays may be located. Viewport Space

37. Display layout space where abstract visualizations are located outside the rendered view in some additional screen area. Display Space

38. The association dimension of IRVE information design is delineated by the Gestalt principles: Association

39. Develop methods to describe and generate IRVE display components Develop quantitative methods to characterize IRVE datasets Use the above to control testbed environment and stimuli and run empirical usability evaluations Approach (Goals 4-6)

40. The composition of an IRVE display will be defined under an XML DTD and Schema. The DTD and Schema will provide syntactic and semantic production rules for IRVE display spaces. In order to instantiate an IRVE display, the testbed will read the information mappingconfiguration from the XML and generate the X3D code for that IRVE. IRVE Testbed Configuration Syntax

41. Exocentric vs. Egocentric: Exocentric metrics have an advantage in that they may be employed on a data set independent of the user or rendering; egocentric metrics may be more advantageous to find dynamic, perspective-specific layouts. This research proposal will investigate exocentric metrics for IRVE data characteristics such as the quantity, density, and distribution of abstract information in the VE. IRVE Stimuli & Metrics

42. Experiment 1 Common Region Proximity Connected ness Similarity Common Fate Object x x x x x World x x x x x User x x x x x Viewport x x x x x Display x x x x x Experiments 2 & 3 IRVE Layout & Association Dimensions

43. The usability experiments will have a factorial, within-subjects, counterbalanced design Cognitive battery test will be used control between-subject variance and may provide insight into how display techniques work with different visuospatial abilities. These experiments will allow us to explore tradeoffs in IRVE information design and identify guidelines and design patterns for integrated information spaces. Evaluation Method

44. Subjects from VT population Demographic questionnaire & Cognitive battery tests Training session for spatial navigations Experimental Protocol & dependent measures Procedure

45. Time to Completion Correctness Satisfaction / Ease of Use User strategy Cognitive Battery Measures

46. Perceptual Speed / Closure Flexibility Spatial Visualization Spatial Orientation Cognitive Battery

47. IRVE Search Tasks require subjects to: find a piece of abstract information based on some perceptual criteria find a piece of perceptual information based on some abstract criteria. IRVE Comparison Tasks require subjects to: compare perceptual attributes of two items with a given abstract criteria compare abstract attributes of two items with a given perceptual criteria IRVE Tasks

48. Object space layouts can be drawn at a fixed orientation or billboarded to always face the user. Is one technique better than the other for common tasks and navigation modes? Here we will investigate Search and Comparison tasks combined with flying or terrain-following navigation. Experiment 1:

49. Search & Comparison Tasks Terrain following and Flying navigation Fixed orientations provide additional wayfinding cues Billboarded orientations require less spatial navigation for legibility In these circumstances, is one display technique better that the other? Object space:Billboard vs. Fixed

50. Experimental Unit Human Subject (n = 8)  Tests: Flexibility of Closure, Perceptual Speed, Spatial Orientation, Factor 1 Object space layout Level 1 Billboarded Level 2 Fixed Factor 2 Navigation type Level 1 Walk Level 2 Fly Factor 3 Task: Search Level 1 Spatial -> Abstract Level 2 Abstract -> Spatial Factor 4 Task: Comparison Level 1 2 Spatial items Level 2 2 Abstract items Response Variables Performance: Time to Completion, Correctness Satisfaction: Ease of Use 16 conditions