Space perception and the display of data in space

1. Space perception and the display of data in space Dr. Yan Liu Department of Biomedical, Industrial and Human Factors Engineering Wright State University

2. Introduction • What are Depth Cues • Sources of information about 3D space • Categories of Depth Cues • Monocular static (static picture): • Linear perspective, texture gradient, size gradient, occlusion, depth-of-focus, cast shadows, shape-from-shading, depth-from-eye accommodation • Monocular dynamic (moving picture) • Structure-from-motion (kinetic depth, motion parallax) • Binocular • Eye convergence, stereoscopic depth

3. Perspective Cues • Overview • Parallel lines converge to a single point • Objects at a distance appear smaller on the picture plane than do nearby objects • Objects of known size may have a very powerful role in determining the perceived size of adjacent unknown object • Uniformly textured surfaces result in texture gradients in which the texture elements become smaller with distance

4. Perspective cues include the convergence of lines and the fact that more distant objects become smaller on the picture plane • A texture gradient is produced when a uniformly textured surface is projected onto the picture plane

5. Perspective Cues (Cont.) • Size Constancy • Humans generally perceive the actual size of the object rather than the size at which it appears on a picture plane (or on the retina) • A useful measure of the relative effectiveness of depth cues • Dual perception mode • Perceive the size of the depicted object as though it were in a 3D space • Perceive the size of the object at the picture plane • The amount and effectiveness of the depth cues used will make it easy to see in one mode or the other • The picture-plane sizes of objects in a very sketchy schematic picture are easy to perceive • The real 3D sizes of objects will be more readily perceived with depth cues

6. Perspective Cues (Cont.) • Amount of Information Displayed • There is little evidence that a perspective picture lets us see more than a non-perspective image (b) Data Mountain display for document management (without perspective cues) (a) Data Mountain display for document management (with perspective cues) • (a) and (b) are of the same effectiveness in terms of showing information • Both figures make extensive use of other depth cues (occlusion and height on the picture plane)

7. Pictures Seen from the Wrong Viewpoint • Issue • Most pictures are not viewed from their correct centers of perspective • Robustness of Linear Perspective • Although people experience some distortion initially when looking at some moving pictures from the wrong viewpoint, they become unaware of the distortion after a few minutes • The mesh is projected on a screen with a geometry based on viewpoint (a) but is actually viewed from position (b) • Laws of geometry predict that large distortions should been seen Illustration of a perspective picture seen from a wrong viewpoint

8. Pictures Seen from the Wrong Viewpoint (Cont.) • Solution • Track a user’s head position with respect to a computer screen and thereby estimate the position of the eye(s), so that the perspective of the scene is kept correct at all times by adjusting the viewpoint parameters in the computer graphics software

9. Occlusion • Overview • If one object overlaps or occludes another, it appears closer to the observer • Provides only binary information • An object is either behind or in front of another; no information is given about the distance between them • Partial Occlusion • When one object is transparent or translucent • There is a color difference between the parts of an object that lie behind the transparent plane and the parts that are in front of it • The plate and the person on the top of it appear closer to the viewer

10. Occlusion (Cont.) • Applications in Design • Occlusion provides rank-order information of the tabbed card, in addition to rapid access to individual cards • Occlusion is used to show overlapping window interfaces

11. Depth-of-Focus • Overview • Eyes change focus to bring the images of fixated objects into sharp focus on the fovea, and the images of both nearby and more distant objects become blurred • Applications • Simulating depth-of-focus is an excellent way to highlight information by blurring everything except that which is critical • This technique is computationally expensive and thus currently limited in utility • The eye adjusts to bring objects of interest into sharp focus. As a result, objects at different distances become blurred

12. Cast Shadows • Applications • A very potent cue to the height of an object above a plane • The shadow locates the object with respect to some surface in the environment • Function the best as a height-above-surface cue when there is a relatively small distance between the object and the surface • Especially powerful when objects are in motion

13. Shadows provide a strong cue for the relative height of objects (the surface is not present in the illustration but assumed by the brain) • The image of the ball actually travels in a straight line, but the ball seems to bounce because of the way the shadow moves. This suggests that shadow motion is a stronger depth due than change in size with perspective • Cast shadows can be useful in making data appear to stand out above an opaque plane

14. Structure-from-Motion • Motion Parallax • As we move objects that are closer to us move farther across our field of view than do objects that are in the distance • Kinetic Depth Effect • The 3-D structural form of an object viewed in 2-D projection can be perceived only when the object is rotating

15. A moving observer looking at trees and mountains will notice that they appear to move different distances, because each object is located at a difference distance, and the separation of distances gives the appearance of each object having different motion across the observer’s visual field Illustration of motion parallax • A wire is bent into a complex 3-D shape and projected onto a screen. The projection is 2-D line. But if the wire is rotated, the 3-D shape of the wire immediately becomes apparent Illustration of kinetic depth effect

16. Structure-from-Motion (Cont.) • Applications • Structure-from-motion information is at least as important as stereoscopic depth in providing us with information about the spatial layout of objects in space • Helps to determine both the 3-D shapes of objects and the large-scale layout of objects in space • It is the reason for the effectiveness of fly-through animated movies that take an observer through a space

17. Stereoscopic Depth • Basic Theory • Stereoscopic depth perception is based on binocular disparity • Brain receives slightly different images from the two eyes which can be used to compute relative distances of pairs of objects • On the left, different images for the two eyes are shown • On the right, both eyes are fixated on a vertical line (a for the right eye and c for the left eye). A second line d in the left eye’s image is fused with b in the right eye’s image. The brain resolves the discrepancy in line spacing by perceiving the lines as being at different depth, as shown Illustration of a simple stereo display

18. Stereoscopic Depth (Cont.) • Terminology • Angular disparity • The difference between the angular separation of a pair of points imaged by the two eyes • Screen disparity • The distance between parts of an image on the screen • Diplopia (Double vision) • The appearance of the doubling of part of a stereo image when the visual system fails to fuse the images • It occurs if the disparity between the two images becomes too great • Panum’s fusional area (the region of binocular single vision) • The 3D area with which objects can be fused and seen without double images angular disparity = α – β screen disparity = (c – d) – (a – b)

19. Stereoscopic Depth (Cont.) • Properties • Stereoscopic depth in monitor-based stereo displays • Given a screen with 30 pixels/cm and viewed at 57cm, we can only display three whole-pixel-depth steps before diplopia occurs, either in front of or behind the screen • In the worst case, it will only be possible to view a virtual image that extends in depth a fraction of a centimeter from the screen • It is likely that antialiased images will allow better-than-pixel resolution • The size of Panum’s fusional area is highly dependent on a number of visual display parameters (e.g. exposure duration of the images and the size of the targets) • Both moving targets and blurred images can be fused at greater disparities • Stereopsis is a superacuity • We can resolve disparities of only 10 seconds of arc at better than chance • We should be able to see a depth difference between an object at 1km and an object at infinity, under optimal viewing conditions

20. Problems of Stereoscopic Displays • Frame Cancellation • If the stereoscopic depth dues are such that a virtual image should appear in front of the screen but the edge of the screen appears to occlude the virtual object, occlusion overrides the stereo depth information and the depth effects collapse • Typically accompanied by a double image of the object that should appear in front • The edge of the screen clips the object, which acts an occlusion depth cue and the object appears to be behind the window, canceling the stereo depth effect • The usable working volume of a stereoscopic display is restricted Illustration of Frame cancellation

21. Problems of Stereoscopic Displays (Cont.) • Vergence-Focus Problem • Relates to coupling of the focusing mechanism in the eyes with the vergence mechanism that makes the eyes converge when we see objects at different distances. • Screen-based stereo displays can only provide the correct information for vergence but not focus (leading to eyestrain) • Distant Objects • Stereoscopic depth cue is most useful for 30m or less from the viewer • For practical purposes, most useful stereoscopic depth is obtained within distances of less than 10m from the viewer and may be optimal for objects held roughly at arm’s length

22. Making Effective Stereoscopic Displays • High-Resolution Display • The ideal stereoscopic display should have very high resolution, much higher than the typical desktop monitor • High-resolution displays enable the presentation of fine texture gradients and hence disparity gradients that are the basis for stereoscopic surface perception • Cyclopean Scale • Deal with the diplopia problem • Scale the virtual environment about the midpoint between the observer’s two eyes until the nearest part of the scene lies just behind the screen • Advantages • Brings distant objects closer to positions where stereo depth becomes available • Reduces vergence-focus discrepancy • Removes the possibility of frame cancellation

23. A virtual environment is resized about a center point midway between the two eyes until the near point is just behind the screen Illustration of cyclopean scale

24. Making Effective Stereoscopic Displays (Cont.) • Virtual Eye Separation Ev: the virtual eye separation Ea: the actual eye separation of the observer zv: the depth in the virtual image zs: the viewed stereo depth ze: the distance from the eye to the screen (Eq. 1) Ev  zs, Ev zs Geometry of virtual eye separation

25. Making Effective Stereoscopic Displays (Cont.) • Telestereoscope • Uses a system of mirrors to increase the effective eye separation and thus increase the stereoscopic depth disparity Illustration of a telesteroscope device

26. Making Effective Stereoscopic Displays (Cont.) • Adjust Virtual Eye Separation Eye Separation(cm) = 2.5+5.0*(Near Point / Far Point)2 (Eq. 2) Near Point: the nearest point in the scene Far Point: the farthest point in the scene • Increases the eye separation to 7.5cm for shallow scenes • Reduces it to 2.5cm for very deep scenes

27. Artificial Spatial Cues • What are Artificial Spatial Cues • Provide information about space that are not based directly on the way information is provided in the normal environment • Examples • Adding vertical lines dropping to the ground in a 2D place can at least as effective as stereopsis in providing 3D position information • One of the most effective ways to estimate the sizes of objects is with reference to the ground plane • Dropping lines enhance the 3D scatterplot

28. Artificial Spatial Cues (Cont.) • Proximity Luminance Covariance • Vary the color of an object depending on its distance from the viewpoint • More distant objects are faded toward the background color (becoming darker if the background is dark and lighter if the background is light) • It can function as an effective depth cue but is weaker than stereo for static displays • With moving displays, it can become a relatively stronger cue in making ambiguous 3D scene unambiguous • Object color is altered with distance in the direction of the background color Proximity luminance covariance as a depth cue

29. Depth Cues in Combination • It is not always the best solution to include all depth cues • There can be considerable costs associated with creating a stereoscopic display or with using real-time animation to take advantage of structure-from-motion cues • Some cues (e.g. depth-of-focus information) are difficult or impossible to compute in the general case • Interactions of Depth Cues • The weighted-average model assumes that depth perception is a weighted linear sum of the depth cues available in a display • Depth cues may combine in a geometric sum • Depth cues are combined additively but are weighted according to their apparent reliability in the context of other cues and relevant information • Some depth cues, occlusion in particular, work in a logical binary fashion rather than contributing to an arithmetic or geometric sum

30. An arrow indicates a particular cue depends on another cue to appear correctly • Does not show absolute rules that cannot be broken, but imply that breaking the rules will have undesirable consequences • The rules are transitive • Occlusion is the most basic depth cue A dependency graph for depth cues

31. Task-Based Space Perception • Perception And Actions Are Intertwined • If we are to understand space perception, we must understand the purpose of perceiving • Some common elementary tasks • Tracing data paths in 3D graphs • Judging the morphology of surfaces and surface target detection • Finding patterns of points in 3D space • Judging the relative positions of objects in space • Judging the relative movement of self within the environment • Reaching for objects • Judging the “up” direction • Feeling a sense of presence

32. Tracing Data Paths in 3D graphs • Nodes represent various kinds of entities • e.g. modules, classes, variables and methods • 3D spars that connect the entities represent various kinds of relationships characteristic of object-oriented software • e.g. inheritance, function calls, and variable usage The structure of object-oriented software code, represented as a 3D network of nodes and arcs

33. The tree branches are arranged around a series of circles • It can display much more number of nodes without visual clutter than a tree in 2D layout • It requires more complex user interactions to access some of the information than are necessary for a tree in 2D layout A cone tree that represents tree graph information in 3D space (Robertson et al., 1993)

34. Tracing Data Paths in 3D graphs (Cont.) • Impact of Depth Cues • Stereo and kinetic depth can improve path tracing ability in 3D graphs, kinetic depth is more potent, and the combination of them is even better • Occlusion is another depth cue that should make it easier to differentiate arcs if they colored differently • Occlusion makes it easier to see which arcs lie above and beneath • It seems unlikely other depth cues can contribute much to a path-tracing task

35. Judge the Morphology of Surfaces and Surface Target Detection • Impact of Depth Cues • Shape-from-shading and texture cues are extremely important in revealing surface shape • The way different depth cues combine in judgments of surface shape is highly complex • The relative values of stereo and structure-from-motion depend on the viewing distance, the texture of the surface, the kind of surface shape, and the viewing time • When arbitrary surface shapes are being viewed, stereoscopic depth, kinetic depth, shape-from-shading, and surface textures can all add to our understanding of surface shape • The most important cues for any particular surface will vary, but including them all will ensure that good shape information is always presented

36. Patterns of Points in 3D Space • Impact of Depth Cues • Stereoscopic depth and structure-from-motion may be the only important depth cue for 3D scatterplot • Stereo depth will be optimal for fine depth discriminations between points that lie near one another in depth • Structure-from-motion will be more important for points that lie farther apart in depth, revealing the structure of the plot by rotating it around an axis

37. Judging Relative Positions of Objects in Space • Impact of Depth Cues • Depends on the overall scale and context • When very fine depth judgments are made in the near vicinity, stereopsis is the strongest single cue • Stereoscopic depth perception is a superacuity and is optimally useful for objects held at about arm’s length • In large environments, stereoscopic depth can play no role at all at distances beyond 30m • When we are judging the overall layout of objects in a larger environment, motion parallax, linear perspective, cast shadows, and texture gradients all contribute, depending on the exact spatial arrangement

38. The Aesthetic Impression of 3D Space (Presence) • Issues • Achieving a sense of presence is one of the most nebulous and ill-defined tasks related to 3D space perception • What is it that makes a virtual object or a whole environment seem vividly 3D? • What is it that makes us feel that we are actually present in an environment • Much of the presence has to do with a sense of engagement and not necessarily with visual information • Impacts of Depth Cues • High-quality structure-from-motion information contributes more to a sense of presence than does stereoscopic display • Stereoscopic viewing is important when subjects are asked to rate the extent to which they feel they can reach for and grasp virtual objects, but it does not contribute to the sense of the overall realism of the virtual condition