Text and Graphics

1. Text and Graphics Reading: Reiter and Dale, chap 7 Multimodal Output Generation workshop (Jan 07)

2. Different Modalities • Many ways to communicate data • Visualisation • Written text • Spoken text (speech) • Combinations of above

3. Text and Graphics • Users want text+graphics combos • Cater to diff individual prefs • Visually oriented vs linguistically oriented • Media often complementary • Graphics for details, text for big picture • Graphics for data, text for interpretation • Redundancy also useful • Very common request

4. How do we combine text and graphics? • Show both separately • Graphically highlight things in text • Textual annotations supplement graphics • Complex integration • Diff info in diff media?

5. Strategy 1: No Integration • Easy approach – generate each separately, show both • Eg, show scuba visualisation • Also show scuba report • No integration between two

6. Example: show both • Your first ascent was a bit rapid. You ascended from 33m to the surface in 5 minutes, you should have taken 6 minutes to make this ascent. Your second ascent was fine.

7. Strategy 2: Graphical Highlighting • Strategy 2: Highlight on graph what we are referring to • Graphical highlighting often more effective than linguistic referring expressions

8. Example: highlighting Your first ascent was a bit rapid. You ascended from 33m to the surface in 5 minutes, you should have taken 6 minutes to make this ascent. Your second ascent was fine.

9. Fancier highlighting Risky dive with some minor problems. Because your bottom time of 12.0min exceeds no-stop limit by 4.0min this dive is risky. But you performed the ascent well. Your buoyancy control in the bottom zone was poor as indicated by ‘saw tooth’ patterns marked ‘A’ on the depth-time profile.

10. Strategy 3: Textual annotations • Use small “pop-up” text boxes to give extra information about the graph

11. Example: pop-ups too rapid OK

12. Stratregy 4: Deeper integration • Communicate different info in text and graphics • Perhaps use text for abstract info • Explanation, causality, what-if, … • Graphics often bad at communicating this kind of information

13. Example: text just for abstract info You should have taken 6 minutes to make this ascent!

14. Which is best? • Poorly understood • Graphical highlighting often better than no-integration (depends on domain) • Deep integration is a nice idea in principle, but difficult to get right in practice • Needs more research!

15. Embodied Conversational Agents • Animated “agents” that combine gestures, speech output, text output • Similar to computer game technology • Example: Microsoft agent • Free, nice programming model • Not actively being developed • Demo

16. When are ECAs useful? • Ideas from class?

17. ECAs • Probably useful in tutoring/teaching contexts (especially for kids) • Possibly useful in marketing • Probably depends on context (first time visitor? What is being sold? Etc) • Perhaps not useful in data interpretation and communication (??)

18. Summary: multimodal DIC • The best way to communicate data • Many open research issues! • HCI: What is useful? • NLG, visualisation: How to produce? • Interaction? • Happy to discuss with interested students