Some Reflections on Augmented Cognition Eric Horvitz ISAT & Microsoft Research November 2000

1. Some Reflections on Augmented CognitionEric HorvitzISAT & Microsoft ResearchNovember 2000

2. Apply computation to support / augment human cognition Toward Augmented Cognition Challenge: Volume of information, overall complexity of defense tasks continue to grow at a rapid pace, in stark contrast to human cognitive abilities, which remain static: Attention Memory Learning Comprehension Sensory bandwidth Visualization abilities Judgment and decision making In contrast, computational capabilities have continued to grow rapidly

3. Augmented Cognition Developing new powers of understanding, remembering, and decision making via new technologies custom-tailored for human—computer collaboration, symbiosis • Methods, designs that harness computation and explicit knowledge about human limitations to open bottlenecks, address biases, deficits in cognition. • Techniques and interfaces that allow for fluid mixed-initiative interaction to solve problems. • Continual background sensing, learning, inference to understand trends, patterns, situations relevant to a user’s context and goals.

4. Modern Psychology: Cognition as information processing • Human cognition: Vast abilities coupled with highly characterized / characterizable limitations & bottlenecks • Mind as an architecture composed of different subsystems that form, remember, transmit representations of the world. • Work on distinct dimensions of cognition: Attention, memory, learning, concept attainment, visualization, judgment

5. Timing: Is Now the Time to Undertake Such an Effort? Understanding of human as resource-limited reasoner, well-characterized bottlenecks, capabilities, biases • Limitations in serial processing • Attention • Concept attainment • Learning and memory • Visualization • Judgment and decisions • Collaboration, group decisions • Psychophysics • HCI expertise, results Results in Cognitive Psychology

6. Timing: Is Now the Time to Undertake Such an Effort? Leaps in computation, memory, algorithmic prowess • Raw computational resources (CPU, memory) • Connectivity • Sensing • Sensor fusion • Inference and classification • Learning • User modeling • Speech recognition • Vision • Interaction, input • UI, display design Computational power

7. Abilities & efficiencies Concept Attainment Memory Divided Attention Computation Augmented Cognition Efforts • Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations • Identify, apply results from cognitive psychology to build computation solutions • Identify new cognitive psychology research • Identify new joint CS—cognitive psychology efforts Target Cognitive Tasks Existing Psychological Results on Cognitive Limitations

8. Abilities & efficiencies ? Concept Attainment Memory Divided Attention Augmented Cognition Efforts • Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations • Identify, apply results from cognitive psychology to build computation solutions • Identify new cognitive psychology research • Identify new joint CS—cognitive psychology efforts Target Cognitive Tasks * New Cog. Psych. Research * HCI, Aug. Cognition Research Existing Psychological Results on Cognitive Limitations

9. Augmented Cognition: Opportunities Enhance learning and memory via reminder systems & methods Automate specific aspects of problem solving & filtering Modulate / triage communications Develop new visualizations, other info. rendering to increase rate of “concept attainment”---raising effective human-computer bandwidth

10. Augmented Cognition: Opportunities Support sensory fusion, judgment, action under uncertainty Provide vigilant monitoring for situations requiring attention Monitoring, knowledge about human and machine errors Extend abilities to coordinate, collaborate with other people and systems Design multimodal, context-aware systems for more efficient, natural human—computer collaboration

11. Low-Hanging(?) Fruit • Information filtering and triage • Mixed-initiative interaction • Context-sensitive UI / computing • Intelligent reminding • Managing attention and disruption • Visualization & comprehension • Human-error--aware systems • Automated sensor fusion • Judgment de-biasing systems

12. Risks in Building a Strong, Valuable Program • Timing: Too early…? • Need more complete understanding of human bottlenecks in real world? • Need more sophisticated computational procedures? • Strength of research • Rigorous, foundational work may be needed • Poor initial stabs may lead to premature discontinuation, disappointment • Focus • Choosing some foci for early traction • Avoiding replication of work in commercial sector

13. Several Sample Projects • Vista: Models for limiting, controlling information based on coarse value of revealed information. • Handsfree decision support:Fluid interaction for providing recommendation, updates; context-sensitive speech. • Lumiere: Context- and competence-based assistance • Lookout: Mixed-initiative interaction. • Priorities: Automated assignment of urgency, alerting, communication, caching by value.

14. Several Sample Projects • Notification Platform:Attention in HCI, notification architectures, UI, psych studies of disruption, reminders. • Continual computation:Use background computation to forecast goals, attention; guide precomputing, caching. Background query, document tracking. • Learning information goals:Learning about a user’s goals from behavior, log files. • Qualia:Models of attention, perception in multimedia rendering and communication • Context-aware devices:Enhance UI with implicit gestures, context-sensing.

15. Project Drill Down & Demos