Incorporating Context and User Feedback in Pen-Based Interfaces

1. Incorporating Context and User Feedback in Pen-Based Interfaces Martin Szummer & Phil Cowans Microsoft ResearchCambridge, UK

2. The Task � Interpreting Ink --- show selection and editing graphically--- show selection and editing graphically

3. One possible architecture [screenshot of Journal] temporal grouping, spatial grouping WARNING: loses momentum; takes time and details are not the main point critique: sequential processing (temporal followed by spatial) hard decisions how correct errors?

4. Making the most of imperfect recognizers The recognizer will make errors. To reduce the impact of the recognizer: Make decisions: soft local (but based on global context) late incorporate user feedback A hypothetical system: Microsoft tablet: show the design Soft � keep uncertainty around; less of a decision Local � When only a small part Late � keep uncertainty from previous stages results in fewer decisions Avoid decisions where possible This talk about the technology needed to support the above.A hypothetical system: Microsoft tablet: show the design Soft � keep uncertainty around; less of a decision Local � When only a small part Late � keep uncertainty from previous stages results in fewer decisions Avoid decisions where possible This talk about the technology needed to support the above.

5. Proposal Probabilistic model for simultaneous use of: spatial and temporal information local and contextual information classification and segmentation user feedback

6. Overview Task overview Description of the model Labeling only model Labeling and grouping model Training and inference Examples and results Further work

7. Previous Work Kumar, Hebert [2003] Discriminative random fields. Saund [2003] Perceptual organization. Mankoff [2001] Uncertainty in user interfaces

8. Task Stages Divide the input data into ink fragments Then� Group the fragments into perceptually salient objects And� Label each object as either a container or connector

9. Example - Input

10. Example - Fragmentation

11. Example � Grouping / Labeling

12. Fragmentation Input data consists of ink strokes, described by sampled pen locations. Strokes may span multiple objects. Divide strokes into straight (within a tolerance) fragments. Fragments are assumed to lie in a single object.

13. The Model Use a Conditional Random Field: Undirected graphical model conditioned on observed data. No need to model ink data (which we already know). For now, just consider labeling.

14. Potentials

15. Nonlinearities Exponential: Probit: (Gaussian CDF) Epsilon is a �noise� parameter.

17. Graph Construction Construct a graph with one vertex per ink fragment. Add edges between nearby fragments. Triangulate graph. Calculate interaction potentials for all edges.

18. Graph Construction II Speed is dependent on the tree-width of the graph. By constraining the tree-width it should be possible to limit the complexity of the algorithm.

19. Features Spatial features Angles, lengths, distances� Temporal features Stroke ordering, same stroke� Template features T-Junction identification� 61 observation, 37 interaction features in total

20. Priors Three feature categories: Binary Counting Continuous Continuous features are replaced by histograms Use a correlated Gaussian prior on continuous features

21. Example Weights

22. Labeling And Grouping Use a slight modification of the model Now three interaction categories: Same object (implies same label) Different object, same label Different object, different label

23. Advantages Of This Approach Joint labeling Makes use of contextual information Simultaneous labeling and grouping Processes can reinforce each other Handles groups directly Doesn�t define groups via labeling Results in a �cleaner� model

24. Training Train by finding MAP weights on example data. Training data set was 40 diagrams, from 17 subjects with a total of 2157 fragments. Maximise using quasi-Newton gradient ascent (BFGS in MATLAB). Evaluate function and gradient using message passing.

25. Message Passing Just like standard sum-product, but we are summing over groupings and labelings together rather than just labelings. Fortunately efficient computation via message passing is still possible. Messages are now a list of possible groupings and labelings of the separator variables.

26. Message Passing II Message update rule: Marginals:

27. Message Passing III

28. Inference Unseen data is processed by finding the most probable grouping and labeling. Use the max-product variant of the message passing algorithm. Messages are now the �best you can do� given the separator configuration.

29. Example - Raw Ink

30. Example � Labeling Only

31. Example � Labeling And Grouping

32. Example 2 - Raw Ink

33. Example 2 � Labeling Only

34. Example 2 � Labeling And Grouping

35. MATLAB Demo

36. Grouping Error Measured using the metric: Not really sure how this relates to �perceptual� error. There�s potentially a better way of doing this via mutual information.

37. Results: Accuracy ---- Grouping error should have a more intuitive measure---- Grouping error should have a more intuitive measure

38. Results: Speed Median evaluation time for labeling only: 0.35 seconds. Median evaluation time for labeling and grouping: 13 seconds. (And that�s not very optimised, using MATLAB!)

39. Conclusions Exact inference is possible in reasonable time. This approach is capable of providing high-quality results. Joint labeling improves performance. Labeling and grouping simultaneously helps too.

40. Further Work Multiclass model: Dashed lines, text, arrows, � Relationships: Which containers are connected together. More �global� features: Closed paths, convex containers, � Possibly via re-ordering of N-best lists. More data, better features, �

41. Observation Weights

42. Interaction Weights

43. File: 33.xmlFile: 33.xml

45. 30.Xml Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide 30.Xml Bad: 51 is mislabelled and misgrouped too, and not corrected on next slide

47. Publication example 6.Xml From 6_uncorrected.fig (left), and 6_corrected2 (right) ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})6.Xml From 6_uncorrected.fig (left), and 6_corrected2 (right) ink_demo('X:\Data\OrgChartRecognition\renamed-frag\6.xml', {}, {[41, 43], [50 55]})