1 / 35

Make my desktop app great with touch, mouse, and pen

Make my desktop app great with touch, mouse, and pen. Nick Waggoner Senior Program Manager 4-022. Demo: Photo annotation. Agenda. Input and hit-testing Windows touch l anguage Panning and zooming Pen and ink. Where we are today . Deep and m ature Diverse ecosystem

rosina
Download Presentation

Make my desktop app great with touch, mouse, and pen

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Make my desktop app great with touch, mouse, and pen Nick Waggoner Senior Program Manager 4-022

  2. Demo: Photo annotation

  3. Agenda • Input and hit-testing • Windows touch language • Panning and zooming • Pen and ink

  4. Where we are today • Deep and mature • Diverse ecosystem • Primarily mouse and keyboard

  5. PC Form Factorshave evolved.

  6. Key platform APIs Desktop Applications Win32 apps Input and Interaction Rendering and composition Input and interaction DirectManipulation DirectComposition DirectManipulation Remainder of Win32 InteractionContext Direct2D, DirectWrite Realtime Stylus Ink Recognition Dynamic DPI InteractionContext Realtime stylus Ink recognition WM_POINTER Family Direct3D WM_Pointer family Windows kernel services

  7. Input and hit-testing

  8. WM_POINTER • Unified messages for touch, mouse, and pen. • EnableMouseInPointer(…) • GetPointerInfo(…) / GetPointerInfoHistory(…) • GetPointerTouchInfo(…) / GetPointerPenInfo(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  9. Touch hit testing Control C Button 1 • RegisterTouchHitTestingWindow(…) • WM_TOUCHHITTESTING • EvaluateProximityToRect(…) or EvaluateProximityToPolygon(…) • PackTouchHitTestingProximityEvaluation(…) Button 2 DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  10. Demo: Moving to WM_POINTER & WM_TOUCHHITTESTING

  11. Windows touch language

  12. Windows touch language DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  13. InteractionContext • Initialization • CreateInteractionContext(…) • SetInteractionConfigurationInteractionContext(…) • RegisterOutputCallbackInteractionContext(…) • Feeding input • AddPointerInteractionContext(…) • ProcessPointerFramesInteractionContext(…) • Handling output DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family voidINTERACTION_CONTEXT_OUTPUT_CALLBACK( _In_opt_  void *clientData, _In_      const INTERACTION_CONTEXT_OUTPUT *output );

  14. InteractionContext – Output structures typedefstruct INTERACTION_CONTEXT_OUTPUT { INTERACTION_ID     interactionId; INTERACTION_FLAGS  interactionFlags; POINTER_INPUT_TYPE inputType; float              x; float              y; union { INTERACTION_ARGUMENTS_MANIPULATION manipulation; INTERACTION_ARGUMENTS_TAP          tap; INTERACTION_ARGUMENTS_CROSS_SLIDE  crossSlide; } arguments; } INTERACTION_CONTEXT_OUTPUT;

  15. InteractionContext – Output structures typedefstruct INTERACTION_ARGUMENTS_MANIPULATION { MANIPULATION_TRANSFORM   delta; MANIPULATION_TRANSFORM   cumulative; MANIPULATION_VELOCITY    velocity; MANIPULATION_RAILS_STATE railsState; } INTERACTION_ARGUMENTS_MANIPULATION;

  16. Demo: Integrating the touch language

  17. Panning and zooming

  18. DirectManipulation • Drives panning and zooming in the system • Slide and pinch • Encapsulates the personality • Inertia • Bounce effects • Snap-points DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  19. DirectManipulation • Pre-declare your behaviors/interactions DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family Harness the power of Windows with DirectComposition

  20. DirectManipulation – Key interfaces/messages • Initialization • IDirectManipulationManager • IDirectManipulationViewport • IDirectManipulationContent • Hit-testing • SetContact() • WM_POINTERDOWN / DM_POINTERHITTEST • Manipulation begins • WM_POINTERCAPTURECHANGED DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  21. Adding panning and zooming

  22. DirectManipulation - threading • Best performance by integrating DirectManipulation with a 3 thread model: • UI thread • Delegate thread • Independent hit-test thread • IDirectManipulationManager::RegisterHitTestTarget(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  23. Pen and ink

  24. RealTimeStylus • Independent of the UI thread • Handwriting recognition DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  25. RealTimeStylus- Key interfaces/methods • Initialization • IRealTimeStylus • AddStylusSyncPlugin(…) • Processing Input • IStylusSyncPlugin • StylusDown(…), Packets(…), StylusUp(…) • IStrokeBuilder • BeginStroke(…), AppendPackets(…), EndStroke(…) • Handwriting Recognition • InkRecognizerContext • Recognize(…), BackgroundRecognize(…), • BackgroundRecognizeWithAlternates(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  26. RealTimeStylus- Key interfaces/methods IRealTimeStylus DirectManipulation IStylusSyncPlugin InteractionContext Realtime stylus Ink recognition IStrokeBuilder InkRecognizerContext WM_Pointer family

  27. Adding ink and recognition

  28. Performance is critical for both inking as well as panning and zooming experiences.

  29. Recap

  30. Takeaways • Use WM_POINTER for touch, mouse, and pen. • Use the InteractionContext to incorporate the Windows touch language into your app. • Use DirectManipulation for great panning and zooming. • Differentiate your app by making ink pop with the RealTimeStylus APIs. DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

  31. Resources • Related talks • 3-017 - Power up your desktop app with Windows • 4-021 - DirectComposition: Smooth composition and animation for desktop apps • 4-9147 – Exploring customized touch interactions for your Windows store app • Reference documentation • WM_POINTER - http://msdn.microsoft.com/en-us/library/windows/desktop/hh454916(v=vs.85).aspx • InteractionContext - http://msdn.microsoft.com/en-us/library/windows/desktop/hh437192(v=vs.85).aspx • DirectManipulation - http://msdn.microsoft.com/en-us/library/windows/desktop/hh446972(v=vs.85).aspx • RealTimeStylus - http://msdn.microsoft.com/en-us/library/windows/desktop/ms701683(v=vs.85).aspx • InkRecognizerContext - http://msdn.microsoft.com/en-us/library/windows/desktop/ms696371(v=vs.85).aspx • Process touch input with DirectManipulation sample • http://go.microsoft.com/fwlink/?LinkID=309852&clcid=0x409 • Design guidelines • Touch target sizes - http://msdn.microsoft.com/en-us/library/windows/apps/hh465415.aspx#touch_targets

  32. Acer Iconia W3, Surface Pro, and Surface Type Cover Get your goodies Device distribution starts after sessions conclude today (approximately 6:00pm) in the Big Room, Hall D. If you choose not to pick up your devices tonight, distribution will continue for the duration of the conference at Registration in the North Lobby.

  33. Required Slide *delete this box when your slide is finalized Your MS Tag will be inserted here during the final scrub. Evaluate this session • Scan this QR codeto evaluate this session and be automatically entered in a drawing to win a prize!

More Related