Direct Manipulation of User Interfaces for Migration

1. Intelligent User Interfaces 2006 Jan 29 - Feb 1 2005, Sydney, Australia Direct Manipulation of User Interfaces for Migration J. P. Molina1,2, J. Vanderdonckt1 and P. González2 1 Université catholique de Louvain 2 University of Castilla-La Mancha www.usixml.org

2. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia

3. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia

4. Introduction • End users of modern interactive systems face a wide variety of computing platforms to support their interactive tasks, from mobile phones to wall screens • The specific needs of the user’s environment at a given moment, or its evolution, can make necessary to remove one application from one platform and make it run in another one, that is, to migrate the application Intelligent User Interfaces 2006, Sydney, Australia

5. Introduction • Migration can be: • Total, when the whole application is migrated • Partial, only the UI is migrated; if it is decomposed into control and presentation, then migration can be: • Control-oriented, migrates only the control component • Presentation-oriented, the control remains • Mixed, parts of both control and presentation are migrated • A user interface (UI) is said to be migratable if it holds the migration ability Intelligent User Interfaces 2006, Sydney, Australia

6. Introduction • A special UI is required for controlling the run-time migration of the UI of the interactive systems • This UI is referred to as the meta-user interface, implemented in very different ways: • Sytem-initiated, the system initiates the migration • User-initiated, the user initiates the migration • Mixed-initiated, user and system collaborate to perform the migration Intelligent User Interfaces 2006, Sydney, Australia

7. Introduction • Related work Intelligent User Interfaces 2006, Sydney, Australia

8. Introduction • Elements represented in in the meta-UI Intelligent User Interfaces 2006, Sydney, Australia

9. Introduction • Tasks are usually distributed among platforms, but without any consistent way to distribute them, the space between platforms introduces a discontinuity that may disrupt the fluency of the interactive task • Direct manipulation is proposed to tackle that discontinuity, and thus avoid possible usability problems Intelligent User Interfaces 2006, Sydney, Australia

10. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia

11. MigriXML • In order to make user interface migration as natural as direct manipulation, a meta-UI in the form of a virtual environment has been developed • MigriXML enables the user to carry out that migration process without the physical presence of the involved platforms, in a virtual and interactive way, relaying on UsiXML language and a set of models Intelligent User Interfaces 2006, Sydney, Australia

12. MigriXML • To generate the virtual reality scene representing the migration environment, a suite of models is used describing relevant aspects of the problem in terms of UsiXML: • The Concrete User Interface • The User Model • The Platform Model • The Environment Model • Describes properties of interest, which can be physical, psychological and organizational Intelligent User Interfaces 2006, Sydney, Australia

13. MigriXML: Environment Model • The physical part consists of: • a scene model based on VRML97/X3D, • the multi-surface interaction ontology, • and the Stanford topology model. • Each scene is composed of physical resources and a series of planes, which are decomposed into areas • Each area may be an interaction surface: a monitor on a table, a wall screen, or any area where the UI is projected and/or recognized with computer vision • Each area is described by attributes like dimensions, relative position, relationships with other objects, etc Intelligent User Interfaces 2006, Sydney, Australia

14. Interactive system architecture GUI application Arch meta-model or ‘Slinky’ meta-model Platform Layered structure MigriXML architecture Intelligent User Interfaces 2006, Sydney, Australia

15. General MigriXML architecture User’s environment GUI application Arch meta-model fits the UsiXML models Platform Virtual Hardware (VHW) Virtual Machine (VM) Virtual Window Manager (VWM) Migration Manager (MM) MigriXML architecture Intelligent User Interfaces 2006, Sydney, Australia

16. Implementation • This work focuses on the original part of providing a virtual control environment for migrating parts or whole of a UI from one platform to another • To address its implementation, the VRML97/X3D languages have been used • User Interfaces and the Arch meta-model: • Interaction toolkit component: VUIToolkit • Domain-specific component: Java and Javascript Intelligent User Interfaces 2006, Sydney, Australia

17. Implementation • VUIToolkit • A set of VRML97/X3D PROTOs that transforms the standard plain widgets into a truly 3D representation (Virtualized User Interface) • This toolkit was developed starting from the object classes described in the Concrete Model of UsiXML language Intelligent User Interfaces 2006, Sydney, Australia

18. Implementation • Java and Javascript • Most VRML97/X3D browsers can interpret Javascript code and execute Java code • VRML97/X3D specifications describe two programming interfaces to access the scene graph: • SAI (Scene Authoring Interface) • EAI (External Authoring Interface) • Allowed us to leave aside, for this first implementation of MigriXML, the complex integration of emulators for each specific platform Intelligent User Interfaces 2006, Sydney, Australia

19. Interactive Java/Javascript application, virtualized user interface rendered in the 3D window of VRML97/X3D browser Implementation Intelligent User Interfaces 2006, Sydney, Australia

20. Interactive application, virtualized user interface and virtual platform in MigriXML Implementation Intelligent User Interfaces 2006, Sydney, Australia

21. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia

22. Case study • The user’s environment is a small office, with five different platforms Intelligent User Interfaces 2006, Sydney, Australia

23. Case study • Display characteristics of each platform Intelligent User Interfaces 2006, Sydney, Australia

24. Case study • UsiXML was used to describe the environment • The virtual world was the result of a structured process of creation: • Requirements: from UsiXML specifications, designers knew the details of the environment • Preparation: compilation of audio-visual material; sketches and mock-ups were created • Design, test and optimize: model creation, edit appearance, enhance world Intelligent User Interfaces 2006, Sydney, Australia

25. Case study • User interfaces were specified in UsiXML too • Different tools have been developed to help the UI designer to visually create this models • IdealXML [Montero et al., 05] Intelligent User Interfaces 2006, Sydney, Australia

26. Case study • User interfaces were specified in UsiXML too • Different tools have been developed to help the UI designer to visually create this models • IdealXML • GrafiXML [B. Michotte] Intelligent User Interfaces 2006, Sydney, Australia

27. Case study • Two different user interfaces Intelligent User Interfaces 2006, Sydney, Australia

28. 1 2 3 4 Case study • Migrating from one laptop to the other Aspire 200015”, 16:10 1280x800 PT-LB10SU4:3 800x600 Latitude15”, 4:3 1024x768 Intelligent User Interfaces 2006, Sydney, Australia

29. 1 2 3 Case study • Migrating from a laptop to the Pocket PC Aspire 200015”, 16:10 1280x800 e7503,8”, 3:4240x320 Intelligent User Interfaces 2006, Sydney, Australia

30. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia

31. Migration process in detail • The migration process is divided in four stages (A, B, C and D), each one having a finite set of steps, representing a total amount of 14 steps Intelligent User Interfaces 2006, Sydney, Australia

32. Migration process in detail • Stage A: Select an interactive application Intelligent User Interfaces 2006, Sydney, Australia

33. Migration process in detail • Stage A: Select an interactive application The user presses button (M) to grab the window, which follows the user wherever she points at Intelligent User Interfaces 2006, Sydney, Australia

34. Migration process in detail • Stage A: Select an interactive application The action is transmitted to the corresponding Virtual Window Manager (VWM) by an event Intelligent User Interfaces 2006, Sydney, Australia

35. Migration process in detail • Stage A: Select an interactive application The VWM forwards the information to the Migration Manager (MM) Intelligent User Interfaces 2006, Sydney, Australia

36. Migration process in detail • Stage B: Select target platform Intelligent User Interfaces 2006, Sydney, Australia

37. Migration process in detail • Stage B: Select target platform The MM broadcasts a message to all platforms Intelligent User Interfaces 2006, Sydney, Australia

38. Migration process in detail • Stage B: Select target platform Platforms change their state to ‘wait-for-selection’, a TouchSensor is activated Intelligent User Interfaces 2006, Sydney, Australia

39. Migration process in detail • Stage B: Select target platform The user selects the target platform by clicking on its screen Intelligent User Interfaces 2006, Sydney, Australia

40. Migration process in detail • Stage B: Select target platform The Virtual Hardware (VHW) casts an event to inform the VWM Intelligent User Interfaces 2006, Sydney, Australia

41. Migration process in detail • Stage B: Select target platform The VWM forwards the message to the MM Intelligent User Interfaces 2006, Sydney, Australia

42. Migration process in detail • Stage C: Migrate to target platform Intelligent User Interfaces 2006, Sydney, Australia

43. Migration process in detail • Stage C: Migrate to target platform The MM commands to detach the application from source platform Intelligent User Interfaces 2006, Sydney, Australia

44. Migration process in detail • Stage C: Migrate to target platform 10 The MM broadcast amessage to the rest ofplatforms, which changeto normal execution Intelligent User Interfaces 2006, Sydney, Australia

45. Migration process in detail • Stage C: Migrate to target platform The MM takes the UsiXML specification and re-generates the app UI Intelligent User Interfaces 2006, Sydney, Australia

46. Migration process in detail • Stage D: Adapt to target platform Intelligent User Interfaces 2006, Sydney, Australia

47. Migration process in detail • Stage D: Adapt to target platform The MM asks the target platform to host the re-generated application Intelligent User Interfaces 2006, Sydney, Australia

48. Migration process in detail • Stage D: Adapt to target platform The application is launched, and its virtualized UI added to the scene graph of the target VHW Intelligent User Interfaces 2006, Sydney, Australia

49. Migration process in detail • Stage D: Adapt to target platform The virtualized UI is adapted to the target screen and windowing system Intelligent User Interfaces 2006, Sydney, Australia

50. Contents • Introduction • MigriXML • Case study • Migration process in detail • Conclusion Intelligent User Interfaces 2006, Sydney, Australia