AR-Enabled Environment: Tracking based AR-Setups

1. 2nd Joined Advanced Student School Ubiquitous Tracking based AR-Setups Benjamin Fingerle Christian Wachinger Benjamin Fingerle, Christian Wachinger

2. Procedure • We will present a scenario in which a user - Gerhard - gets into the benefits of an AR-enabled Environment • First the scenario will be presented from the user’s perspective • In a 2nd stage we’ll look behind the surface and see how the AR-environment can be modeled using the DWARF Service concept • Finally we’ll go even deeper into detail and present how spatial relationships of objects can be represented using the Ubiquitous Tracking Framework Benjamin Fingerle, Christian Wachinger

3. The User’s Equipment • Gerhard wears an Optical See Through Head Mounted Display (OST-HMD) • Up on this HMD two 6DOF markers - one optical, one magnetic - are rigidly mounted • Additionally a stereo-vision camera is installed on top of his head • Gerhard wears special gloves with attached optical 6DOF markers Benjamin Fingerle, Christian Wachinger

4. A Day in Gerhard’s Life … • Gerhard strolls down the TUM hallway greeting colleagues of his through closed doors • Eventually he reaches his own office and steps in • Sitting down in front of the desk he reads a virtual message of his Russian friend Vladimir from St. Petersburg asking for a chess game • He starts the RemoteChess application whereupon a virtual chess board appears aligned on the desk and the virtual counterpart of Vladimir takes a seat across from him Benjamin Fingerle, Christian Wachinger

5. … A Day in Gerhard’s Life • Suddenly an emergency request for instant help by his befriended Spanish colleague José shows up • José asks Gerhard - a renowned surgeon - for advises regarding a difficult surgery José is currently conducting • Just where a minute ago the chess board was visible a 3D - Model showing the patient crops up • Gerhard studies the patient while having a look from different sides and getting computer tomography images registered on the patient on demand • In vivid discussion with José a life is saved Benjamin Fingerle, Christian Wachinger

6. The Scenario Can Be Modeled With Services <service name=“OpticalTracker”> <need name=“video” type=“VideoStream”> <connector protocol=“sharedMemory”> </need> <need name=“marker” type=“MarkerData”> <connector protocol=“ObjectReference”> </need> <ability name=“markerPose” type=“PoseData”> <attribute name=“location” value=“$(marker.location)”> <attribute name=“identity” value=“$(marker.identity)”> <connector protocol=“NotificationPush”> </ability> </service> • Services • Have needs • Refined by predicates • Offer abilities • Refined by attributes • Connectors • Offer interfaces for data exchange • Service Managers • One for each network node • Detect mutually satisfying services • Provide services with connectors Context = {location, identity, activity, time} can be modeled using Predicates and Attributes Benjamin Fingerle, Christian Wachinger

7. Services are Specified Using XML <service name=“HMDOpticalTracker”> <need name=“ ... </need> <ability name=“HeadPose” type=“PoseData”> <attribute name=“location” value=“$(landmark.location)”> <attribute name=“identity” value=“Gerhard”> <connector protocol=“NotificationPush”> </ability> <ability name=“markerPose” type=“PoseData”> ... </ability> </service> Benjamin Fingerle, Christian Wachinger

8. Hallway Services Benjamin Fingerle, Christian Wachinger

9. Office Services Benjamin Fingerle, Christian Wachinger

10. Mutually Satisfying Services are Found by ServiceManager • For each network node (e.g. Room, Hall, …) one ServiceManager exists • This ServiceManager observes needs and abilities of those services belonging to his particular network node • When a match is found the ServiceManager provides all involved parties with connector-objects • These connectors offer push-, pull-, or shared memory access to abilities via various interfaces Benjamin Fingerle, Christian Wachinger

11. Matching of Mutually Satisfying Services Benjamin Fingerle, Christian Wachinger

12. Ubiquitous Tracking - The Formal Layer • The crucial problem of Augmented Reality applications is the correct tracking of objects • It is common to integrate the tracking procedure into the application • Different tracking technologies are combined to get better results • To release the AR application of tracking and to enable a seamless integration of new tracking devices a formal layer is introduced • The formal framework, called UbiquitousTracking, forms the formal layer Benjamin Fingerle, Christian Wachinger

13. Ubiquitous Tracking – The Formal Framework • Requests to the framework about the spatial relationship of objects can be send • The answer delivers the optimal relationship available Graph-model: • nodes == objects • edges == spatial relationships Benjamin Fingerle, Christian Wachinger

14. Ubiquitous Tracking - The Underlying Graph Model • Properties of a spatial relationship • Represents the transformation and translation of the source coordinate system to the target coordinate system • Attributes characterizing the quality of the relationship • Three different types of graphs • Real relationship graph • Measured relationship graph • Inferred Relationship Benjamin Fingerle, Christian Wachinger

15. Real Relationship Graph Each pair of objects has at every point of time a geometric relationship Benjamin Fingerle, Christian Wachinger

16. Measured Relationship Graph • Estimates of relationships are just available for certain objects for discrete points of time • Additionally Attributes characterizing the quality of the measurement exist • Error function describing the quality of a relationship Benjamin Fingerle, Christian Wachinger

17. Inferred Relationship Graph • Knowledge about spatial relationships not just for discrete points of time is necessary • Knowledge has to be inferred about the relationship of objects • Error functions help to find optimal inferences Benjamin Fingerle, Christian Wachinger

18. What Is a Good Error Function? Aim: Finding optimal paths in the graph! = { Latency, Update frequency Confidence value Pose accuray Time to live } Benjamin Fingerle, Christian Wachinger

19. Optimisation of Graph Algorithms • Precomputing the Data Flow Graphs • Infrequently changing structure of the spatial relationships • Infrequently changing Attributes • No dependency on the pose measurements • Grouping Nodes • Several nodes can be represented by a single supernode • Enabling level of detail hierarchies  Faster graph search What is the coordinate system of the new supernode? Benjamin Fingerle, Christian Wachinger

20. Security & Safety Aspects • What if the “WhatsBehind” Service can be used in front of every door? • What if your boss knows what you have done this weekend? • What if other users know about every place where you have been this week? • What if Indra the intern uses the AR setup of Gerhard? • What if Eve corrupts the data showing you the false distance to the oncoming bus? • What if you can’t see the car because the latest sports news occluded it? Privacy Restrictions Authenticity Benjamin Fingerle, Christian Wachinger

21. Finding Matches Forms Crucial Performance Issue Matching problem of high computational complexity • One Service Manager for each Network Node • Number of potential matchings grows exponentially with the number of services available inhibiting scalability Different strategies for coping with this issue could include • Heuristics, based on context information, realized as graph searches on the spatial relationship graph • Interpretation of matching problem as predicate logic formula, applying specialized algorithms known from model checking • … Benjamin Fingerle, Christian Wachinger

22. Issues Concerning Graph Representation Possible ways of storing the spatial relationship graph include • One graph-service holding the complete Graph + all graph algorithms applicable- contradicts the distributed computing paradigm • Each Service knows its adjacency + complies with distributed computing paradigm - not all graph algorithms applicable Benjamin Fingerle, Christian Wachinger

23. Issues Concerning Access to the Graph Information Possible ways of accessing spatial relationships include • Request formulated as need for spatialRelationship with predicates source and target, abilities offered by • Certain GraphInformationServices + Graph can be split up between different such services - different graph search strategies for the same relation cannot coexist • One service for each relationship instantiated by special relationShipGenerators + relationships instantly available - different graph search strategies for the same relation cannot coexist - number of services raises dramatically • Requesting a graphInformerObject providing an interface getSpatialRelationShip(source, target) + Objects implementing different graph searches can coexist - centralization Benjamin Fingerle, Christian Wachinger

24. Open Questions • How much information about the world do we have to put in the error function? • Which graph algorithms have to be applicable? • Which graph representation allows theses algorithms? • How to perform context changes? • How to enable new users to enter AR environments? • What has then to be calibrated? Benjamin Fingerle, Christian Wachinger

25. Critical success factors • Number of AR ready buildings • Development and acceptance of standards • Prices and convenience • Legal issues • Products and applications urging the user to buy Benjamin Fingerle, Christian Wachinger