Data Sharing Middleware Prototype (DSMP) for Information Dissemination Among Heterogeneous Sources

1. Data Sharing Middleware Prototype (DSMP) for Information Dissemination Among Heterogeneous Sources Quarterly Review Meeting, January 21, 2009 Hairong Qi (PI), University of Tennessee Xiaorui Wang (co-PI), Seddik Djouadi (co-PI), UT Oak Ridge National Laboratory* Oracle Corporation* Microsoft Research Rutherford Appleton Laboratory, UK* * Oracle, Microsoft Research, and ORNL verbal commitments for in-kind support (consulting and research software)

2. Contact Information • Academia • Hairong Qi, 865-974-8527, hqi@utk.edu, 1508 Middle Dr., 319 Ferris Hall, EECS Department, University of Tennessee, Knoxville, TN 37996 • Xiaorui Wang, 865-974-0627, xwang@eecs.utk.edu, 421 Ferris Hall, UT • Seddik Djouadi, 865-974-5447, djouadi@eecs.utk.edu, 307 Ferris Hall, UT • Raghul Gunasekaran, 865-385-5857, raghul@utk.edu, 536 SERF, UT • Ming Chen, Ying Sun, Samir Sahyoun, Ben Taylor, UT Graduate Students • Research Laboratories • Frank DeNap, 865-576-8786, denapfa@ornl.gov, Oak Ridge National Laboratory, PO Box 2008, MS6085, Oak Ridge, TN 37831 • Mallikarjun Shankar, 865-574-2704, shankarm@ornl.gov, Oak Ridge National Laboratory, PO Box 2008, MS6085, Oak Ridge, TN 37831 • Steve Fisher, RAL, s.m.fisher@rl.ac.uk, Rutherford Appleton Laboratory (RAL), UK • Industry, Private sectors • Dieter Gawlick, Ronny Fehling, Aravind Yalamanchi, 650-560-8706, {dieter.gawlick, ronny.fehling, aravind.yalamanchi}@oracle.com, Oracle Corporation • Vijay Dialani, Microsoft Research Center

3. Collaborative Team • Academia • University of Tennessee • Vanderbilt University • Research Laboratory • ORNL (Oak Ridge National Laboratory, US) • RAL (Rutherford Appleton Laboratory, UK) • Industry • Microsoft Research • Oracle

4. Project Description • The objective of this project is to develop a data sharing middleware that is able to handle multiple distributed data sources and dynamically changing items, and to assist in real-time information dissemination across multiple agencies for homeland security purposes. • The ultimate target scenarios are first responders and consequence response at the urban area of Memphis (e.g., Shelby County) with stakeholders including the Fire Department, Weather Services, the E911 Operations Center, Law Enforcement Agencies, etc.

5. Minutes from Last Quarterly Review Meeting • General discussion • Can INFOD behave like an aggregator? • Can there be registry of registry? • Human factors of INFOD haven’t been discussed (e.g., no sight, no hearing) • Clarify the difference between the traditional model and the INFOD model • Security is an essential issue to be discussed in INFOD • Different consumers should have different right time to receive the message • Suggest to demonstrate INFOD in a more confined environment • Make closer connection of INFOD and the real-time meta-data modeling • The uniqueness of the proposed plume tracking model should be clarified and emphasized. Instead of moving the sensors, we can think about a sensor grid • Need to design a scenario where under the same setup, INFOD would make a difference • Comments on client-side development • Pre-meeting discussion on Privacy with data sharing systems • Frances Butler and Janet Murrill

6. Existing pub/sub systems Topic-based, Type-based, Content-based Message-oriented Middleware (MOM) Service-oriented system (binding AFTER event) Repository – data center, processing center Static system. Extending the system is difficult. Establishes a framework for info flow Matches publishers and consumers based on information needs expressed through subscriptions and limited by properties Event-based system (binding BEFORE event) Registry - NOT a data (event) repository Better handling of dynamics. Extensibility is good. (Vocabulary, e.g., NIEM) Alerting system Consumer/Publisher Landscape Assessment INFOD Model Traditional Model Subscriber (Subscription) Consumer INFOD Registry Publisher Push Publisher Push Repository Consumer Publisher Push Consumer Pull Subscriber/ Consumer Publisher Right Info  Right Person @ Right Time

7. Entry Notification (by Publishers) INFOD Components and Setup Procedures Publisher Consumer • Register community property and data vocabularies in the INFOD registry • Define subscriptions binding entities, defining events and which entity needs to be alerted on which other entities presence. • Entities register to the INFOD registry • Perform matching and notification message sent to matched entities. Consumer Subscriber INFOD Registry Publisher Entry Subscriber Entry Subscription Consumer Entry Data Source Entry Property Vocabulary Instance Data Vocabulary Property Vocabulary Resource – not an entry Creation of resource Reference (EPR) Notification (by INFOD registry)

8. Today’s Presentation Outline • Development track • Demo: INFOD Web Application - Raghul Gunasekaran • Research track • Real-time Metadata Matching in INFOD - Ming Chen • Application Scenario: Chemical Plume Tracking - Samir Sahyoun • Discussion of future plan

9. Collaborative Opportunities Y-12 Privacy policy and NIEM ORNL Testbed setup Emergency plan Oracle In-kind support MSU Real-time operations support for emergency evacuations UT - Emergency Plan Team All the research findings and software developments are accessible through public domains, maintained at UT http://panda.ece.utk.edu/wiki/InfoD

10. Project Timeline The development of the DSMP (Task 1.2) has been divided into a 4-phase implementation plan. Because of the close collaboration with Oracle, Microsoft Research, ORNL, and RAL, we are able to finish all four phases of prototype development ahead of schedule. Phase 1 - simplest scenario with a known data vocabulary and a trivial subscription Phase 2 - 2 publishers services, 2 consumer services with the addition of property vocabularies Phase 3 - multiple data vocabularies, publish, consumer, and subscriber services Phase 4 - a standard notification interface

11. Budget Information Project budget (June 5, 2007 - May 31, 2009): $400,000 Spending as of December 31, 2008: $250,759.16

12. Commercialization Progress The INFOD working group is approached by OGC (Open Geospatial Consortium) to join the consortium. This would help getting more publicity of the product on geospatial and location based services Potential feature to Oracle product line

13. IP STATUS Open source development Will be available through sourceforge to stimulate broader participation

14. Achievements Finished all four phases of the prototype development (ahead of schedule) Papers accepted “Control-based real-time metadata matching for information dissemination,” 14th IEEE Int. Conf. on Embedded and Real-Time Computing Sys and App, Taiwan, August 2008. (Acceptance rate: 26%) “Dynamic target classification in wireless sensor networks,” Int. Conf. on Pattern Recognition (ICPR), Tampa, FL, December 8-11, 2008. “Source localization using stochastic approximation and least squares methods,” 2nd Mediterranean Conference on Intelligent Systems and Automation (CISA'09). Students graduated Y. Sun, Dynamic Target Classification in Wireless Sensor Networks, MS Thesis, Summer 2008. Presentations “INFOD Use Case Scenario & Demo,” Open Grid Forum (OGF), Feb 2008, Boston “An INFOD Reference Implementation,” Open Grid Forum (OGF), Oct 2007, Seattle Papers submitted/in-preparation “Dynamic cyber-attack detection and classification based on the information dissemination model” “Plume estimation and tracking using mobile sensors,” To be submitted to IEEE Conference on Decision and Control 2009. “INFOrmation Dissemination middleware”

15. INFOD Development Raghul Gunasekaran

16. Implementation Model Stage 3 Stage 2 – INFODweb Stage 1 Publisher App/ Service PostgreSQL Database INFOD Registry Service Tomcat Client Library OC4J INFOD Web Service Servlets Servlets INFOD Spec. ConsumerApp/ Service ORACLE 10g ORACLE PL/SQL PL/SQL Java procedures Client Applications Web Management Interface Client Environment  app.jar : API for communication between applications and INFODweb.

17. Mutual Filtering – INFOD Registry Subscriber/Subscription Consumer Publisher Note: All constraints and meta data (in identical colors) needs to be evaluated and MUST be true for communication between a publisher and a consumer.

18. Mutual Filtering – INFOD Registry Publisher Subscription Mapping Publisher RowID Publisher Subscriber Subscription RowID Publisher URI SubscriptionURI Consumer constraint Publisher constraint Match results Subscriber constraint Consumer constraint Meta Data Subscription RowID Meta Data DataSource RowID Consumer RowID Publisher RowID Subscription RowID Consumer RowID Consumer RowID Consumer URI Publisher Consumer Mapping Consumer Subscription Mapping Publisher constraint Subscriber constraint Meta Data Consumer

19. INFOD Registry – System Performance Plot 1 – System Scalability Table 1: Mutual Filtering time for meta data with 500 XML elements and no. of entities representing the number of publisher, consumer and subscribers individually. Method 1 – No intermediate Step Method 2 – Intermediate Mapping table Plot 2 – Performance w.r.t. constraints for 500 entities M1 – All constraints; M2 – No constraints on Subscriber

20. INFOD ‘Hello, World!’ Example

21. Use Case description Scenario An accident has occurred, and a bystander reports the event to the E911 center. The E911 center then requests for police, ambulance and/or fire truck to be dispatched based on the current description of the event. As the first officer arrives at the location, the officer reports more details of the accident to the E911 center and also updates on the developments and states of the incident. The E911 center based on the description of the event and the current state of the event alerts the necessary services (fire, medical and police) for action. If resources in a particular region are not sufficient, then the E911 center needs to make a decision in calling for additional resources based on their capabilities and availabilities.

22. Use Case description The example scenario here is to demonstrate the functioning of INFOD. The scenario might not completely reflect the real world. The goal behind this demo is to have a simpler version of the real world for developers/collaborators/users to understand how INFOD functions. Setup • The E911 center receives information of any emergency within an area, and based on the incident information it communicates with first responders such as police, hospital, fire and medical services. The E911 center operates closely with the police (control center) • The police control center instructs and maintains information of its resources in an active duty. Similarly, the fire service and medical services control and command over their resources. • The E911 center instructs these control centers and the control center communicates to their resources.

23. First Responder Use Case Fire Service B Observer Fire Service A Accident E911 Center Medical Emergency Service Police Control Center Hospital B Hospital A

24. First Responder Use Case Fire Service B Observer Fire Service A E911 Center Medical Emergency Service Accident Police Control Center Officer at Accident spot Hospital B Hospital A

25. INFOD - First Responder Use Case INFOD helps • Monitor resources • All entities registered in the system • Gain knowledge of the capability/availability of the resource • In terms of property vocabulary instances • Associate entities (Publishers – Consumers) • Handle incident objects • What defines as event at the publisher • Policies/ constraints which need to be enforced based on the event description • Select consumers (dynamic consumers) based on the event

26. INFOD - First Responder Use Case INFOD Registry Consumer Fire Service B Consumer Consumer Observer Fire Service A Publisher Consumer E911 Center Medical Emergency Service Consumer Accident Police Control Center Datasource/ Consumer Consumer Hospital B Hospital A Consumer Consumer Consumer DataSource/ Consumer

27. INFOD in 4 steps 1. Register community property and data vocabularies in the INFOD registry 2. Entities register to the INFOD registry Create Publisher, Consumer and Subscribers Create and update property instances – Meta Data 3. Define subscriptions Property Constraints – on Meta Data Data Constraints – policies/ event constraints Dynamic Consumer Constraint – identify consumers based on event 4. Mutual Filtering in the INFOD registry and notification message sent to matched entities. INFOD - First Responder Use Case

28. INFOD Web Services Step 1 : Register Property and Data vocabularies for a community

29. Property Vocabulary

30. Data Vocabulary

31. INFOD Web Services Step 2 : Create Consumers and Publishers Property Constraints for $publishers in fn:collection("$$INFODpublisher") where $ publishers //OrganizationSubUnitName=”RedCross”

32. INFOD Web Services

33. INFOD Web Services Step 3 : Subscribers and Subscriptions Property Constraints for $publisher in fn:collection("$$INFODpublishers") where $publisher//OrganizationSubUnitName=”E911Center” for $firstresponders in fn:collection("$$INFODconsumers") where $ firstresponders//OrganizationSubUnitName=”RedCross” Data Constraints declare namespace $data = http://infod.firstrespondernet.com/AlertDataVocabulary; let $msg1 := for $firstresponders where $data:AlertStatus = ‘Actual’ and $data:EventCategory = ‘CBRNE’ and $data:EventSeverity > ‘Moderate’ return {$data, $data:Instruction = ’Evacuate people in the region’ } let $msg2 := for $firstresponders where $data:capAlertStatus = ‘Actual’ and $data:EventCategory = ‘Fire’ and $data:EventSeverity > ‘Moderate’ return { $data:Substance, $data:Volume, $data:EventCategory } Dynamic Consumer Constraint for $firstresponders where $firstresponders//OrganizationSubUnitName=”Police” return msg1 for $firstresponders where firstresponders//OrganizationSubUnitName=”FireService” return msg2

34. INFOD Web Services Step 4 : Notification Messages

35. Ming Chen, Ben Taylor, Xiaorui Wang Integrated Control of Matching Delay and CPU Utilization in InformationDissemination Systems

36. Information Dissemination System • Hundreds even thousands of subscriptions are registered; • Different subscriptions have different priorities and different costs to evaluate • Updates arrive with unpredictable intervals • Valuable Information at the Right Time(VIRT).

37. Challenges • Updates may arrive at any inter-arrival intervals • We can NOT trigger all subscription reevaluations upon each update. • Reevaluating all subscriptions may cause severe system workload and unacceptable long delays • We can NOT accurately calculate how many subscriptions to reevaluate to meet with the specified matching delay. • The time of reevaluating a subscription may vary significantly.

38. Solutions Our goals: matching delay system throughput Batching windowHong long? SubscriptionsHong many?

39. Architecture • Integrated controller: to control both response time and CPU utilization • Admission controller: to enforce maximum average matching interval for low-priority subscriptions

40. System Modeling • Relationship between the matching delay and CPU utilization and the batching window size and job budget. • System identification (Matlab) • White noise. • Randomly choose a job budget such that the CPU utilization > 50%. Difference Equation System Model

41. Controller Design LQR controller: • Larger Q leads to faster response to workload variations; • Larger R makes the system be less sensitive to system noise; • LQR controller is designed by using the Matlab command • dlqry to solve the optimization problem.

42. Experimental Setup • Two servers • Server1: INFOD system, Oracle 10g • Server2: Java code to simulate requests from consumers, publishers, etc • JDBC connectivity • Updates follow a Poisson process • Set point • 2s for matching delay • 0.8 for CPU utilization • 10s for low-priority subscription reevaluation

43. Exp 1: Selection of set points of CPU utilization • a trade-off between the control accuracy and the system throughputs.

44. Exp 2: Control performance • Show the control performance of the integrated controller by comparing it with the open-loop system. oscillations Overloaded

45. Exp 3: Comparison with baselines • Delay-only • Util-only

46. Exp 4: Admission controller • Control performance of the admission controller

47. Future Work • Stability analysis on model variations • different platforms; • different execution time of subscriptions; • More analysis on the admission control

48. Chemical Plume TrackingProgress and Future work Samir S. Sahyoun Advisor: Seddik M. Djouadi

49. Contents Summary of Achievements till Last Meeting New Results Future Work

50. Summary of Achievements till Last Meeting