1 / 18

Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams

Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams. Marlon Pierce, Galip Aydin, Zhigang Qi Community Grids Lab Indiana University. California Real Time Network.

felix-serrano
Download Presentation

Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Topic-Based Publish/Subscribe for Managing Real Time GPS Streams Marlon Pierce, Galip Aydin, Zhigang Qi Community Grids Lab Indiana University

  2. California Real Time Network Continuous GPS Stations (CGPS) are depicted as triangles while the Real-Time stations are represented as circles. Image is obtained from SOPAC GPS Explorer at http://sopac.ucsd.edu/projects/realtime How does one manage all the data generated by the 85 stations? How can you get just the data you want? Note this is fundamentally different from traditional request/response style Web Services.

  3. RDAHMM: GPS Time Series SegmentationSlide Courtesy of Robert Granat, JPL GPS displacement (3D) length two years.Divided automatically by HMM into 7 classes. • Complex data with subtle signals is difficult for humans to analyze, leading to gaps in analysis • HMM segmentation provides an automatic way to focus attention on the most interesting parts of the time • Features: • Dip due to aquifer drainage (days 120-250) • Hector Mine earthquake (day 626) • Noisy period at end of time series

  4. Use Case - GPS Sensors • GPS is used to identify long-term tectonic deformation and static displacements. SCIGN has 250 Real-Time GPS Stations. • SOPAC GPS networks: • 8 networks for 80 stations produce 1Hz high resolution data. • Socket based real-time binary-RYO format access is available. • We developed filters to provide multiple format (RYO, ASCII, GML) real-time streaming access. • OHIO principle (a general principle required by DOD) and chain of filters. • Our Architecture • Uses publish/subscribe based NaradaBrokering for managing real-time GPS streams • Utilizes topics for hierarchical organization of the sensors • Deploys successive data filters ranging from format translators to data analysis codes • Could potentially be used to run RDAHMM clones to monitor state changes in the entire GPS network • We are partner in a pioneering project to use the real-time GPS data for the first time in this context. 4

  5. Processing Real-Time GPS Streams RYOPorts 7010 Scripps RTDServer Raw Data 7011 7012 NB Server GPS Networks ascii2gml ascii2pos Single Station Displacement Filter RDAHMM Filter Station Health Filter ryo2nb ryo2ascii ascii2pos Single Station RDAHMM Filter ryo2ascii ryo2nb /SOPAC/GPS/CRTN01/RYO Raw Data /SOPAC/GPS/CRTN01/ASCII /SOPAC/GPS/CRTN01/POS /SOPAC/GPS/CRTN01/DSME 5 A Complete Sensor Message Processing Path, including a data analysis application.

  6. Application Integration with Real-Time Filters • Station Monitor Filter records real-time positions for 10 minutes and calculates position changes • Graph Plotter Application creates visual representation of the positions. • RDAHMM Filter records real-time positions for 10 minutes and invokes RDAHMM application which determines state changes in the XYZ signal. • Graph Plotter Application creates visual representation of the RDAHMM output. 6

  7. GIS Grid Usage Model – Earthquake Science • Supporting geophysical repositories and real-time sensors is essential • To analyze a typical earthquake it is important to access to precise measurements of the initial earthquakes and aftershocks • To support earthquake forecasting and the time and spatial positions of the forecasts • PI can be used with existing data • RDAHMM can be used with the real-time data • Earth Science field is moving from a previously data poor field to a data rich world. We will have thousands of sensors spread around the world. (i.e. GPS sensors, InSAR satellites)

  8. Recording and Replaying Sensor Streams • Filters can be used to record and replay scenarios, such as Earthquakes in GPS case. • We developed RYO Recorder and RYO Publisher Filters. • The RYO Recorder creates daily archives of the GPS Streams. • RYO Publisher can be used to play daily or certain segments of the records. • We replayed the 2004 Southern California Earthquake using Parkfield GPS network archive • These filters are used in the performance and scalability tests.

  9. SensorGrid Performance Tests • Two Major Goals: System Stability and Scalability • Ensuring stability of the distributed Filter Services for continuous operation. • Finding the maximum number of publishers (sensors) and clients that can be supported with a single broker. • Investigate if system scales for larger number of sensors and clients.

  10. Test Methodology RYO Publisher 2 • The test system consists of a NaradaBrokering server and a three-filter chain for publishing, converting and receiving RYO messages. • We take 4 timings to calculate mean end-to-end delivery times of GPS measurements. • The tests were run at least for 24 hours. • GridFarm001-008 servers are used in these tests. 1 3 RYO To ASCII Converter 4 NB Server 1 Ttransfer = (T2 – T1) + (T4 – T3) Simple Filter

  11. 1- System Stability Test • The basic system with three filters and one broker. • The figure shows average results for every 30 minutes. • The average transfer time shows the continuous operation does not degrade the system performance.

  12. 2 – Multiple Publishers Test • We add more GPS networks by running more publishers. • The results show that 1000 publishers can be supported with no performance loss. This is an operating system limit. Topic 2 Topic 1A Topic n Topic 1B

  13. 3 – Multiple Clients Test 1000 Clients • We add more clients by running multiple Simple Filters which subscribe to the same ASCII topic. • The system can support as many as 1000 clients with very low performance decrease. Topic 1A Topic 1B

  14. Extending Scalability • The limit of the basic system appears to be 1000 clients or publishers. • This is due to an Operating System restriction of open file descriptors (1024 for Red Hat Linux) which can be increased by changing OS parameters. • To overcome this limit we create NaradaBrokering networks with linking multiple brokers. NB supports scalable linkage of the brokers for building tree like architectures. • We run 2 brokers to support 1500 clients. • Number of brokers can be increased indefinitely, so we can potentially support any number of publishers and subscribers.

  15. 4 – Multiple Brokers Test RYO Publisher • NaradaBrokering allows creation of Broker networks. • We create a two-broker network. • Messages published to first broker can be received from the second broker. • We take timings on each broker. • We connect 750 clients to each broker and run for 24 hours. We chose 750 clients to stay well below the saturation limit. • The results show that the performance is very good and similar to single broker test. RYO To ASCII Converter Topic 1A NB Server 1 Topic 1B Simple Filter 1 Simple Filter 2 Simple Filter 750 Simple Filter 751 Simple Filter 752 Simple Filter 1500 NB Server 2 NB Server 2 Topic 1B

  16. 4 – Multiple Brokers Test

  17. Real-Time Filters Test Results • The RYO Publisher filter runs at 1Hz and publishes 24-hour archive of the CRTN_01 GPS network, which contains 9 GPS stations. • The single broker configuration can support 1000 clients or publishers (GPS networks - 9000 individual stations). • The system can be scaled up by creating NaradaBrokering broker networks. • Message order was preserved in all tests.

  18. Acknowledgement • Galip Aydin: Designer and implementer • ZhiGang Qi: SensorGrid Performance Tests • We thank Prof. Yehuda Bock and his group at SIO for their help with real-time GPS data streams. • Robert Granat, JPL: RDAHMM developer

More Related