Presented by: Prof Mark Baker

Using a RESTful messaging and registry system to support a range a distributed applicationshttp://acet.rdg.ac.uk/projects/tycho Presented by: Prof Mark Baker SSE, University of Reading Tel: +44 118 378 8615 E-mail: Mark.Baker@computer.org Web: http://acet.rdg.ac.uk/~mab mark.baker@computer.org

Outline • General Introduction. • Representational State Transfer (REST). • Tycho’s motivation, architecture and deployment. • Some Tycho applications: • GridRM, • Necho, • SORMA, • Map Service. • LinkSphere. • Utilities. • Summary and future work. mark.baker@computer.org

General Introduction • Various technologies seem to appear in waves, some are taken up and are successful, and others die out quickly. • I have been working in the parallel, distributed computing and HPC arena for 20+ years: • Seen lots of interesting technologies come and go! • CORBA, Jini… etc… • Spent a lot of time working on grid technologies (1995 onwards) and e-Science… now interested in e-Research! • However, the Web 2.0 area seems to have been one of those domains of interest that has taken off like a rocket! mark.baker@computer.org

Representational State Transfer (REST) • The representational state transfer (REST), was originally described by Roy Fielding in his doctoral dissertation. • REST is a collection of principals that are technology independent, except for the requirement that it be based on HTTP: • All components of the system communicate through interfaces with clearly defined methods and dynamic code. • Each component is uniquely identified through a link (URL). • A client/server architecture is followed (web browser and server). • All communication are stateless. • The architecture is tiered, and data can be cached at any layer. mark.baker@computer.org

RESTful • These principals map directly to those used in the development of the Web, and account for much of its success. • The HTTP protocol, its interface of methods (GET, POST, HEAD, and so on), the use of URLs, HTML, and JavaScript, all map directly to these principals. • Overall, REST can be described as a technology and platform-independent architecture where loosely coupled components communicate via interfaces over standard Web protocols. • The software, hardware, and data-centric designs maximise system efficiency, scalability, and network throughput. mark.baker@computer.org

Tycho - motivation • Tycho was conceived in 2003 in response to a need by the resource-monitoring project for a “light-weight”, scalable and easy to use wide-area distributed registry and messaging system. • Tycho’s was released in 2006; a number of modifications have been made to the system to make it easier to use and more flexible. • Tycho has been utilised across a number of application domains including – a listing is shown at the end of the talk. • Tycho can be found here… • http://acet.rdg.ac.uk/projects/tycho/ mark.baker@computer.org

Tycho • We decided that we wanted system that did communication and a had a registry… • Tycho is a based a Service Oriented Architecture – it uses the publish, subscribe and bind paradigm. • Design Philosophy: • The system has an architecture similar to the Internet, where every node provides reliable core services, and the complexity is kept, as far as possible, to the edges. • Tycho’s core is small, simple and efficient, so that it has a minimal memory foot-print, is easy to install, and is capable of providing robust and reliable services. • More sophisticated services can then be built on this core and are provided via libraries and tools to applications. • This allows Tycho to be flexible and extensible so that it will be possible to incorporate additional features and functionality. mark.baker@computer.org

Tycho’s Architecture • Producers, Consumers and Mediators. • Mediators create virtual registry. mark.baker@computer.org

Tycho Deployment • Mediators allow producers and consumers to discover each other and establish remote communications: • URL published in registry, • P2P virtual registry (VR). • Consumers can search for producers in VR. • Adding functionality to registry – metadata and Semantic Web technologies. • Consumers typically subscribe to receive information or events from producers. • Producers gather and publish information for consumers: • Creating producer/mediator pairs, • Gather metadata from source and publishing it in VR. mark.baker@computer.org

Tycho consists of the following components: Mediators that allow producers and consumers to discover each other and establish remote communications, Consumers that typically subscribe to receive information or events from producers, Producers that gather and publish information for consumers. There is an asynchronous messaging API. In Tycho, producers and/or consumers (clients) can publish their existence in a Virtual Registry (VR). Tycho’s Deployment mark.baker@computer.org

Tycho Applications mark.baker@computer.org

GridRM • GridRM is an extensible, wide-area, monitoring system that specialises in combining data from existing agents and monitoring systems so that a consistent view of the underlying resources and services can be achieved, regardless of heterogeneity. • Gateways provide access to local resource information at each site. • Clients connect to gateways to perform resource queries and to subscribe for events. • GridRM uses Tycho in a number of ways to bind together clients and Gateways for wide-area communications, and to provide the basis of an event mechanism (both wide-area events to clients, and events from local monitoring). mark.baker@computer.org

GridRM Structure • Global layer of peer-related gateways: • Which in turn have a local layer that interacts with the local data sources, and/or a hierarchy child gateways. Gateway Gateway mark.baker@computer.org

XDB • Integrated search facilities across multiple archival databases. • The test system searches for records from the Silchester IADB, held in an ordinary relational database, and a classics-oriented collection of Roman-era inscriptions found in Vindolanda database, which is based on a RDF store. • The cross-database search engine is now a key component in http://LinkSphere.org mark.baker@computer.org

Necho • The Necho project is creating a multi-tiered peer-to-peer system, which is akin to BitTorrent, for distributing multi-terabyte scientific datasets across the Internet. • The concepts for this project first appeared when working with the Sloan Digital Sky Survey (multi-terabyte), where it was necessary to split the original dataset up and use a modified version of WGET to download and update the database. • It consists of a hierarchical Peer-to-Peer (P2P) system that is based around shared-portal services and unique peers donated by participating individuals and organisations. mark.baker@computer.org

Necho • The goal of the project is to combine P2P, volunteer computing and social networks to provide a way to distribute, contribute to, and manage very large datasets. • Necho uses Tycho to distribute, index and retrieve chunks of data that comprise each overall dataset. • We are currently testing Necho against Azureus, our single tier version of Necho is proving to be much faster. mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management) mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management) • The Economically Enhanced Resource Manager (EERM) exists at each resource provider’s site, and acts as a centralised resource allocator (e.g. business goals and resource requirements) in order to achieve maximum economic profit and resource utilisation. • The EERM utilises GridRM to obtain resource information for system and per-process monitoring in order to determine if Service Level Agreements (SLAs) have been violated. • The EERM is composed as a confederation of loosely bound components for scalability and availability reasons. • Tycho provides messaging and event mechanisms. mark.baker@computer.org

SORMA (Self-Organizing ICT Resource Management) mark.baker@computer.org

Map Service • Geographical maps are used to describe the Earth’s surface and its contours: • These maps are hosted on servers called “map servers” around the globe and can be used by scientists in many ways. • The environmental science community lacks a searchable registry of available Map Services. • Created Services to find these service via Tycho. mark.baker@computer.org

LinkSphere.org mark.baker@computer.org

LinkSphere.org • LinkSphere is a JISC funded project that includes • A Social Web site that helps researchers and academics get to know each other. • The site has fine-grained security. • We are also joining together databases around the University. mark.baker@computer.org

LinkSphere.org • We are joining the various databases is based on the XBD project. • The idea is instead of the databases being remote, we can search across them, and create a list, that has a ranking. • This means that researchers will be able to do useful searches, plus they will get links to ePrints, and books in the Library. • The idea is to make it easier to R&D within the University. • The underlying infrastructure is based on Tycho, which includes metadata and URIs too. mark.baker@computer.org

Tycho Utilities • BonJour - Locates devices, these devices offer a local network using multicast DNS records. • Synchronous API - Many application need blocking communications, so we have add this to Tycho. • HTTP pipelining - To optimise the communication performance, it requires us to open multiple parallel pipes, with buffering, when sending data to a remote destination. • Lightweight Transactions - There is a transaction manager, which uses main memory that decouples the performance of transactions from the disk. • Additional Caching - to improve performance by altering caching in the mediator to include local data-store queries in addition to remote responses. mark.baker@computer.org

Summary • Tycho is a RESTful asynchronous messaging system with an integrated peer-to-peer virtual registry. • Since, Tycho was released in 2006, we have increasingly used the system to support a range of distributed applications. • We have used Tycho as RESTful services are easy to install and use. • Tycho uses HTTP (HTTPS) and Sockets (SSL) for communications. • Internally, it uses SQL as the query language and uses LDAP LDIF to mark up responses from the VR. • None of these standards have changed, and it ensures that applications, based on Tycho, will continue to work for the foreseeable future. mark.baker@computer.org

List of Applications • GridRM, • VOTEchBroker, • Slogger, • XDB, • DNWAy • Necho, • SORMA, • Map Service, • LinkSphere.org, • Sun SPOT Wireless Sensor Networks. mark.baker@computer.org

Presented by: Prof Mark Baker