The GRIDCC Instrument Element: from prototypes to production environments

1. EU FP6 Project The GRIDCC Instrument Element: from prototypes to production environments Roberto Pugliese On Behalf of the GRIDCC Collaboration

2. The GRIDCC project: Goals & Objectives • The GRIDCC - Grid Enabled Remote Instrumentation with Distributed Control and Computation – project has the main aim to bring Instruments to the GRID • It is a 3-years EU FP6 project started in September 2004 • Web site: www.gridcc.org • More in particular the project goals are: • Definition of a “Instrument Element” allowing a standard remote access to any type of instrumentation • Tight integration between instrument grid and classical computational grid • Human interaction with Grids via Virtual Control Room (collaborative environment) • Enactment of complex workflows

3. The GRIDCC partners

4. Instruments Grid Computational Grid CE IE IE DATA Instrument Element CE Instrument Manager Information & Monitoring System Grid SE Problem Solver VCR Workflow VCR ES VCR VCR Collaborative Environment Execution Services GRIDCC: Architecture (I) • The GRIDCC services are Web Services compliant, according to the WS-I convention. • Any type of client WS-I compliant, can access the GRIDCC services without any specific software library. • X509 based security is used • When performances are required , a kerberos based mechanism is used to access IEs. Service Oriented Architecture – SOA

5. Instrument elements (IE) Storage Element (SE) Instrument elements (IE) Storage Element (SE) Instrument Element (IE) Storage Element (SE) Virtual Control Room Storage Element (SE) Storage Element (SE) Storage Element (SE) Grid Virtual Control Room Compute element (CE) Compute element (CE) Computing Element (CE) Execution Service WfMS WMS AS GRIDCC Architecture Information and Monitoring Services (IMS) Global Problem Solver Security Services Collaborative Services (CS) Web Service Interface WMS Work Management System WfMS Work Flow Mng System AS Agreement Service Information System (BDII) Broker

6. GridCC PermanentTest Bed VCR, IE WfMS VOMS IE, CE/CREAM,SE, LFC AS, KrbServer WMProxy VCR, SE, IE, BDII

7. GRIDCC main target areas Remote process control Accelerator control (Tele-) Biomedicine Robotics Automotive Electronic microscopes (Large-scale) scientific experiments High energy particle physics (Radio-) Telescopes GRIDCC Middleware Widely Sparse Instrumentation Power Grids Monitoring of the territory Monitoring of the sea Geo-hazard prediction Distributed laboratories Transportation monitoring Sensor network

8. GRIDCC pilot applications PowerGrid Particle Accelerator High Energy Physics CMS Meteorology Device Farm Geohazard Monitoring

9. Accessing GRID Enabled Instrumentation: Requirements • Interactive access to allow the control and the monitor of the instrumentation. Uniform access to the physical devices. • Fast Data Publishing to allow the publishing acquired by the instrumentation • Information (logs, errors, etc.) Publishing to track the behaviour of the instrumentation and possibly to correct the malfunctioning • GRID integration. The acquired data should be visible by the GRID protocols both to : • Move them to large GRID data storage (Storage Element) • Process them from the GRID computing power (Computing Element)

10. A new 4 channels element to fit the requirements: The Instrument Element (IE) Custom (JMS, ..... ) Discovery Data/Info Pubblishing I’m here Instrument Element Instrumentation Contorl Web Service GRID Access SRM/GRIDFTP Instruments Access Advance Reservation Custom QoS

11. Instrument Element: Versatile I/O multichannel • The IE permits • SOA accessible operations to control and monitor the instruments (via VIGS), such as: • execute a command • get / set parameters • different data outputs: • data mover to/from a grid Storage Element (via SRM) • high bandwidth channel for data publishing (via IMS). • low bandwidth channel for logs, states etc. (via IMS). Logs, Errors, States, Monitors Data Subscribers VCR IMS Grid Interaction IE Commands Storage Element (SE) Storage Element (SE) VIGS Storage Element (SE) SRM ES Status Parameters Instruments VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SRM Storage Resource Management

12. Data Flow State Flow Error Flow Monitor Flow Control Flow Access Control Manager Instrument Element Architecture IMS The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. IMS Resource Service SRM/SE GridFTP SE Instrument Element Problem Solver VIGS Data Publisher IMS Instrument Manager IMS Proxy Control Manager Data Collector Control Manager Event Processor FSM Engine Instrument Protocols Input Manager Resource Proxy Real Instruments

13. Access Control Manager Configuration Examples IMS Resource Service Configuration A, B, C can run concurrently sharing services like RS, PS, IMS SE SRM/SE GridFTP Instrument Element Problem Solver IMS Data Publisher Instrument Manager Top IM Top IM B C Top IM A IM IM IM IM IM IM Resource Service Instruments Pool

14. IE Instrument Instrument Instrument Instrument Instrument Element Framework Characteristics Resource Service IMS • The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled. • Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments. • the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM). • relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP). • Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS). Data Mover Local Problem Solver VIGS Access Control Manager Instrument Manager Instrument Manager Data Collector IMS Proxy Control Manager Control Manager Event Processor FSM Engine Input Manager Resource Proxy

15. Access Control Manager Instrument Manager IMS IMS Resource Service SRM/SE GridFTP SE Instrument Element Problem Solver Data Publisher VIGS IMS Instrument Manager Instrument Manager IMS Proxy Control Manager Data Collector

16. Instrument Manager Control Manager IMS Proxy Event Processor FSM Engine Data Collector Input Manager Resource Proxy Instrument Manager Plug-in modules to interface to the instruments Customizable Control Flow Data Flow Monitor Flow State Flow Error Flow Instruments • IM is composed by 3 main components: • - Control Manager: • - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs,errors/state/log/monitor messages. • - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability • - FSM. A finite state machine is implemented • - Resource Proxy. It handles all the outgoing connections with the resources. • Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. • - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS

17. Data Subscribers (Monitor, data analysis, etc.) Access Control Manager Control Manager Event Processor FSM Engine Instrument Instrument Instrument Input Manager Resource Proxy Instrument DB persistency Data Collector RMM-JMS Resource Service IMS Data Mover enable/disable data stream Local Problem Solver Instrument Element VIGS IE File System Instrument Manager IMS Proxy Control Manager Data Collector • Status • data collection • data publishing on RMM-JMS • data on file in progress • data on DB to be started • sync commands (enable/disable • done trough the “set param” of the • IE façade (VIGS) data stream

18. Access Control Manager Resource Service IMS IMS Resource Service SRM/SE GridFTP SE Resource Service Instrument Element Problem Solver Data Publisher VIGS IMS Instrument Manager IMS Proxy Control Manager Data Collector

19. Resource Service Architecture Partition/Configuration retrieve methods Discovery Manager Available Resources Partition and Lock setting methods Subscribe Manager Partition Definitions RS Data Bases • The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). • A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) • RS stores the configuration data of the resources and download them to resource target when necessary • Resources can be discovered, allocated and queried. • It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. • A periodic scan of the registered resources keeps the configuration database up to date. Configuration setting methods Partition&Lock Manager Configuration Definitions Discovery methods Configuration Manager

20. Access Control Manager Information and Monitor Service IMS IMS Resource Service SRM/SE GridFTP IMS SE Instrument Element Problem Solver Data Publisher VIGS IMS Instrument Manager IMS Proxy Control Manager Data Collector

21. Information and Monitor System (IMS) IMS Proxy • The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs. • The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. • The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. Publish / Subscrib IMS Proxy PERSISTENCY SUBSCRIBERS SUBSCRIBERS IMS Proxy

22. IE IMS Fast Data Publishing: RMM-JMS Same data are sent to several subscribers. Multicast protocols can have a benefic impact on performances IE Data Producer Data Subscribers • JMS provides a standard set of • APIs for the communication • Many commercial and academic • JMS implementations • both in C/C++ and Java • (NaradaBrokering, Sun, IBM) • GRIDCC (IBM Haifa lab) has • implemented a Reliable Multicast • protocol (RMM) JMS compliant • RMM-JMS works within a LAN • but an efficient bridge technology • has been developed to allow • inter-LAN multicast communication

23. Access Control Manager Embedded SRM compliant Grid Storage Element (SE) Resource Service IMS SRM SE Backend GRID accessibile data Local Problem Solver Instrument Element VIGS IE File System Instrument Manager • Status: installed and running • SRM 2.2 • STORM implemetation (INFN) • Bestman implementation (Berkeley)) • XFS IE file system

24. IE IE IE Discovery Instruments (I) • The discovery of the instruments or of the IEs is an issue when the number of elements is high • We can have two cases: • Quasi static cases • The number of IEs are well defined and the single IE is quite complex with a good hardware support • In this case a register based discovery mechanism can be used. • GRIDCC tesbed is using the LCG BDII (Berkely Database Information Index) based on LDAP • The information collected in the BDII follow a GLUE schema and can be used for match making querries Querring Client Querring Client Querring Client BDII Local BDII

25. IE IE IE IE M1 M2 Querring Client Querring Client Querring Client Discovering Instruments (II) • Dynamic cases • The number of IEs can change very quickly, they are very simple devices, often with poor hardware support • The discovery is just use to know which are the online IEs • A new approach has been developed based on Peer to Peer (P2P) protocols • See demo on this workshop

26. Instrument Elements at Work

27. DAQ IM DAQ RS DAQ IMS FB RB FF xdaq xdaq xdaq Pilot applications: CMS - I • The GridCC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory. • CMS Magnet Test and Cosmic Challenge (MTCC) I and II, a milestone in the CMS construction, positively carried out. CMSDetector CMS Control Structure User Interface Top IE CSC IE Tracker IE HCAL IE DAQ IE RPC IE Trigger IE DT IE ECAL IE

29. CMS MTCC phase I and phase II carried out in 2006 Scale MTCC versus CMS FEDs: 20 out of 600 3% EVB RUs: 6 out of 600 1% Filter Farm: 14 out of 2000 0.3% Trigger rate: 100 Hz out of 100 kHz 0.1% Event size: 200 kB out of 1 MB 20% IEs 15 out of 150 10 %

30. Virtual Control Room Virtual Control Room Pilot applications: Power grid • GRIDCC deployed to monitor: • A 50kW generator • A 1 kw Photo-Voltaic array See DEMO on this workshop Power Grid. GRIDCC deployed to monitor a 50kW generator and a 1 kW Photo-Voltaic array Instrument Manager Power Grid V.O. Instrument Element Solar Panel ... Gas

31. Elettra Synchrotron Pilot applications: Remote Operation of an Accelerator See DEMO on this workshop

32. GRIDCC: other applications • Meteorology (Ensemble Limited Area Forecasting) • Weather forecasting system to detect hazardous weather • Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories • The Device Farm consists of a pool of Measurement Instruments for Telecommunication Experiments • Geo-hazards: Remote Operation of Geophysical Monitoring Network • The monitoring net will be characterized by different levels of activity: stand-by, pre-alert, alert, plus a control modality • An event worth to be monitored is for example the evolution of a “landslip”

33. Custom Board Xilink Virtex IV Grid Custom Electronics FPGA PPC 405 JavaVM Web Service GridCC IM Linux Instrument Element: Scalable on embedded systems 1 Gbps Ethernet IE Instrument Manager Custom Logic Standalone Axis Montavista www.montavista.com USER INTERFACE JamVM http://jamvm.sourceforge.net

34. IE technologies • Web Service compliant (WS-I) • Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE • All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application • Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project • Sun • Narada Brokering • RMM - JMS (GridCC IBM) • MySQL and Oracle are used as Data Base for the RS • SRM interface version 2.2 used

35. Conclusion - I • The GridCC project is integrating instruments into the “classic” computational/storage Grids. • One of the novel concepts introduced by GridCC are: • The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid • hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times • Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. • The IE is highly customizable and can be adapted in different environments.

36. Conclusion - II • 3 pilot applications deployed, one of them in a real production environment • The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation • Several heterogeneous pilot applications are deploying and running the IE middleware • We support and encourage the adoption of our middleware in other projects/experiments.