1 / 18

Distributed Control and Monitoring of High-Level Trigger Processes on the LHCb on-line Farm

This paper discusses the control and monitoring of high-level trigger processes on the LHCb online farm, including the experiment control system, event filter farm, and the use of the GAUDI software framework. It also covers the implementation of hierarchical control using a commercial SCADA system.

sharrow
Download Presentation

Distributed Control and Monitoring of High-Level Trigger Processes on the LHCb on-line Farm

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Distributed Control and Monitoringof High-Level Trigger Processeson the LHCb on-line Farm Philippe Vannerem CERN / EP ICALEPCS - Oct03

  2. Contents • the Experiment Control System of LHCb • the Event Filter Farm • GAUDI, the software framework for event processing • GAUCHO, how to glue GAUDI to SCADA • Communicating histograms • Conclusions Philippe Vannerem

  3. ECS Scope DCS Devices (HV, LV, GAS, Temperatures, etc.) Detector Channels Trigger Front End Electronics Experiment Control System Readout Network Processing/Filtering Storage DAQ External Systems (LHC, Technical Services, Safety, etc)

  4. ECS: Generic SW Architecture ECS T.S. LHC DAQ DSS DCS ... ... GAS Status & Alarms DetDcs1 DetDcsN DetDaq1 Commands SubSys1 SubSys2 SubSysN Dev1 Dev2 Dev3 DevN To Devices (Hardware or Software) • Distributed Hierarchical Control • Commercial components and industrial standards Philippe Vannerem

  5. ECS: Hierarchical control • Each node is able to control and monitor its subtrees: • Propagate commands to child level • Summarize information for the parent level • Implement specific behaviour & Take local decisions • Sequence & Automate operations • Recover errors • Partitioning • Exclude one or more of its subtrees which then can run stand-alone • User Interfacing • Present information and receive commands • Filter and display alarms DCS Tracker Muon HV Temp HV GAS Philippe Vannerem

  6. ECS: implementation • Implementation in commercial SCADA system software: PVSS II • device description + access (OPC, Profibus, drivers) • alarm handling+display • user interface builder + control scripting • archiving • Joint COntrols Project between the LHC experiments: • The Framework: hierarchical control templates + tools and control modules for common HW in PVSS • Important building block: SMI++: • State Management Interface, distributed control • Abstract behaviour modeling Finite State Machines • Automation & Error recovery with a Rule based system • LHCb ECS is based the JCOP Framework and will extend it for its specific needs. Philippe Vannerem

  7. Switch Switch Switch Switch Switch SFC SFC SFC SFC Switch Switch Switch Switch CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU Level-1 Traffic Gb Ethernet HLT Traffic Mixed Traffic LHCb’s DAQ&trigger architecture Level-1Traffic HLTTraffic Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE TRM 126Links 1100 / 25 = 44 kHz 5.5 GB/s 323Links 40 / 10 = 4 kHz 1.6 GB/s Multiplexing Layer 64 Links 32 Links Readout Network L1-Decision Sorter TFCSystem 94 Links 7.1 GB/s … StorageSystem 94 SFCs CPUFarm ~1800 CPUs

  8. Event Builder Switch SFC SFC SFC CPU CPU CPU . . . CPU CPU CPU . . . . . . . . . CPU CPU CPU Event Filter Farm Control • EFF: ~1800 commodity rack-mounted CPUs • CPUs+SFCs linked to ECS PCs in separate Ethernet LAN • ECS Tasks: • Hardware control (remote boot, reset,…) + monitoring (CPU usage, memory, IO, etc.) • Software control + monitoring • Level-1+ High-Level Trigger algorithms are implemented using the GAUDI framework Philippe Vannerem

  9. GAUDI Framework: design choices • Separation between data and algorithms • Separation between “transient” and “persistent” data • Generic component interfaces • OO Framework implemented with C++, runs on Win2000+Linux Philippe Vannerem

  10. GAUDI: component architecture • Interface defined in C++ pure virtual class • User specifies the components to be used in the Joboptions file • DLLs are loaded at run-time • Algorithms can be configured in the joboptions file Philippe Vannerem

  11. Interface Requirements • Real-time system to use on the EFF • Interface to the trigger algorithms written in Gaudi event data processing framework. • Interface to the SCADA-based ECS implemented in PVSS. • Light-weight software: • Low CPU load to interfere as little as possible with event data crunching. • Disk-less operation on the nodes. • Low network traffic. • Flexible service-like component: easy to change/add monitored variables by physicist-Gaudi-users (trigger people). • Scalable with number of nodes • Compatible with farm partitioning. Philippe Vannerem

  12. GAUCHO: GAUdi Component Helping Online • Gaucho package: monitor and control the Gaudi work on the farm. • “users” (=future trigger algorithm programmers) can use this component in their Gaudi application to use the Gaudi-style Algorithm Service “MonitorSvc” and then declare the variables to be published in their algorithms. • “MonitorSvc” publishes internal variables (integer, float, string) and histograms in the transient store using the DimMonitorServer. • A DimPropertyServer gets/sets values of the properties of the ApplicationMgr and Algorithms (configuration, steering). • A DimController with internal states receives commands via DIM and steers the ApplicationMgr. Philippe Vannerem

  13. farm node - Gaudi DIM ApplicationMgr monitor node - PVSS HLT Algorithm DimController DIM API HistogramSvc Gaucho EventDataSvc ControlMgr DimPropertySvr DetectorDataSvc ControlPanel DimMonitorSvr MonitorSvc MonitorPanel Gaudi-Gaucho-SCADA Philippe Vannerem

  14. Inter-process tool: DIM DIM: • Distributed Information Management system • Light-weight inter-process communication package based on the client/server paradigm • Asynchronous interrupt-driven data transfer across platforms • Here: C++ version used Philippe Vannerem

  15. SCADA: controlling Gaudi apps • PVSS control panel for EFF. • ControlMgr runs control script • Subfarm functionality: • Distributed control of its nodes • Farmnode functionality: • Connect to DimController, which steers ApplicationMgr with commands (e.g. configure, start, pause,stop) • Mapping of finite state machine in GaudiDimController in PVSS FSM control object • Display tree of the configured sequence of Algorithms by talking to DimPropertySvr • Get/Set Properties of Algorithms Philippe Vannerem

  16. SCADA: monitoring Gaudi apps • PVSS monitor panel for EFF. • ControlMgr connects to DimH1D histos. • Subfarm functionality: • Add histograms from node together in one subfarm histogram. • Farmnode functionality: • PVSS control manager “pulls” each published value (float, int, string, histogram) at fixed time intervals. • Panel displays monitored values • Panel displays PVSS trend of monitored value Philippe Vannerem

  17. Communicating histograms on-line GAUDI app SCADA app • Generalized concept of updated on-line histograms with DIM. • Uses the AIDA interface classes for histograms (GAUDI-independent). • Tested with ROOT histogram viewer. • ROOT subscribes as a client to a DimH1D histogram, served by a Gaudi application, PVSS,… ROOT app Philippe Vannerem

  18. Conclusions • The LHCb design philosophy of a fully integrated hierarchical Experiment Control System is applied both to control hardware and software devices. • GAUDI trigger algorithms running on the event filter farm will be controlled and monitored in a distributed way with a SCADA system. • The interfacing package GAUCHO facilitates steering and monitoring of trigger algorithms for supervision and data quality checking purposes in an on-line environment.  philippe.vannerem@cern.ch Philippe Vannerem

More Related