IT Application Monitoring Using PI AF and Notifications

1. IT Application Monitoring Using PI AF and Notifications Josiah Thoen - AEP

2. AEP – Who We Are – By Assets • One of the largest U.S. electricity generators ( ~ 38,000 MWs) with a significant cost advantage in many of our market areas • Largest consumer of coal in the Western Hemisphere • Operations within four RTO’s • PJM • SPP • ERCOT • MISO • A leading consumer of natural gas • Major wind power developer(#3 in U.S. in 2005) • 39,000 miles of transmission • 186,000 miles of distribution • Coal transportation • 7,000 railcars • 2,230 barges and 53 towboats • 5 million customers

3. AEP IT - Service Performance • Monitoring AEP IT Assets for over 7 years • Servers • Hardware, Operating Systems (Windows and Unix) • Middleware Platforms • IIS, BEA WebLogic, etc. • Applications • In house custom applications • Vendor Applications • BlackBerry Server, Lotus Notes, etc.

4. PI Server Collective 2 Servers 8 CPU, 8gb RAM, 85gb SAN Drive Space Software: PI Server AF & Notification MS SQL Express ACE (Backup) Data Acquisition Node 2 CPU, 2gb RAM, 30gb Local Drive Space Software: IT Monitor Interfaces Custom PISDK monitoring services Analytics Server 2 CPU, 2gb RAM, 30gb Local Drive Space Software: ACE (Primary) Stats: Tags: 94,049 Modules: 54,076 AF Elements: ~10,000 AF Element Templates: ~25 Notifications: 331 Data Storage: 1gb every 23days AEP IT - PI Infrastructure

5. AEP IT and PI – the beginning • PI IT Monitor Pilot • Customer • IT Application Support Teams – Primary • IT Operations – Secondary • Business Units - Secondary • Purpose • Production Server and Application visibility • Result • Infrastructure monitoring approach • Application Monitoring

6. AEP IT and PI - Problems • Time Series data is not relational • Convoluted naming conventions and overloaded tag attributes • Network > Datacenter > Server > Components > Application > Middleware > Database > and back • Maintenance nightmare • Everything is manual • Numerous duplicate displays • No Event Management • All this data but no way to knowing when something is broken.

7. AEP IT and PI - Solutions • Problem: Time Series data is not relational • Solution: AF – Relational Meta Data on top of Time Series data. • Problem: Maintenance nightmare • Solution: AF – Template based management of elements and their attributes • Problem: No Event Management • Solution: PI Notifications – Template based triggered event delivery

8. AF Implementation • WebMethods BrokerClient QueueLength monitoring • SoftwareAG WebMethods • SOA, B2B messaging platform • Pub-Sub Message Brokering • Publisher sends a message to a Broker. • Broker delivers the message to one or more Subscribers of that message. • Used in many Tier 1 and 2 application services and business processes • Facilitates communication with external counter parties • Used Internally to send data between applications.

9. Business Need • Monitor the Broker to make sure that client queues are not getting full. • Publishing clients send messages to the Broker • Broker delivers the message to subscribing clients queue. • Subscribing clients retrieve messages from their queue. • Implications of failure: • Client may be unavailable. • Broker may be having problems.

10. Functional Requirements • Store the Broker Client Info data. • Alerting: • Warn when queue contains 25+ messages. • Alarm when queue contains 75+ messages. • Alerts must send a Clear message when the alerting condition is gone. • Alerting Recipient: • Email notification to those interested. • AEP IT Command Center: • 24x7 Support Center. • All Events go to Central Event Management System.

11. Monitoring Process • Data Acquisition – PISDK Custom Monitor • Invoke a service within the WebMethods environment every 60 seconds. • Parse data. • Do exception testing • Send data to PI. • Automatically Add new Broker Clients to AF as they are discovered. • Automatically create tags for Broker Clients if they do not exist.

12. AF Element Templates • BrokerClientInfo: • Defines the attributes of a BrokerClient as it exists in WebMethods. • Each Attribute references a PI Point. • Appname = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\Element%_%Element%_%Attribute%_Info • QueueLength = \\%Server%\BrokerClientInfo_%..\..\Element%_%..\Element%_%Element%_%Attribute%_Count

13. BrokerClientInfo Template

14. BrokerClientInfo Element

15. AF Element Structure • WebMethods • Brokers • <BrokerServer> • <Broker> • <BrokerClient> • <BrokerClient> • <Broker> • <BrokerClient> • <BrokerClient> • <BrokerServer> • <Broker> • <BrokerClient> • <BrokerClient>

16. AF Element Structure

17. Notification Process - Requirements • Notification is based on the QueueLength attribute of a BrokerClientInfo. • Needs to be customizable for each client. • Needs to be disabled for certain clients. • Default Warning = 25. • Default Alarm = 75.

18. Notification Process - Template • Attribute Template:

19. IF Notification enabled And Warn enabled And QueueLength >= Warn value And QueueLength < Alarm value Notification Process – Warn Trigger

20. IF Notification enabled And Alarm enabled And QueueLength >= Alarm value Notification Process – Alarm Trigger

21. Delivery Channel • WebService • Sends Target and Trigger value to our EventManagement Web Service. • Target = AFElement Path • Value = CRITICAL, WARN, OK • Event Management Web Service • Gets additional details from AF using the Target. • Creates an Event Management message and sends it to our Event Management server. • Closes the previous event using the EventId attribute from the AF element. • Sends new EventId back to AF Targets EventId attribute.

22. Event Management View • BMC Impact Explorer • Transpara