Messaging Toolkit InfoSphere Streams Version 3.0

1. Anjali Agarwal Toolkit Messaging ToolkitInfoSphere Streams Version 3.0

2. Important Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR • ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. The information on the new product is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information on the new product is for informational purposes only and may not be incorporated into any contract. The information on the new product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any features or functionality described for our products remains at our sole discretion. THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

3. Agenda • Introduction to Messaging Toolkit • Use cases -Why use Messaging Toolkit • Operators • Terminologies and Concepts • Software Dependency • Enhancements/Features common to XMSSource and XMSSink • Enhancements/Features for XMSSink • Enhancements/Features for XMSSource

4. Introduction to Messaging Toolkit

5. What is Messaging Toolkit • A set of edge adapters which help to read/write to/from a queue or topic in WebSphere Message Queue • XMSSource • XMSSink • Uses the JMS api's exposed by the IBM XMS client in C++ language

6. Brief Architecture Diagram WebSphere MQ Queue/Topic Create connection XMS Source Create Producer/ Consumer Create connection XMS Sink XMS Streams Uses MQ Client Read Message Write Message

7. Use Cases

8. Real time campaign management • Let's take the case of a Large Bank which has lots of credit card customers. Imagine if someone performs a credit card purchase in a classy mall. We know where the transaction has been committed and the bank can offer various cross sell and up sell products. The transaction committed is captured by Streams and then Streams processes this and posts the message on to the Messaging queue layer. Unica further connects to the messaging queue and instantiates the campaign. This example illustrates how organisations are utilizing SOA technologies to integrate their enterprise systems and how messaging toolkit adapters fit into the eco- system seamlessly to respond in real time and increase efficiency and timeliness of the processes.

9. Tracking stolen vehicles • DMV issues licences to lots of vehicles. Electronic Toll Collection(ETC) allows for electronic collection of tolls by identifying cars and electronically debiting the account of registered car owners. Information from ETC can be used for smarter crime detection and prevention. • Tracking stolen cars: Assuming all toll information is continuously sent and filtered against the stolen car list by Streams, when the stolen car passes a toll, it can be flagged and sent to BPM systems. • Tracking stolen devices: When a device theft is reported to enforcers, an entry is made to systems containing details about stolen devices. When a car containing the stolen device passes a toll, it can be flagged and sent to BPM systems. • Further we can use GeoSpatial (GIS) toolkit to find out the tentative location of the vehicle and based on the speed and the route which that vehicle took we can help in tracking it. Further this information can also be plotted in a map, and displayed to interested parties.

10. Operators

11. XMSSource • This operator reads data from a WebSphere MQ queue or a topic and pushes the data into Streams. The main functionality includes: • Zero input ports, One or Two output ports • Validates if the Streams application can connect to WebSphere MQ during runtime • Validates that the queue/topic name specified in the initial context element exists during runtime • Validates that all the Stream tuple elements which do not have an explicit values assigned to them are present in the native schema and the datatype of these elements/attributes match during compile time • Validates if the message class specified in the native schema is a valid MQ message class during compile time • For Initial/ Transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user • If an error occurs while reading the input from queue/topic, writes the error string into the error output port (if one is specified) • On successful read, submits the tuple to the output stream corresponding to the message on MQ

12. XMSSink • Publishes data from Streams to a WebSphere MQ queue or a topic. The main functionality of this operator includes: • One input port, One output port(optional) • Validates if the Streams application can connect to WebSphere MQ during runtime • Validates that the queue/topic name specified in the initial context element exists during runtime • Validates that all the attributes in the native schema are present in the input stream tuple and the datatype of these elements/attributes match during compile time • Validates if the message class specified in the native schema is a valid MQ message class during compile time • For initial/transient connection failures, the operator tries to reconnect based on the reconnection policy specified by the user • If an error occurs while writing to a queue/topic, writes the error string and the erroneous input tuple into the error output port (if one is specified) • On successful write submits the tuple in the input stream to the corresponding message on MQ

13. Terminologies and concepts

14. Connection Specification Document • xml file which specifies the details of connection to WebSphere MQ

15. Message class and native schema • Message class specifies how a Stream tuple is to be serialized into a MQ message. There are 7 types of message classes: • map : A message containing a collection of name/type/value triplets • stream : A message containing ordered list of type/value pairs • empty : An empty message • xml : A message containing a generic XML message • bytes :A Message containing a stream of binary data • wbe : A message contains an XML document in the WebSphere Business Events event packet format • wbe22 : A message contains an XML document in the old WebSphere Business Events event packet format 2.2 • Native schema: Specifies the data attributes (their name, type(external type) and length(optional)) for a message which is read from /written to a queue/topic in WebSphere MQ

16. Mapping between message class, native schema

17. Software Dependancies

18. Software Dependancies • Operating System: All Operating System supported by Streams except Power • IBM Message Service client for C/C++ (also known as XMS client) version 2.0.1 is required • The XMS client must be installed on the same machine as Streams is installed and an environment variable (XMS_HOME) needs to be set to point to the install location of XMS Client • The WebSphere MQ Server and WebSphere MQ Client need to be at: • WebSphere MQ 7.0 + All Fix Pack 7.0.0.1 or later • WebSphere MQ 6.0 + Refresh Pack 6.0.1.1 or later • WebSphere MQ 7.1 is not supported by the current version of XMS : 2.0.1 • The WebSphere MQ client libraries must be on the same machine as Streams and an environment variable (MQ_HOME) needs to be set to point to the MQ Client libraries.

19. Features common to both Operators

20. Blob support for multiple message classes • The SPADE version supported sending payloads in the message • Blob support introduced for message class map, stream and bytes in both operators • Blob support introduced for message class xml in XMSSink as an enhancement <access_specifications> <access_specification name="OutTrading1"> <destination identifier="Dest" delivery_mode="persistent" message_class="map" /> <uses_connection connection="Conn1"/> <native_schema> <attribute name="l" type="Bytes"/> </native_schema> </access_specification> </access_specifications> composite XMSSinkMain { graph stream<blob l> BlobData = FileSource(){ param file : "Image.png"; format : block; blockSize : 1276431234u; } () as Sink1 = XMSSink(BlobData){ param connection : "Conn1"; access : "OutTrading1"; } }

21. Support for Reconnection Policies • The default reconnectionPolicy in case of initial/transient connection failure was to try the connection infinitely • Support was introduced to allow the user to specify the reconnectionPolicy parameter with valid values of NoRetry, InfiniteRetry and BoundedRetry • Additional parameters reconnectionBound and period were introduced to give more flexibility to the user • Default is BoundedRetry with reconnectionBound of 5, period of 60 sec composite XMSSourceMain { graph (stream<blob l>InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : NoRetry; } } composite XMSSourceMain { graph (stream<blob l>InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : BoundedRetry; reconnectionBound : 10u; period : 1.0; } } composite XMSSourceMain { graph (stream<blob l>InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; reconnectionPolicy : InfiniteRetry; period : 1.0; } }

22. Support for an optional error output port • Support was introduced in both the operators to write error string on to the error output port in case of read/write failures • Optional error output port is non-mutating and its punctuation mode is free composite XMSSourceMain { graph (stream<blob b>InputData; stream<rstring message> Error) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; } () as Sink1 = FileSink(InputData){ param file : "XMSSourceOutput.txt"; flush : 1u; } () as Sink2 = FileSink(Error){ param file : "XMSSourceError.txt"; flush : 1u; } }

23. Other Enhancements • Supporting external types in native schema • Previously SPL types were specified in the native schema • Changes to accommodate specifying external types in native schema for ease of use • Introduction of environment variables to point to MQ and XMS install locations • The XMS install location was hard coded to /opt/IBM/XMS and the MQ install location hard coded to /opt/mqm • Environment variables XMS_HOME and MQ_HOME introduced for allowing flexibility • Support for uint8, uint16 and uint32 data types. • Support initially existent only for signed integers • Support for uintb data types introduced

24. Enhancements/Features for XMSSink

25. Negative length support on message class bytes • Message class bytes allows specification of -2,-4, and -8 as valid values in the native schema for string and blob • Negative length indicates that a 2,4 or 8 byte length needs to be pre-pended with the data representing the length of the data <access_specifications> <access_specification name="OutTrading1"> <destination identifier="Dest" delivery_mode="persistent" message_class="bytes" /> <uses_connection connection="Conn1"/> <native_schema> <attribute name="k" type="String" length=”-4”/> </native_schema> </access_specification> </access_specifications> composite XMSSinkMain { graph stream<rstring k>InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; }

26. Truncation and padding support for string • When a positive length is specified in the native schema for string in message classes map, stream and xml, it is treated as an upper bound for that string • For message class bytes, the string is truncated/padded in accordance with the length in the native schema composite XMSSinkMain { graph stream<rstring k>InputData = Beacon(){ param iterations:1u; output InputData :k="ThisIsALongString"; } () as Sink1 = XMSSink(InputData){ param connection : "Conn1"; access : "OutTrading1"; } <access_specifications> <access_specification name="OutTrading1"> <destination identifier="Dest" delivery_mode="persistent" message_class="stream"/> <uses_connection connection="Conn1"/> <native_schema> <attribute name="k" type="String" length=”7”/> </native_schema> </access_specification> </access_specifications>

27. Performance Metrics Support • XMSSink has the following metrices • nTruncatedInserts: The number of tuples that had truncated attributes while converting to a message • nFailedInserts: The number of failed inserts to the WebSphere MQ • nReconnectionAttempts: The number of reconnection attempts made before a successful connection

28. Enhancements/Features for XMSSource

29. Support for initDelay parameter • Support for initDelay parameter introduced in XMSSource composite XMSSourceMain { graph (stream<blob l>InputData) = XMSSource(){ param connection : "Conn1"; access : "InTrading1"; initDelay : 10.0; } () as Sink1 = FileSink(InputData){ param file : "XMSSourceOutput.txt"; flush : 1u; }

30. Performance Metrics Support • No performance metrics was existent in the SPADE version • XMSource supports the following performance metrics • nMessagesRead: The number of messages read successfully • nMessagesDropped: The number of messages dropped • nReconnectionAttempts: The number of reconnection attempts made before a successful connection