Data integration in real time ( pvmanager , graphene and all of that)

1. Data integration in real time(pvmanager, graphene and all of that) Gabriele Carcassi

2. The problem • Different domains (v3, v4, DISCS, CS-Studio, …) produce and consume data • How do we integrate that data? • How does CS-Studio gets CA and PVA data? • How do we integrate archive data with live data? • How do web services get CA data? • How does v4 get web service data? • How does MATLAB gets PVA and web service data? • How does CS-Studio gets results from a script? • ….

3. Objective of the talk • Discuss strategically how to reshape pvmanager/epics-util/graphene to achieve that goal • Discuss tactically what our priorities are • Efficient use of the resources we have • Please, interrupt!

4. High level view

5. Caching orqueuing Rate throttling: <= request <= the client can handle Pluggabledatasources Desired rate Source rate Client

6. Formulalanguage Caching orqueuing Rate throttling: <= request <= the client can handle Pluggabledatasources Desired rate Source rate Graphene Client

7. Formulalanguage Caching orqueuing Rate throttling: <= request <= the client can handle Pluggabledatasources Desired rate Source rate Graphene Client Datasources(publish/subscribe) Services(command/response) Pluggableservices Serviceregistry

8. Status • The framework around pvmanager has been growing in functionality • Originally started as small layer to handle caching/queueing • Now includes access to services (jdbc, pva, exec, web, …), extensible data manipulation language, graphs (graphene), … • Used outisde CS-Studio (embedded in Vasu’s Proteus, Xihui’sWebPDA) • Increasing contributions • CosyLab contributing pvasupport, 3 UMich students working on graphene, ITER planning to contribute archived data integration, FRIB contributing array manipulation functions

9. Current problems • Focus dominated by CS-Studio • Other use cases always left in the backburner (e.g. rebroadcast through pva, support for other languages, MATLAB) • No active effort into an overall strategy • Where does one ask for support? • Unclear mailing list • pvmanager and graphene mailing list exist but they are not used. Updates on pvmanager were posted on cs-studio mailing list. Some reports are in CS-Studio github issues. Support for everything is private e-mail. • Not well thought out process of contribution • Contributors often have to redo the CS-Studio integration themselves just to test their changes • Split into 3 repositories causes problems • Trying to step back and reassess

10. Possible way forward • Time to start to focus on this layer on its own merits! • Focus more equally on CS-Studio, v4, DISCS, web access and other uses; facilitate independent integration and contributions • Create a Data Integration In Real Time (diirt) top project • pvmanager is not a good name for something that does services and other data sources • Single mailing list for all pvmanager/vtype/epics-util/graphene development and support • Single github issue tracker, will move all repositories to github • More transparent release planning for diirt • Gather requirements from CS-Studio, v4 and DISCS more equally. A contact from each? Eric/Greg/Vasu • Increase effectiveness of contributions • Branch of CS-Studio with development version • Contributor can push changes in pvmanager repository, and get CS-Studio update for free • Independent framework for load/integration/normalization tests • Contributor focuses his testing only up to diirt integration

11. Links to github Links to subprojects Summary ofcurrent issues Build status

12. Automaticallycommittedfrom mainpvmanagerrepo Download tolatest dev build

13. For example diirtautomatic integration Updates pva services Greg Matej Get cs-studio devversion and test Adds a formula function Kunal Eric

14. Future goal pva gateway dev diirtautomatic integration web gateway dev cs-studio dev Matej, Kunal,UMICH students, ITER, … Contribute to the data integration framework: automatically contribute to a varied set of applications!

15. Testing

16. Integration test framework • We need a way to run integration tests • test IOCs -> CAJ -> pvmanager • Including disconnects due to power cycle and network downtime • Corner cases (e.g. different type at reconnect) • Ability to check server state (e.g. number of monitors open) • Ability to drop in and run the tests in production environment (to check specific versions of EPICS and network configurations) • Start a script on the server side, start a script on the client side, come back in 15 minutes

17. public final void run() throws Exception { // Start first IOC init("typeChange1"); // Connect pv addReader(PVManager.read(channel("double-to-i32")), TimeDuration.ofHertz(50)); pause(1000); // Stop current IOC and start the second restart("typeChange2"); pause(2000); } public final void verify(Log log) { // Expect 3 connection notifications: connect/disconnect/connect log.matchConnections("double-to-i32", true, false, true); // Expect 3 value notifications. Match value/alarm, but not the timestamp log.matchValues("double-to-i32", ALL_EXCEPT_TIME, newVDouble(0.0, newAlarm(AlarmSeverity.INVALID, "UDF_ALARM"), newTime(Timestamp.of(631152000, 0), null, false), displayNone()), newVDouble(0.0, newAlarm(AlarmSeverity.UNDEFINED, "Disconnected"), newTime(Timestamp.of(631152000, 0), null, false), displayNone()), newVInt(0, newAlarm(AlarmSeverity.INVALID, "UDF_ALARM"), newTime(Timestamp.of(631152000, 0), null, false), displayNone())); }

18. Integration test framework • Covered • Simple reboot: connect pv, ioc down, ioc up, only 1 monitor open • Simple network outage: connect, network down, network up, only 1 monitor open • Multiple reboots: connect pv, ioc cycle 10 times • Type change: connect double pv, ioc cycle, pv become integer • Constant pv: conect to double/int/string/enum that do not change • Slow changing pv: conect to double updating at 1 Hz (same rate received) • Fast changing pv: conect to double updating at 100 Hz (reduced rate received) • Alarm changing pv: conect to double updating at 1 Hz for alarm only • Write pv: change value for double/int • Not yet covered • Add all remaining types for disconnection test • Add all types for type change • Add all types for slow changing pvs • Add all types for fast changing pvs • Add all types for alarm changing pvs • Add all types for write pvs • Add metadata changes • Add access control changes • Add multiple reader on a single pv (only 1 monitor open) • Add nanosec out of range for time • Old RTYP handling

19. Integration test framework • Martin Konrad took what I had and modified for his own purposes • Testing archiver, ca gateway, … • We diverged a little bit: need to re-align • Andrej Babic is tasked to improve testing for CS-Studio • My hope is that these efforts can be aligned into one • FLINT: framework for load/integration/normalization tests • Load: large number of pvs, large data size • Integration: different IOC versions, different OS, different network configurations • Normalization: data comes out looking the same, predictable event count and order • Work with multiple protocols (pva and ca at least)

20. Integration test framework • Still not enough: testing on my machine does not guarantee anything • We have found that performance profile changes significantly from machine to machine (even with same OS) • Active scanning does not produce noticeable load on my machine on any os (Win/SL/Debian) while it does on other • We have found that some Java network libraries behave not up to spec on different machine • Currently, I have to disable my wireless every time I run tests with ca or pva • We need: • Set of automated tools that run tests and gather performance data • Set of people that actually run those tests pre-emptively on their HW/OS combination

21. Overall architecture

22. Data integration in real time (Diirt) architecture Client tools Servers CS-Studio Python MATLAB pvagateway web gateway graphene Visualization Data serialization/conversion Historic data formula Processing NTTypes png/jpg archiver support vTypes Data Definition csv HDF5 … arch-app ch-arch pvmanager core Aggregation Producer/Consumer Command/response datasource support services support ca pva file http … jdbc pva exec cf …

23. Diirt consumers

24. CS-Studio • pvmanager now serves data to most applications in CS-Studio • Allows formulas in BOY • Allows access to services in BOY • Set of common features for pvmanager configuration in eclipse product • Graphene widget continued improvements

25. CS-Studio • Areas of possible improvements in CS-Studio • Better integration in BOY (e.g. allow color widget to be a formula) • Take advantage of pause/resume • Single point to configure color/fonts/time format/… for all applications • Areas of possible improvements in diirt • Performance optimization (passive scanning) • Improve graphene widgets

26. python • pvmanager and other diirt components are currently implemented in Java and there is a desire to access the functionality in python • Different options: • Jython • Re-implement the whole stack in Python • Re-implement the whole stack in C++ • Implement a thin library in python that uses an embedded JVM • Others?

27. python – Jython • No starter • No access to all the library that make people want to use python in the first place

28. python – python implementation • Benefits • Have all the functionality in pure python • Portability across python platforms • Costs/challenges • Lack of well-spec’d memory and threading model? • Keeping Java and python implementations in line • Solves python, does not solve anything else

29. python – C++ implementation • Benefits • Have all the functionality in C++, which makes it possible to support other languages (such as python) • Costs/challenges • Threading and memory support OS dependent? C++11 may solve this • Probably can’t rely on immutable objects as in Java implementation • Significantly harder to write and support: very few can do this • Keeping Java and C++ implementations in line

30. python – embedded JVM • Benefits • Leverages all the work already done • Versions keep in synch • Probably the cheapest • Costs/challenges • Is it actually doable, with reasonable performance? Different techniques to investigate (JNI, Py4J, javacpp, …) • Can we keep the interface reasonable? Maybe we can start with this, and re-implement later? • Contributions to the core are limited to Java (but maybe this is not so bad) Python apps Py4j diirt CA PVA

31. MATLAB • I’d be interested to at least write a simple example of use in Matlab • At least to know if it’s possible and how easy it is • Is this something v4 or others are interested in? • My current showstoppers: • I don’t have MATLAB and it costs few k$ • I don’t really know how to use MATLAB • I don’t really know what to do with it (need use cases) MATLAB diirt CA PVA

32. Web gateway using web sockets • Desire to have web tools on top of CA • Web sockets now allow updates from servers • Couple of projects are already starting to use pvmanager within a web server • Should have general purpose web gateway, powered by diirt and websockets? • Standardize JSON serialization of vTypes? XML (InfoSet, Fast InfoSet, EXI)? • Maybe a thin javascript client library as well? Re-implement WebCA? Binding for widgets (d3js)? • Should this support services as well? Provides access of pva services to web clients Web apps Mobile apps Web server diirt CA PVA

33. Pva gateway on top of pvmanager • We have talked about this forever! • Create a pva server that re-broadcasts what gets from pvmanager • Protocol translation (ca -> pva) • Data aggregation and processing • Could also work for services (web services -> pva) • ChannelFinder could be a good candidate pva clients pva services pva server diirt CA Web services

34. Diirt producers and integration

35. Historic data • Desire to integrate archived data and live data to provide a single client resident cache for real-time processing • Graphs in time • Computation (histrograms, correlation, fits) • Data export (to csv, HDF5, …) • ITER is looking into contributing in this space pva gateway CS-Studio diirt CA Archived data

36. Functions • Better support for disconnect/alarm/time • Adding element-wise array operations • arraySum(VNumberArray, Vnumber) • arrayDiv(VNumberArray, VNumberArray) • Adding experimental DFT

37. Support for nd-arrays • VNumberArray is already n-dimensional • Was extended to include cell boundaries • Boundaries usedby histogram plot • Formula functionthat constructs thefull object from ahistogram record

38. Support for nd-arrays • Students working on intensity graph • Hopefully will be readyby end of May • Support fornon-homogenouscells as well

39. v4 integration • Cosylab has been contributing pva support for • Datasources (already integrated) • Services (in late-March release) • Other things we already stated we need • Extract the VTypes <-> NTTypes conversion/wrapping library

40. v4 service example <?xml version="1.0" encoding="UTF-8"?> <pvAccessRPCServicever="1" name="pvRPCServiceSample" description="A test pvRPC service"> <channelName>testChannel</channelName> <methods> <method name="sum" description="sum the two numbers"> <structureid>uri:ev4:nt/2012/pwd:NTXY</structureid> <argument name="a" description="first number" type="VDouble"/> <argument name="b" description="second number" type="VDouble"/> <result name="c" description="sum result" type="VDouble"/> </method> <method name="concate" description="concate two strings"> <argument name="a" description="first string" type="VString"/> <argument name="b" description="second string" type="VString"/> <result name="c" description="concate result" type="VString"/> </method> <method name="getTable" description="get table as a standalone result"> <result standalone="true" description="table result" type="VTable"/> </method> <method name="getImage" description="get image as a standalone result"> <result standalone="true" description="image result" type="VImage"/> </method> <method name="multipyString" description="multiply the string"> <argument name="string" description="string to multiply" type="VString"/> <argument name="times" description="how many times to multiply" type="VInt"/> <result name="c" description="multiply result" type="VString"/> </method> <method name="getTimeInMiliseconds" description="get time in milliseconds"> <result name="c" description="time result" type="VDouble"/> </method> <method name="nop" description="method without arguments and no result"> </method> </methods> </pvAccessRPCService>

41. v4 service example <?xml version="1.0" encoding="UTF-8"?> <pvAccessRPCServicever="1" name="pvRPCServiceSample" description="A test pvRPC service"> <channelName>testChannel</channelName> <methods> <method name="sum" description="sum the two numbers"> <structureid>uri:ev4:nt/2012/pwd:NTXY</structureid> <argument name="a" description="first number" type="VDouble"/> <argument name="b" description="second number" type="VDouble"/> <result name="c" description="sum result" type="VDouble"/> </method> <method name="concate" description="concate two strings"> <argument name="a" description="first string" type="VString"/> <argument name="b" description="second string" type="VString"/> <result name="c" description="concate result" type="VString"/> </method> <method name="getTable" description="get table as a standalone result"> <result standalone="true" description="table result" type="VTable"/> </method> <method name="getImage" description="get image as a standalone result"> <result standalone="true" description="image result" type="VImage"/> </method> <method name="multipyString" description="multiply the string"> <argument name="string" description="string to multiply" type="VString"/> <argument name="times" description="how many times to multiply" type="VInt"/> <result name="c" description="multiply result" type="VString"/> </method> <method name="getTimeInMiliseconds" description="get time in milliseconds"> <result name="c" description="time result" type="VDouble"/> </method> <method name="nop" description="method without arguments and no result"> </method> </methods> </pvAccessRPCService> Best part:I have no idea how itworks! Cosylabsupports it andCS-Studio picks it up.

42. DISCS integration • Proteus is already embedding pvmanager for ca data access • Could benefit from standard web gateway • We should more actively try DISCS service integration through diirt (and CS-Studio) • Maybe standard web service bindings to JSON, so we can (like JDBC and PVA) only require to write an XML file?

43. diirt service interface public abstract class ServiceMethod { public String getName(); public String getDescription(); public Map<String, Class<?>> getArgumentTypes(); public Map<String, String> getArgumentDescriptions(); public final Map<String, Class<?>> getResultTypes(); public Map<String, String> getResultDescriptions(); public abstract void executeMethod( Map<String, Object> arguments, WriteFunction<Map<String, Object>> callback, WriteFunction<Exception> errorCallback); }

44. Goals for diirt services • Not general purpose! Some services in some cases • Have parameters that can be expressed by vTypes (e.g. VString and VNumbers) • Can return data that can be expressed in a vTable, vNumericArray, … • These can be digested by general purpose clients • For example, channel finder • Will need a specific UI to manage tags and properties • Can export into a table the result of a query • For the general purpose only we use diirt services

45. Scope

46. Where to draw the line? • What do I use for creating a Java UI client? diirt! • What do I use to access services? diirt! • What do I use to do protocol conversion? diirt! • What should I use to toast my bread? diirt! • What should I give my spouse for Christmas? diirt!

47. Where to draw the line? • All data can be accessed in diirt != all data access should use diirt • I am fully committed to the first, I am really skeptical about the second • I don’t know all possible use cases • Software design is about tradeoffs • Waste of resources

48. Where to draw the line? • For services, I think it’s already clear: only calls that can be used by “general purpose clients” • Is the client specific to your service? (e.g. log entry for logbook, property management for channelfinder) • Are VTypes unreasonable representations for request and response? • Is the data useful only in the context of its service? • Services integrated in diirt should be the partial “view” that is “general purpose”

49. Where to draw the line? • For datasources (CA and PVA specifically) • Some clients that will never use diirt because they need direct access to the protocol API: • Debugging tools • IOC • Some clients should use something like diirt because they are just going to re-implement it: • UI tools • General purpose clients • Web socket gateway • Some things are in between, and I think it depends on different trade-offs • PVA gateway • Do you want to faithfully forward the precise semantic of the pva protocol? • Do you want make available data from other sources and do aggregation?

50. Where to draw the line? • There is a problem of architecture decency: • An interface can’t be both general purpose and give access to everything protocol specific • If you expose all the options of all the possible protocol implementations… • Current: ca:// pva:// file:// • Formulas - e.g. =`pvArray1` + columnOf(‘pvTable’, “column2”) • Planned: http:// jmx:// • Talked about: jms:// snmp:// • I think we just end up with an unusable mess • Each new feature needs to be somehow understood by all clients and all datasource implementation