1 / 47

Metadata Driven Data Services for SOA

Metadata Driven Data Services for SOA. Paul Anderson Technical Sales Director 30 October 2014. Agenda. SOA Review Data Integration Data Services Use Cases Wrap-Up. Overview.

amity-meadows
Download Presentation

Metadata Driven Data Services for SOA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metadata Driven Data Services for SOA Paul Anderson Technical Sales Director30 October 2014

  2. Agenda • SOA Review • Data Integration • Data Services • Use Cases • Wrap-Up

  3. Overview Data architecture and its management are evolving toward an data services management system that delivers information as a service.

  4. SOA Timeline • Personal involvement since 1998 • “Candle's Roma product of 1998 is the ESB's most direct ancestor" (Roy Schulte, Gartner) • SOA concepts have existed for a long time • Business needs did not drive SOA adoption • Fragmented technology slowed adoption • Gained widespread acceptance in last 5 years • Web Service Technology has enabled broad adoption of the concept

  5. What is SOA? • Gartner • "Web services are about technology specifications, whereas SOA is a software design principle.” (Yefim V. Natis) • SOA Concepts • Decoupling of service production from consumption • Service Interchange • Service Discovery • SOA is an architectural pattern • The architectural pattern has/does appeared in many guises • ESB tools represents the most widely accepted toolset for SOA implementation • Just because you use an ESB does not mean you have an SOA architecture • SOA benefits from the platform neutrality of Web Services • Enables interchange of services • Requires decoupling • Provides discovery • The changing business environment is driving SOA • The scope of business functionality is changing

  6. SOA Business Drivers • Business Process enhancements • Migration of customer management to relationship management • Risk Management across/between organizations • Order/Shipment tracking across/between organizations • Tighter Partner Management • Cross-Sell / Up-sell • Based on history/demographics/relationship/value • Business Agility • Services are re-usable building blocks • Cataloging of available services • Enhanced time to market • Focus on information rather than data • Integrated Businesses • Transactions span multiple organizations

  7. Characteristics of SOA • Requires Horizontal integration • Scope crosses lines of business boundaries • Scope crosses corporate boundaries • Requires Horizontal understanding • Requires understanding of multiple lines of business • Requires development of common vocabulary • Require concept generalization • Exploitation of commonality • Drilldown into specifics • Requires easy access to information • Need to remove the complexity of differing semantics, location, and access methods

  8. SOA often slowed by • Vertical Fragmentation • Differing technologies across verticals • Differing semantics across verticals • Infrastructure Fragmentation • Application fragmentation • Hardware fragmentation • Data fragmentation • LOB focus • Multiple technologies • Replicated data • Organizational fragmentation • Ownership Issues • Organizational issues • Governance issues

  9. Common Misconceptions #1 SOA is an technology issue: • Web Services are a suite technology specifications • You can implement a web service based stove pipe • SOA does not have to be web service based • Looks like an arcitectural diagram but is just a list of standards

  10. Common Misconceptions #2 SOA is an application issue: • Focuses on services but ignores the underlying data • Characteristics of an application layer are very different from the characteristics of a data layer • Focuses on reuse of functionality and ignores reuse of data

  11. Common Misconceptions #3 An ESB tool is all you need:

  12. Common Misconceptions #3 (Continued) • Fragile implementation • Data access is embodied in the business process • Adding an additional data source requires business process change • The data services are source specific • Poor performance • Data reduction is likely happening in the Business Process layer • XML well suited for business process layer but not as well suited to the data services layer • Unmanageable • Difficult to audit data access • Difficult to secure data access

  13. SOA reality • SOA is fueled by data • SOA combines People, Machines • SOA embodies Business Process • SOA is based on data • SOA is fueled by metadata • You can only reuse what you know exists • You can only reuse what you understand • SOA based applications need to co-exist with existing applications

  14. Data Integration Implementation Evolves Toward Data Services – Gartners View Data Data Data Users Applications Processes Business Services Get customer Close account Calc lifetime value Archive history Get single view of product Request for data operations:(Type, Format, Latency, Quality) Response: (Data or Metadata) Data Services Metadata Sync Aggregate Transform Access Profile Move

  15. Agenda • SOA Review • Data Integration • Data Services • Use Cases • Wrap-Up

  16. Enterprise Information Problem customers sales supplier data financial billing accounts shipping customers The current state of corporate data Disparate Information Silos ? Data Services Enterprise Information Consumers Enterprise Information Sources

  17. Data Integration Challenges • Organizations cannot address basic questions • What information do I have? • Where is it? • How do I access the most current information? • How do I manage it? • What is the impact of change during and after implementation of my system? • Information is stored in myriad data stores across the Enterprise. • Different formats/data types • Different structures/semantics • Different access methods (API’s)

  18. Data Integration Landscape EAI CRM ERP Reporting / Dashboard DW / ODS EAI Web Services BI BAM Apps DS DW / ODS EAI CRM ERP ETL • EAI - Target is the application • Even Driven data movement between apps • Data movement pre-wired • Message based. • Requires application coordination / workflow. • Requires complex translation / programming. • ETL - Target is the database • Scheduled extraction of data • Bulk loading of data warehouses. • Batch driven, Large Volumes. • Inflexible, unable to meet business agility needs • Historical data analysis. • One-way data movement, read-only access. CRM ERP ETL • Data Services - Target is the end-user • On-demand delivery of information • Real-time integration of disparate data. • Universal data access layer. • Integrates multiple data sources. • Push / Pull any data across the enterprise. • Single View of Customer. • Supports dynamic / evolving reporting. • Supports / Enables SOA / Web Services.

  19. Integration Technologies Integration Style Process Data DataServices Real Time EAI Data Integration Timeliness ETL Batch

  20. What if………. <sale/> <value/> </ sale > Reporting, Analytics Web Services,Business Processes CustomApplications Packaged Applications EAI, ESB, BPM Enterprise Data Services Virtual Data Client sees single database containing enterprise data Clients are freed from the specifics of underlying sources Each client has their own desired “view” Clients issue queries against their “view”

  21. Enterprise Data Services DW DBMS FILES XML APPS ESB/ Portals Applications / Dashboards Reporting / Analytics • High Performance Infrastructure • Universal data access layer. • Integration of structured/unstructured data. • Data access security enforcement. • Insulates data consumers from sources. • Real-time integration of disparate data. • Federation of operational & historical data. • Provides a virtual representation of data. • Extract & Combine information on-demand. • Supports dynamic reporting / dashboards. • Read & Write to any data source. Web Service/ Relational Relational Web Service

  22. Model-driven Data services Enterprise Information EAI, Data warehouses Data Server geo-spatial real-time discover, access, update real-time services Packaged Apps access & update Virtual Data Server warehouses Web Services,Business Processes package, deploy, version databases MetadataCatalog spreadsheets Custom Apps model, describe, relate share, version, manage, discover xml model, describe, relate <sale/> <value/> </ sale > rich media Reporting, Analytics <sale/> <value/> </ sale > … Modeler Understand, relate, harmonize, rationalize, and use

  23. Agenda • SOA Review • Data Integration • Data Services • Use Cases • Wrap-Up

  24. SOA Challenges • Majority of data is held in relational sources • Relational structure optimized for LOB usage • Probably do not want the existing data structure to drive data service structures • Performance concerns may rule out XML federation • Granularity Issues • Granularity of data needed to support SOA may be different than current use cases • Typically there will be data reduction moving from existing data sources to data service presentation • Web-Service data is XML based • Data is organized hierarchically • Need to map from relational to Hierarchical • Scope of application typically wider that traditional applications • Spanning lines of business and/or corporate boundaries

  25. SOA Challenges (continued) • Performance • XML based data is exceptionally verbose • Exposing existing data in XML clearly not viable • Security • Broader access to data drives security concerns • Applications that span political boundaries may be subject to local confidentiality regulations • May need to integrate data security with application security infrastructure • Auditability • Regulations demand access audit trails

  26. What is a Data Service? • Decouple data sources from application • Data implementation shielded from application • Semantic/Format Mediation • Standard vocabulary • Single access point • Web Service/XML • SQL • Federation • Scalability • Security, performance SOAP/XML Bridge the Gap Data Service API Call SQL SQL Master Data SAP Divisional Operational Application

  27. Data Service Layer in SOA Data Services Layer Client Process & Applications App App App App App App Business Process Services Business Services Message Services (ESB) Data Service Data Service Data Service Data Service Data Service Data Service Data Sources

  28. Obtain XML From Non-XML Sources «Relational» «Application» «XML» <customerPositions> <accounts> <account ID=…> … </account> </accounts> </customerPositions> <<XML Doc>> GIVEN: Data Sources containing Information to integrate GIVEN: Fixed XML Schema WANT: Data complying to schema NEED: Mapping from Data to XML ?

  29. XML Modeling to achieve business agility The Solution: EIS Data XML Document <X> <X> T … </X> </X> The Problem:

  30. Data Service Design Three ‘typical’ use scenarios: • Bottom-up: expose relational-style sources in ‘table-like’ form • Business view: starting from XSD/XML-based business views • Top-down: starting from WSDL definition of Web service operations Bottom Up WSDL WSDL descriptor VDB Container Web Svc Operation Web service operation XSD IN XSD OUT XSDs - in/out <X> XML views Relational views Source models Import sources Start

  31. Data Service Design Three ‘typical’ use scenarios: • Bottom-up: expose relational-style sources in ‘table-like’ form • Business view: starting from XSD/XML-based business views • Top-down: starting from WSDL definition of Web service operations Business View WSDL WSDL descriptor VDB Container Web Svc Operation Web service operation XSD IN XSD OUT XSDs - in/out Start <X> XML views Relational views Source models Import sources

  32. Data Service Design Three ‘typical’ use scenarios: • Bottom-up: expose relational-style sources in ‘table-like’ form • Business view: starting from XSD/XML-based business views • Top-down: starting from WSDL definition of Web service operations Top Down WSDL WSDL descriptor Start VDB Container Web Svc Operation Web service operation XSD IN XSD OUT XSDs - in/out <X> XML views Relational views Source models Import sources

  33. Data Service Design Three ‘typical’ use scenarios: • Bottom-up: expose relational-style sources in ‘table-like’ form • Business view: starting from XSD/XML-based business views • Top-down: starting from WSDL definition of Web service operations Bottom Up Business View Top Down WSDL WSDL descriptor Start VDB Container Web Svc Operation Web service operation XSD IN XSD OUT XSDs - in/out Start <X> XML views Relational views Source models Import sources Start

  34. Data Services – Design Time Virtual Databases Metadata Repository DW UML DBMS XML APPS Metadata Modeler Metadata Reports • Model and Manage metadata. • Combines technical & business metadata • Enrich models with custom metadata. • Infer relationships. • Infer ownership. • Maintain consistent data dictionary • Understand overlaps in business areas • Reduce time to perform impact analysis. • Metadata open standards • MOF • XMI • Import metadata from : • Rational Rose • ERWIN • System Architect Metadata Relational View Security Runtime Metadata Design time Metadata

  35. Data Services – Run Time Virtual Databases Metadata Repository DW DBMS FILES XML APPS • Model driven Integration • GUI based modeler • Metadata Repository MetaData Modeler Console (Admin) Query Builder • Data Connectivity via • SQL-92 • JDBC • ODBC • SOAP/HTTP • SOAP/JMS • Robust security and access control • Row & column level • Supports existing authentication solutions Security/Audit • Distributed query optimization & processing • Data Caching vs. Real Time Query Processing / Optimise Engine Data Cache • Connectivity to • Databases • ERP • MOM • Data Warehouses • Apps / Legacy • Files Connector Framework

  36. Data Services benefits • On-demand information • Real time data integration • Information sharing between business units • Federation of disparate Information • Structured, unstructured • Relational + XML + Enterprise Apps + Legacy • Faster time to market • Integrated information in days, weeks • Tight coupling of design & implementation phases • Leveraging the skill-set of the data architects for integration • Costs across application lifecycle reduced • Model-driven abstraction layer between information sources and applications eases development and maintenance

  37. Agenda • SOA Review • Data Integration • Data Services • Use Cases • Wrap-Up

  38. Typical Use Cases • Financial Services • Market Reference Data • Risk Management:BIS - Basel II • Transaction Monitoring:Anti-money Laundering, Patriot Act • Supply Chain • Visibility • Reporting • Customer Service • Single View of Customer • Account Aggregation & Cross-marketing • Financial Reporting • Corporate Governance & Compliance • Sarbanes-Oxley, Executive Dashboard • Homeland Security • Watch Lists

  39. Securities Reference Data Challenge • Securities data is inconsistent • Consistent pricing & other data needed for better trades & accounting • “Hard-coding” was too expensive and inflexible Solution • Models define common “view” of securities data • Transform data to XML formats • Publish to pub/sub system • Support over 50 front-end applications ROI • Consistent data = better trades, better accounting, better risk management • New data sources online faster (weeks) • $3 million in IT savings Front-End Applications IBM MQ Series MetaMatrix Server Security Price Security Description Security Price

  40. Implementing Web Services & Service Oriented Architecture • Business Drivers • Information sources available and discoverable as location-independent Services on the CSFB Network • CSFB Lines of business have access to information when and where it is needed • Web Services Loosely Coupled IT Architecture • Technology Use Case – Universal Data Services • Web Service consumers integrated to discoverable Services • Encapsulate existing/legacy functionality • Solution Benefits – ROI • 70% of IT Budget spent on integrating CSFB systems • Found new uses for old data • Squeezed more value out of legacy systems • Embrace heterogeneity • Increase business agility

  41. Single View of Customer Challenge • Call center application • Full view of customer interaction • Data distributed across multiple systems • Data inconsistent • Need a enterprise-wide data layer for more applications MetaMatrix Solution • Real-time access • Virtual views of data • Federated queries across systems • Common re-usable data definitions • Enterprise-wide data layer for re-use ROI • Better service of customers • Lower data integration costs Call Center Application MetaMatrix Data Sources

  42. Conceptual Architecture Information Consumption Application Integration XML / XML XML XML / Binary XML / Binary Binary Enterprise Information Integration Information Services Information Bus Metadata Management Data Quality Extract, Transform and Load Information Transport & Protocols Near Real Time Information Assets LOB App(s) Meta Rules ODS Content Data Staging External Data Data Data Warehouse (structured) (unstructured) LOB Data Data Marts

  43. Agenda • SOA Review • Data Integration • Data Services • Use Cases • Wrap-Up

  44. Recommendations • Start small, incrementally grow toward information as a service • Build the infrastructure on a project by project basis • Data security must be a top priority • Ensure performance as you go • Keep it in SOA context — information fabric is a means to an end, promoting information as a service • But ability to access and use information is a longer-term vision

  45. Thank You MetaMatrix Paul Anderson Technical Director 508 720 9266 panderson@metamatrix.com

More Related