1 / 20

Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services”

RDA 9th Plenary. Breakout 3, 5 April 2017 16:00-17:30. Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services”. Wrap-up and next steps (Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….). Why are we here?.

idellaj
Download Presentation

Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services”

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RDA 9th Plenary Breakout 3, 5 April 2017 16:00-17:30 Joint meeting: IG Brokering, IG Data Fabric “Advancing Data Fabric with Brokering services” Wrap-up and next steps(Stefano Nativi, Bridget Almas, Peter Wittenburg, Jay Pearlman, Larry Lannom, ….)

  2. Why are we here? • Data Fabric Core Components, such as the PIT and DTR, provide essential pieces of infrastructure for managing and sharing Research Data • But in order to be useful, they must be connected to provide functionality in research workflows • Brokers have traditionally served this role in SOA architectures, providing facade interfaces, mediation and abstraction

  3. GAP Composition Abstractions and Implementations PID Data Brokers Abstract Composition Executable Workflow(s) Objet Types Processing Brokers • PID and Object typing help fill the gap • Brokering services (e.g. Data services brokering and Processing services brokering) help fill the gap.

  4. Community Services architecture End-to-end architecture N N Client applications Client applications Mediation task Harmonization task Intermediation services N x M N + M Mediation task Mediation task Data Server M M From Human-Controlled Process to Type-Triggered Automatic Processing Community(Third-party) Data Servers

  5. Community Services architecture Client applications Intermediation services Community services(Third-party) Data Server Client-Broker-Server architecture Web-as-a-Platform Broker(s)

  6. Useful (architectural and Software) engineering Patterns • Separation of concerns (SoC) • separate architecture into distinct components,such that each component addresses a separate concern • Information Expert • Assign a responsibility to the component that has the information needed to fulfill it • Low coupling • Implement low dependency between the components

  7. Brokering Services • Provide the intermediation functionalities to pass from Abstraction to Implementations and vice versa • Apply Service-Oriented Architecture approach • Utilize Web-as-a-Platform (Web 2.0 patterns) • Intermediation-as-a-Service • Third-party services: not managed by Client/Server organizations

  8. Brokering Services • Benefits • Multi-purpose, Reusable, Participate in Composition • Sustainable through ability to evolve • Flexible, Configurable, Extensible • Challenges • Trust, Governance • New cultural and business model • Requires specialist knowledge • Types • Discovery and Access • Processing and Transformation

  9. Future: Data and workflow integration Set 1 Set 2 Set 3 • Goal: Make the life easier for scientists who are no experts in programming and handling data • Portals: integrate data and compute workflows Data Workflow Result Preview: ✔ ✔ … Data: <PID>

  10. Future: Data and workflow integration • Label data with PID • Label (parts of) workflows with PIDs #Load data mRNA = W1(D1) miRNA = W1(D2) #Analysis res1 = W2(mRNA, miRNA) res = W3(res1) #PID for result file = writeToFile(res) create_PID(file) #Plot for preview Plot(res) PID D1 PID D2 PID D3 PID W1 PID W2 PID W3 PID W4

  11. Matching in federation environment Repository metadata of DO bit sequences of DO PID Resolver MD bs 1 PID record DTR 5 PID Rights DB Type Record 3 2 12 4 6 rights record 13 10 Agent 9 Y 8 7 11 Controller WF 14 Matcher 14 14 Broker Profiles Broker Broker

  12. Scalability, efficiency, flexibility, sustainability, close to the Client needs Clients Application Community services & Interface new breed of organizations providing intermediation services to “abstract” current “implementations” and enable composition Composition Intermediation Supply system

  13. PID Centric Data Management and Access Brokering & Mediation services

  14. Biodiversity Use Case

  15. Climate data processing use case (I) Challenge: • Worsening ratio of bandwith vs. volume • Server-side processing – cultural change in scientific work Solution: • Favour support for most common processing tasks over complexity/variety • Service innovation rather than technical innovation • Understand users, train influencers, reach high quality

  16. Climate data processing use case (II) Motivation: • multiple initiatives to build data processing solutions – some exist, some to come • variety of data sources: CMIP6, Copernicus, ... • some standardization across them, but detail differences • brokering as middleware service – not for end-users, but software agents

  17. Climate data processing use case (III) Agent (user need) Broker component with a specific role: • Middleware service catalogs will come; defines broker role • Needed: Clear definition for broker component, respecting multiple roles • common interface for broker tasking and output format (to be fed into controller) WPS (final, preprocessing) Data sources Broker

  18. Broker acting as Controller

  19. Discussion

  20. Thank you !

More Related