1 / 26

DWD GISC Implementation

DWD GISC Implementation. GISC Development Team. Agenda. Welcome Architecture High level Architecture Hardware Performance Tests International Collaborations and Co-operations Contributions to WIS Q&A. Design Principles of DWD GISC.

mira
Download Presentation

DWD GISC Implementation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DWD GISC Implementation GISC Development Team

  2. Agenda • Welcome • Architecture • High level Architecture • Hardware • Performance Tests • International Collaborations and Co-operations • Contributions to WIS • Q&A

  3. Design Principles of DWD GISC • Reuse of existing well established software and implementation methodology • Use of established Interfaces – in particular to „critical components“ to allow later exchange if necessary • Metadata Providing • Authentication and Authorization • Modular concept to address performance and scalability requirements • Clear separation of responsibilities • Keep things simple • Describe external interfaces

  4. DWD GISC - Development • In house development team and support of business units • Professional Services • Fraunhofer – research facility • Con terra – experts on Metadata • Co-operation with companies • ask • Contributes to the development with 2-3 persons • Supports WMO events (EC Demo) • IBL • Long term partner • Supports WMO events (EC Demo) • Face lift of DWD GISC • Offers professional services (integration and maintenance) for DWD GISC

  5. DAR Components • – Metadata Harvesting • – Metadata Providing Data & Product Import Management & GISC Synchroniser Metadata Database SRU Search and Retrieval by URL DAR Generator/Importer Subscription Management Web Interfaces – Customer – Operator – Administrator Data & Product Access Management File pools, Archives, Databases 24 h Cache WIS Global Ex. High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer

  6. Impl. Start: Web Interface Subscription DAR Catalogue Perf. Study Concept Phase DAR MetadataGenerator Project Start Test MoU CMA G2G com. Workshop G2G com. Tests Test G2G com. Workshop VGISC Backup WS VGISC Workshop Timeline Data & Product Management MetadataStudy SRU OAI Infrastructure 11/08 03/09 11/09 01/10 05/10 06/10 01/09 05/09 07/09 09/09 03/10 08/10 09/10 OAI & WIS Workshop OAI Tests EC Demo

  7. DAR Components • – Metadata Harvesting • – Metadata Providing Data & Product Import Management & GISC Synchroniser Metadata Database SRU Search and Retrieval by URL DAR Generator/Importer Subscription Management Web Interfaces – Customer – Operator – Administrator Data & Product Access Management File pools, Archives, Databases 24 h Cache WIS Global Ex. High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer

  8. High Level Architecture: Web Interfaces • GISC • Redesign of WebWerdis • Navigation structure • Search functions • Simple Search • Extended Search (e.g. Full Text Search) • Expert Search • Internationalization (I18N) • Integrate logging framework (necessary for monitoring) • Code clean-up • Entry points for External Application (e.g. Online Resource) • AUST • Redesign of AUST

  9. DAR Components • – Metadata Harvesting • – Metadata Providing Data & Product Import Management & GISC Synchroniser Metadata Database SRU Search and Retrieval by URL DAR Generator/Importer Subscription Management Web Interfaces – Customer – Operator – Administrator Data & Product Access Management File pools, Archives, Databases 24 h Cache WIS Global Ex. High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer

  10. High Level Architecture: DAR Generator, Importer and Components • DAR Metadata Provider • jOAI (also used by NCAR/UCAR – reference site) • DAR Metadata Harvester • OAIHarvester2 • Modified Source Code with CMA together • In house development of a scheduler and workflow component (done by ask) • DAR Metadata Generator • In house development by DWD shared with CMA, Hong Kong, WMO, Brazil, Norway, Australia, South Africa, … • DAR Metadata Importer • In house development by DWD

  11. DAR Components • – Metadata Harvesting • – Metadata Providing Data & Product Import Management & GISC Synchroniser Metadata Database SRU Search and Retrieval by URL DAR Generator/Importer Subscription Management Web Interfaces – Customer – Operator – Administrator Data & Product Access Management File pools, Archives, Databases 24 h Cache WIS Global Ex. High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer

  12. High Level Architecture: Delivery Components • Subscription Management • Controller: time and event triggered management • AFD • Workhorse in the background • GISC-to-GISC communication • File distribution • White List • Message Switching System • REST Web-Service to manage and control Moving Weather • First proposal for generic Interface

  13. DAR Components • – Metadata Harvesting • – Metadata Providing Data & Product Import Management & GISC Synchroniser Metadata Database SRU Search and Retrieval by URL DAR Generator/Importer Subscription Management Web Interfaces – Customer – Operator – Administrator Data & Product Access Management File pools, Archives, Databases 24 h Cache WIS Global Ex. High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer

  14. High Level Architecture: SRU • SRU • WMO SRU2JDBC implementationCooperation between WMO and DWD available as Open Source • In house development by DWD: Mapping of CQL (Contextual Query Language) to SQL (done by ask) • External configuration file for Mapping • It is easy to add new indices • Database specific feature (e.g. functions, special operators) are kept outside of the code • All Queries and Responses are encodedaccording to the SRU-Standard allowingfor both interactive as well as automatedusage – e.g. by catalogue systems

  15. Integration System Operations OAI Infrastructure Web Cache DMZ OAI Infrastructure INTRANET Application Server Adapter & Scheduler Application Server Adapter & Scheduler Hardware Architecture –Infrastructure

  16. Performance • 05/2009 Performance Study of DAR Metadata Catalogues – Both Systems terraCatalog 2.3 and Geonetworks 2.2 do not fulfill all Performance Requirements • 09/2009 CMA and DWD did performance study of GISC-to-GISC communication solutions – JMA Blog solution and AFD • 03/2010 CMA and DWD did performance tests of OAI Provider solution • 04/2010 INMET and DWD did performance tests of GISC-to-GISC communication solutions – AFD and Moving Weather (IBL) • 07/2010 Internal performance tests of DAR Metadata Catalogue and SRU – Lessons learned • Database is hot spot, especially when we talk about Full Text Search • Oracle standard reports • Limit the result set – configuration parameter • Optimize SQL Statements • Explain plan • Rewrite SQL Statements • Indices • Improved responses for most critical queries by an order of magnitude

  17. International collaboration: CMA (GISC) • GISC-to-GISC communication • Synchronization of 24 hour cache • Extensively tests of JMAs Blog solution • Extensively tests of AFD • GISC-to-GISC communication • Description for GISC-to-GISC backup based on XML Schema

  18. International collaboration: CMA (GISC) • DAR Metadata • DAR Metadata Generator • Metadata Template • Request for Clarification paper 11/2009 (CMA, JMA and DWD) • Metadata Providing / Harvesting • Functional tests of standard OAI provider • Common development of OAI Harvester • Extensively performance tests for DAR Metadata synchronization • SRU • Mapping indices to DAR Metadata elements • Workshops: • GISC-to-GISC communication workshop 04/2009 at DWD • GISC-to-GISC communication workshop 09/2009 at CMA • OAI workshop 03/2010 at CMA

  19. International co-operation: JMA (GISC) • DAR Metadata • Design of DAR Metadata • Request for Clarification paper 11/2009 (CMA, JMA and DWD) • Metadata Providing / Harvesting • Synchronization tests based on OAI • GISC-to-GISC communication • Knowledge exchange JMA Blog solution • JMA will take XML Schema (based on a proposal from CMA and DWD)as working draft for GISC-to-GISC backup • Workshop • WIS Workshop at JMA 03/2010

  20. International co-operation: Brazil (GISC) • DAR Metadata • DAR Metadata Generator • Metadata Providing / Harvesting • Synchronization tests based on OAI • GISC-to-GISC communication • Moving Weather • AFD

  21. International collaboration: EUMETSAT (DCPC) • DAR Metadata • Conceptual design of DAR Metadata • Knowledge exchange • DAR catalogue • Extensively knowledge exchange • VGISC ITT • Attended as expert • EUMETSAT is connected as DCPC since 23th of August 2010 • Test Phase 06/07 2010

  22. International co-operation: Hong Kong (DCPC) • DAR Metadata • DAR Metadata Generator • Metadata Providing / Harvesting • Synchronization tests based on OAI International co-operation: NCAR/UCAR (DCPC) • DAR Metadata • Knowledge exchange • Metadata Providing / Harvesting • Synchronization tests based on OAI

  23. International co-operation: VGISC • Since 2002 VGISC – Météo France, UK MetOffice, DWD, met.no, ECWMF and EUMETSAT • SIMDAT • VGISC ITT • Several workshops regarding Backup and Recovery

  24. Contributions to WIS • DWD participates several ETs, ICG, … • Released Metadata catalogue performance study to community  Geonetwork improvements • Shared results of DAR Metadata analysis with WIS community • DAR Metadata Generator • Released SRU2JDBC in co-operation with WMO as Open Source http://sru2jdbc.svn.sourceforge.net/ • Active member of jOAI community • Contribute experts for WIS Jump Start Offer

  25. Q&A

  26. Live Demohttp://gisc.dwd.de/

More Related