Metadata and Data Management Activities winthin NA 3-7

1. Metadata and Data Management Activities winthin NA 3-7 S. Iona (HCMR), D. Schaap (MARIS), L. Rickards (BODC), F. Nast (BSH)

2. Outlines • SeaDataNet Objectives • Training and Capacity building • Discovery System • Maintenance • Upgrade • Current Status • Summary

3. SeaDataNet Objectives • To network existing oceanographic data centres already nationally funded • To develop an efficient distributed pan-European marine data management Infrastructure (a “unique Virtual Data Centre”) • To provide on-line access to integrated databases of standardised quality by using adapted communication & information technology

4. NA3-Training and Capacity building • Aims: • to ensure that the data and metadata to be integrated in the system will be formatted, checked for quality and disseminated according to the common protocols developed during the project • to transfer expertise and to train IT experts of the SeaDataNet data centers in the basics, installation and operation of the SeaDataNet technical components

5. NA3-Training and Capacity building • Training Workshops, IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) • 1. February 12-17 2007: • focused on data re-formatting, quality control, and processing using ODV software. • training on project metadata and related web services.

6. NA3-Training and Capacity building • Training Workshops, IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) 2. June 4-5 2007: • dedicated on generating XML records with the use of MIKADO tool

7. NA3-Training and Capacity building • Training Workshops, IOC Rroject Office for IODE, Oostende, (Belgium): • (Training material and presentations available on SDN Extranet) 3. June 16-19 2008: • use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies) • data quality control and assessments, using ODV software • analysis and data presentations, using ODV – DIVA software

8. Discovery System • The SeaDataNet Discovery System is an integrated catalogue service, aiming at facilitating marine data searching, location and retrieval. • The Objectives are to: • Maintain and expand the national metadata-bases. • Standardize the information using common vocabularies and reference tables. • Interconnect the national inventories in a common Pan-European directory.

9. Discovery System • It is composed of several thematic inventories of different levels: • European Directory of Marine Environmental Datasets (EDMED) • Cruise Summary Reports (CSR) • European Directory of the initial Ocean-observing Systems (EDIOS) • European Directory of Marine Environmental Research Projects (EDMERP) • European Directory of Marine Organizations (EDMO) • Common Data Index (CDI)

10. Discovery System Structure

11. High Level Directories – EDMED, CSR, EDIOS for describing collected observation datasets (by ships, by laboratories, by continuous observing system) Discovery System Structure

12. Common Reference Tables – EDMERP, EDMO hold research projects and organizations metadata common to higher directories Discovery System Structure

13. The Common Data Index (CDI) provides access to the data, information and products by data type and/or any other field distributed by the TA platforms Discovery System Structure

14. Maintenance Version 0 – 2006-2007 • Continuation and maintenance of existing Sea-Search system : • the data access needs several different requests to each data centres • and the data sets are delivered in different formats

15. Discovery System Upgrate • Version 1 – 2008-2010 • Setup of the integrated online data services to users : • networking of 10 “interoperable” data centres of the Technical Task Team • unique request to the interconnected data centres • and the data sets are delivered with a unique format • Progressive integration of 10 data centres by end of 2008

16. Current Status • Frozen directories • Content upgrade from Version 0 to Version 1 using on-line (like CMS forms) and off-line tools that produce XML ISO 19115 compliant exchanges and developed by the Technical Task Team in the joint research activities JRA1,JRA2:

17. Cruise Summary Reports • Cruise Summary Reports (CSR = former ROSCOPs) are the usual means for reporting on cruises or field experiments at sea. Traditionally, it is the Chief Scientist's obligation to submit a CSR to his/her National Oceanographic Data Centre (NODC) not later than two weeks after the cruise. This provides a first level inventory of measurements and samples collected at sea.

18. New features in SDN V1 • The major differences to previous V0 version are: • CSR Local ID given by Data Centrefor future updates- •  makes it easier to modify/improve existing reports • Most entities now havedefined vocabularies • eg. EDMO for organisation, EDMERP for projects and many more •  no more spelling/typing mistakes or deviant interpretations • Mandatory fields •  improves quality of report

19. Structure of Report • Similar to the former ROSCOP forms the Cruise Summary Report has 4 basic parts: • General Cruise Information • Mooring Description • Sampling/Measurement Description • Information on Geographical Coverage

20. How to submit CSRs? • 2 methods for generating CSRs • Online for individual entries (CMS) • XML files for bulk submission with the use of MIKADO tool • Both tools can be applied for new entries as well as updates !

21. Link to CSR Discovery Online Content Management System http://seadatanet.bsh.de/csr/on_line/V1_index.html

22. How to update existing reports? Online update: • contact BSH-DOD requesting CSRs to be updated • BSH-DOD loads requested CSRs to entry database • E-Mail with list of CSR ref. no. and passwords • modify CSRs in entry database • save and submit

23. How to update existing reports? Offline – XML update: • contact BSH-DOD requesting CSRs to be updated • BSH-DOD sends requested CSRs as XML V0 files • 1 file/cruise using BSH CSR ref. no. as file name • all free text information included • submit modified CSRs in XML V1 format to BSH-DOD • use BSH CSR ref. no. as central identifier (CSR ID) • include local identifier (from NODCs) for future updates

24. User Interface - Discovery http://seadatanet.bsh.de/csr/retrieve/V1_index.html Results

25. Download in V0 format User Interface - Report

26. Content Conversion into V1 • Status of preliminary conversion for39839cruise reports: • Responsible Laboratories (EDMO codes) ~ 20% • EDMO codes of Chief Scientists ~ 20% • Ship/Platform (ICES codes) ~ 60% • Port of departure/return (C381): ~ 30% • General Ocean Area (C161+C162): ~ 90% • Sampling units (common vocab. L181) ~ 20% • EDMO codes of Principal Investigators ~ 3%

27. Continuing operation and population • Heraklion/Triest we got 1071 CSRs • Online 33 % • XML 34 % • ICES 25 % • Others 8 % • Since Triest we got 1263 CSRs • Online 52 % • XML 44 % • ICES 4 % • Others 0 %

28. Status • Number of entries (in an Oracle DB): 39 839 • Characteristics: • From 1873-2008, 2105 ships from 48 countries

29. CSR Contents 2156 CSR Entries during SDN Project (1.4.2006 - 15.09.2008) • V1 in operation since 21. July 08 • Russia and Spain with online upgrades • 42 already using new V1 from 8 countries, • Lithuania first after the 3rd SDN workshop in Oostende,

30. European Directory of Marine Organizations • The directory lists the organization profiles of all (1000+) Data Holding Centres, Research Institutes, Monitoring Agencies and Research Vessel operators, that have an active role in one or more of the SeaDataNet Discovery services (EDMED - data sets, EDMERP - research projects, CSR - research cruises, EDIOS - observing stations/ systems, and CDI - index to data). • Direct crosslinks are provided to their entries in these directories. • The organization entries are maintained online per country by the SeaDataNet partners. • new Web service for retrieving EDMO entries in XML

31. Online Content Management System http://seadatanet.maris2.nl/vu_organisations/welcome.asp EDMO CMS geo-locator via Google maps

32. EDMO V1 search and retrieval http://seadatanet.maris2.nl/edmo

33. EDMO EDMO Contents • Number of EDMO entries is quite stable • As part of producing CDI records a lot of new organisations have been added; also existing entries have been altered due to re-organisations, double entries etc. • Content upgrade from V0 to V1 is finished

34. EDMO European Directory of Marine Environmental Data • EDMED is a directory of data sets relating to the marine environment. It covers a wide range of disciplines including marine meteorology; physical, chemical and biological oceanography; sedimentology; marine biology and fisheries; environmental quality monitoring; coastal and estuarine studies; marine geology and geophysics etc.

35. EDMED Contents • Currently, EDMED describes: over 3500 datasets from 700 data holding centres

36. Total No. of Datasets and Data Holding Centres

37. Progress • Content Management System will be launched soon (Content upgrade will start then) • Web interface is under development

38. European European Directory of Marine Environmental Research Projects • EDMERP is a European directory of research projects relating to the marine environment. It covers a wide range of disciplines including marine meteorology; physical, chemical and biological oceanography; sedimentology; marine biology and fisheries; environmental quality; coastal and estuarine studies; marine geology and geophysics etc.

39. EDMERP developments • double entries have been taken out, because projects are entered and maintained by the country of the coordinator, who can add via EDMO entries all related partners • capability of creation of sub-accounts for institutes in the NODC’s country, while the NODC safeguards the quality by having the chief editor role before publishing • new Web service EDMERP entries in XML (export and import).

40. EDMERP Retrieval Browse list Additional details

41. http://seadatanet.maris2.nl/vu_edmerp/welcome.asp EDMERP – CMS

42. EDMERP Contents • Number of EDMERP entries reduced during upgrading because of removal of duplicates

43. European Directory of Ocean Observing Systems • EDIOS is the European Directory of the Ocean-observing System, a unique searchable metadatabase of observing systems operating repeatedly, regularly and routinely in European waters. • It contains metadata on European observing systems such as platforms, repeated ship-borne measurements, buoys, remote imagery, etc.

44. Progress • No new input has been requested over the year • Awaiting new technological developments (XML schema and new version of Mikado) • BODC has produced an improved Oracle (database) schema • Supported by common vocabularies, EDMO and EDMERP

45. New User Interface: http://seadatanet.maris2.nl/v_edios/search.asp

46. Common Data Index The CDI provides an index (metadatabase) to individual data sets. For comparison: the present European Directory of Marine Environmental Datasets (EDMED) gives an overview of datasets at a high metalevel. Each EDMED data set description covers a broad set of individual measurement data. The CDI gives references to these individual measurement data, providing a more detailed insight into the available datasets.

47. New interface

48. CDI Contents • Partners are now working with the new Mikado to upgrade V0 to V1 • The IIrd training workshop has been very useful for transferring expertise and instructing partners in use of CDI format and tools.

49. Summary • 2007/2008 • EDMED: 3.500 • CSR: 39.648 • EDMERP: 1.600 • EDMO: 1.134 • CDI: 341.499 • 2006/2007 • EDMED: 3.000 • CSR: 38.525 • EDMERP: 1.507 • EDMO: 1.122 • CDI: 195.131 A very important increase in the CDI contents but extra coverage of national data is need