1 / 40

The Open source Project for a Network

The Open source Project for a Network Data Access Protocol (OPeNDAP): A Discipline Neutral Approach to Interoperability on the Internet Presented at the Fall AGU Meeting in San Francisco, CA 9 December 2003 Peter Cornillon and Dan Holloway Graduate School of Oceanography University of Rhode

naeva
Download Presentation

The Open source Project for a Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Open source Project for a Network Data Access Protocol (OPeNDAP): A Discipline Neutral Approach to Interoperability on the Internet Presented at theFall AGU Meeting in San Francisco, CA9 December 2003Peter Cornillon and Dan HollowayGraduate School of OceanographyUniversity of Rhode and James Gallagher The Open source Project for a Network Data Access Protocol (OPeNDAP)

  2. Outline • Trends in data system development • Data system elements • OPeNDAP • A data access element • In NVODS – an example • Status • Conclusions

  3. Trend in Data System Development Is away from centrally designed, implemented and maintained systems toward The integration of independently designed, implemented and maintained system elements.

  4. Data systems generally involve some combination of the following elements Discovery Analysis/ Visualization Archive Access/ Delivery

  5. Historically these elements or subsets thereof have been developed and managed by the group assembling the data system.

  6. Discovery Analysis/ Visualization Data System A Archive Access/ Delivery

  7. Discovery Analysis/ Visualization Analysis/ Visualization Analysis/ Visualization Analysis/ Visualization Data System B Archive Access/ Delivery

  8. With the advent of the Internet data system elements are increasingly being designed, implemented and managed independently by different groups.

  9. Discovery Discovery Access/Delivery Application/ Visualization With a Plethora of System Elements … Archive Archive … Access/Delivery .. Application/ Visualization

  10. And “complete” data systems being created from different combinations of the elements

  11. GFDL netCDF URI HDF GSFC Binary Matlab IDL GrADS Ferret IDV Excel VisAD Access ncBrowse An example: NVODS – The National Virtual Ocean Data System … OPeNDAP …

  12. More NVODS – Data Discovery GFDL netCDF URI HDF GSFC Binary IDL IDV Access ncBrowse Matlab GrADS Ferret VisAD Excel GCMD NVODS OPeNDAP

  13. NVODS – Defining the Data System GFDL netCDF URI HDF GSFC Binary Matlab Access IDL ncBrowse IDV VisAD Ferret GrADS Excel ODC GCMD NVODS OPeNDAP

  14. Responsibility Note that in distributed systems responsibility is distributed. For NVODS responsibility for • The data lies with the data providers. • The data access protocol lies with OPeNDAP. • Application packages (Matlab, Ferret, Excel…) with the developers of these packages. • Data location with the GCMD and NVODS.

  15. Interoperability And Metadata

  16. The Ultimate Objective of a Data System • To provide requested data to the user’s analysis/visualization package in a consistent, readily useable form. For example: A user might want all ocean temperature values (with associated times and locations) that lie between 90 and 110 m and have uncertainties less than 1ºC.

  17. Interoperability • Know the format of these data objects. To achieve this objective, system elements must interoperate; i.e., the system must: Be capable of finding all data of interest. • Be capable of transforming from any of these formats to that required by the application software. • Understand the semantics of the data.

  18. Metadata The degree of system interoperability is determined by the associated metadata. These interoperability requirements require in turn descriptions of the data; i.e., metadata.

  19. Syntactic and Semantic Metadata The required metadata falls in two classes: • Syntactic metadata – Informationabout the data types and structures at the computer level - the syntax of the data; e.g., variable T is a 20x40 element floating point array • Semantic metadata – Information about the contents of the data set. e.g., variable T is sea surface temperature with units of ºC [RBH Note: This syntax is actually data structure. Syntax is the format of request.]

  20. T 10 20 30 40 Syntactic and Semantic Metadata • Syntactic metadata provides the information needed to read and plot the data, but in general not to label the axes.

  21. Syntactic and Semantic Metadata • Semantic metadata provides the information needed to label the axes in a plot. Temperature (ºC) 0 4 8 12 16 18 • 20 30 40 • Time (Hours)

  22. Interoperability and OPeNDAP The two types of metadata suggest two levels of interoperability: • Syntacticinteroperability – Consistent format representation across data sets. • Semantic interoperability – Consistent semantic interpretations of the data. OPeNDAP mandates syntactic interoperability via a strict syntactic description of all data available via the system.

  23. Interoperability and OPeNDAP OPeNDAP does not mandate semantic interoperability although it does allow for it. • The OPeNDAP data access protocol supports containers for semantic metadata, but places no requirements on the contents of these containers. • Some data sets are well described, others are not.

  24. OPeNDAP in NVODS GFDL netCDF URI HDF GSFC Binary Matlab Access IDL ncBrowse IDV VisAD Ferret GrADS Excel ODC GCMD NVODS OPeNDAP

  25. Data Selection • The OPeNDAP data access protocol allows for data subsetting at the server. • All data sets may be subsetted by variable, by the value of a variable for sequences and by array index for arrays. • If the data sets semantics are known, the data may be subsetted based on these semantics.

  26. Discipline Neutrality • The OPeNDAP data access protocol is discipline neutral. It provides for • Consistent syntactic representation of the data regardless of the semantics. • Discipline specific semantics may be layered on the data.

  27. Discipline Neutrality • Because of its discipline neutrality it is being used in a variety of data systems: • Oceanography • Meteorology • Solar-terrestrial physics • Hydrology

  28. OPeNDAPStatus

  29. OPeNDAP Server Sites in the US

  30. OPeNDAP Client and Server Status Flat Binary netCDF HDF4 Matlab DSP Tables SQL FITS CDF CEDAR Data Data Data Data Data Data Data Data Data Data netCDF Matlab JGOFS FITS FreeFrom HDF4 DSP JDBC CDF CEDAR OPeNDAP Data Connector IDL Client Matlab Client netCDF Java netCDF C Web Browser Ferret GrADS IDV VisAD ncBrowse Matlab IDL Excel

  31. Special Servers

  32. Center for Ocean-Land-Atmosphere Studies Statistics

  33. COLA/GDS Statistics

  34. COLA/GDS Statistics

  35. COLA/GDS Statistics

  36. Lamont Statistics

  37. Interesting OPeNDAP Access Statistics LDEO data accesses for 1st quarter of 2002 LDEO data accesses for 4th quarter of 2002

  38. Conclusions • Responsibility in distributed data systems of the future will be distributed among the developers of the elements comprising the system. • To maximize the flexibility in assembling systems, system elements should remain as general as possible. • The OPeNDAP data access protocol has been designed to be discipline neutral in anticipation of the shift in data system development.

  39. Conclusions Data systems based on the integration of independently developed system elements offer many more opportunities than more traditional centrally developed ones.

  40. Conclusions “When systems evolve separately it is like sexual reproduction as opposed to centrally developed systems which evolve asexually” J. Caron 2003

More Related