1 / 23

NetCDF: Data Model, Programming Interfaces, Conventions and Format

NetCDF: Data Model, Programming Interfaces, Conventions and Format. Adapted from Presentations by Russ Rew Unidata Program Center University Corporation for Atmospheric Research Sponsored by NSF and NASA. www.unidata.ucar.edu. Overview. What is Unidata? What is netCDF? Data model

baba
Download Presentation

NetCDF: Data Model, Programming Interfaces, Conventions and Format

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. NetCDF: Data Model, Programming Interfaces, Conventions and Format Adapted from Presentations by Russ Rew Unidata Program Center University Corporation for Atmospheric ResearchSponsored by NSF and NASA www.unidata.ucar.edu

  2. Overview • What is Unidata? • What is netCDF? • Data model • Programming interfaces • Format • Limitations • Standardization Possibilities • Summary

  3. Unidata: A Community Endeavor • Community of educators and researchers at 120 universities, 30 other institutions, international in scope • Mission: providing data, tools, support, and community leadership for enhanced earth-system education and research • Funded primarily by the National Science Foundation • Atmospheric science community, expanding to oceanography, hydrology, other geosciences • Unidata Program Center: 25 staff, 15 developers

  4. Unidata Provides ... • User support, software development, maintenance, and community advocacy • Near real time “push” data dissemination: Internet Data Distribution among 170 institutions for use in research and education (IDD, LDM) • Tools for analysis and visualization (Gempak, IDV, McIDAS) • Data access and collections infrastructure (netCDF, udunits, THREDDS, ...)

  5. NetCDF Overview • Model for scientific data: variables, dimensions, attributes, coordinates • Libraries for data access: C, Fortran, C++, Java, Perl, Python, Ruby, Matlab, IDL, ... • Associated Conventions convey semantics • Format for portable binary data that supports direct access, metadata, appending new data, shared access (note that users need not know anything about format)

  6. Some Applications for netCDF • Gridded output from models: forecast, climate, ocean, atmospheric chemistry, ... • Observational data: surface, soundings, satellite, lightning, aircraft, radar, ... • Various scientific data archives • Chromatography, mass spectrometry, medical imaging, ... • In general: platform-independent, language-independent interface to insulate data providers from data consumers

  7. NetCDF’s Niche • Simple data model for scientific datasets • Portable, self-describing data • Appendable, sharable, archivable • Direct access for efficient subsetting • Metadata via attribute conventions such as CF • Flexible remote access via OPeNDAP, HTTP, WCS • Lots of applications: NCO, ncbrowse, ncview, IDV, IDL, MATLAB, ArcGIS, ... • Language interfaces include C, Java, Fortran, C++, Perl, Python, Ruby, ...

  8. NetCDF Data Model • Variables: name, shape (list of Dimensions), type, Attributes, values • Dimensions: name, length • Attributes: name, type, value(s) • Coordinate Variables: values associated with Dimensions

  9. Dataset Dimension Attribute name: String length: int isUnlimited: boolean name: String value: Vector type: DataType Variable name: String shape: Dimension[ ] type: DataType CoordinateVariable ValuesArray name: String shape: Dimension[1] type: DataType type: primitive shape: int[ ] NetCDF Data Model location: String same name

  10. Availability of netCDF • Freely available source, binaries: • www.unidata.ucar.edu/packages/netcdf • Unidata-supported interfaces: • C, C++, Fortran-77, Fortran-90, Java, Perl • Platforms: • Unix, Windows, Mac OS X, ... • License: • X Windows/MIT style open source license • Used in several commercial applications • Status: • version 3.5.1 released February 2004 • netCDF-4 under development

  11. Widespread netCDF Use • 500 entries representing 32 in netcdfgroup mailing list: academic institutions, government agencies, commercial firms • Downloaded by more than 2000 distinct hosts in 55 countries in last 10 years • Software for manipulating/displaying netCDF: more than • 15 commercial packages, • 56 freely available package • Adopted by Analytical Instrument Association [AIA] to implement the Analytical Data Interchange Protocols • Neuro-Imaging Laboratory of the Montreal Neurological Institute

  12. netCDF Language Interfaces • Conventions of each language supported by interfaces • C: core implementation • C++, Fortran-77, Fortran-90, Perl: call C library • Java: independent pure-Java implementation • Python, Ruby, IDL, Matlab, Ada: third-party layers over C library • Data is language-neutral

  13. Java Interface • Currently most advanced netCDF interface, used to test new ideas • Client access to data servers: • HTTPD • OPeNDAP/DODS • Advanced coordinate system support for general and georeferenced coordinates • Virtual datasets to aggregate, subset, redefine, using NcML (XML representation for netCDF)

  14. NetCDF Format • Purpose: insulate users, applications, and data from machine architectures, format details, additions to data • Provides data that is • self-describing • portable • directly accessible • appendable • Still support original XDR-based format (1988) • 32-bit file offset limits file size • New format with 64-bit offsets for huge files

  15. Records grow in the UNLIMITED dimension NetCDF File Format • Header • Dimensions • Attributes • Variable metadata • Fixed-size variables • Record variables • Use UNLIMITED dimension • XDR-like represenations for all data and metadata

  16. Some Current netCDF Limitations • Not a database system: no keyed access, transactions, multiple concurrent writers • Strings are just character arrays • No nested data structures • No “ragged arrays”, variable length records • Only one growable dimension per dataset • No built-in compression or packing • Georeferencing requires agreement on conventions • ...

  17. NetCDF CF Conventions • Climate and Forecast metadata conventions • Metadata for use, not discovery • http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html • Adopted by: PCMDI, PRISM, ESMF, NCAR, Hadley Centre, GFDL, various EU projects • Developed by volunteers, active email list • Includes georeferencing conventions for multidimensional coordinate systems • Includes a “Standard Name Table” for quantities • Google for “CF standard name table” • 76 of 416 entries (as of June 3, 2004) relate to hydrology

  18. FES Client Applications GIS Client Applications FES Servers GIS Servers THREDDS Server GIS Server THREDDS Server OGC or OPeNDAPADDE. FTP… protocols OpenGIS Protocols: WMS, WFS, WCS Satellite, radar, forecast model output, … datasets OGC or proprietary GISprotocols GIS Server Demographic, infrastructure, societal impacts, … datasets Taking Advantage of Web Services for Data System Interoperability

  19. Obstacles to GIS-netCDF Connection • netCDF data model (finite sampling of real functions) differs from relational, GIS datamodels • Some netCDF data not spatial • Much netCDF data multidimensional, > 2D • Georeferencing for netCDF data uses multiple conventions • Typical netCDF datasets may be very large • Large netCDF datasets may be on a remote server

  20. Unidata GIS-netCDF Efforts • Prototype netCDF to geoTIFF converters • Parser for georeferencing conventions implemented in Java • Unidata GeoGrid from georeferenced netCDF variable supported in Java • GridDataset: collection of GeoGrids in a netCDF dataset with methods for getting volumes and slices • NcML for wrapping any new conventions around existing data

  21. netCDF-relatedChanges Proposed in WCS 1.1 • WCS encoding “profiles” instead of fixed list of encoding formats • CF-netCDF is first proposed WCS encoding profile • Multiple “variables” or “parameters” in a coverage (e.g., pressure, temperature, etc.) • Coverages with 3 spatial dimensions • Coverages with multiple time dimensions (e.g. forecast time in model output) • Non-spatial “height” dimension, (e.g., atmospheric pressure, ocean density) • Irregularly-spaced grids • Technical committee votes on entirety of WCS 1.1 this summer.

  22. Possible netCDF Standardization • File format • Applications programming interfaces • Conventions (e.g., CF, Climate and Forecast) • Release level: • netCDF 3 • netCDF 4 later this year (will include access to HDF 5 files) • Common Data Access Model (with elements of netCDF, HDF, and OPeNDAP data models) • (Relationship to WCS encoding profile specification?)

  23. More Information • NetCDF documentation and codehttp://www.unidata.ucar.edu/software/netcdf/ • CF Conventionshttp://www.cgd.ucar.edu/cms/eaton/cf-metadata/index.html • Russ Rewruss@unidata.ucar.edu

More Related