1 / 27

THREDDS Data Server

THREDDS Data Server. John Caron Unidata/UCAR Mar 14, 2005. Outline. THREDDS Catalog Services THREDDS Data Server NetCDF-4 NetCDF-Java 2.2. THREDDS. Thematic Realtime Environmental Distributed Data Services Middleware between data providers and data users

breena
Download Presentation

THREDDS Data Server

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. THREDDS Data Server John Caron Unidata/UCAR Mar 14, 2005

  2. Outline • THREDDS Catalog Services • THREDDS Data Server • NetCDF-4 • NetCDF-Java 2.2

  3. THREDDS • Thematic Realtime Environmental Distributed Data Services • Middleware between data providers and data users • Enable discovery and use of scientific data • Not a data portal • XML over HTTP • Java Servlets

  4. Catalog.xml Catalog Generator Catalog.xml THREDDS Catalogs HTTP Server Catalog.xml Data Server(s) DODS, ADDE, FTP, HTTP Client Application Datasets hostname.edu

  5. Catalog Service Catalog Generator Catalog.xml Query Resolver Service DQC.xml Resolver Service URI URL Dynamic Catalogs, data services HTTP Tomcat Server Catalog.xml Client Application Data Server(s) DODS, ADDE, FTP, HTTP Datasets hostname.edu

  6. Dataset Query Capability (DQC) • Dynamic and very large datasets • NEXRAD (level 3) from IDD • 150 stations • 20 products • Irregular times • The DQC describes the queries that the data server is capable of responding to.

  7. Resolver Services • Logical Dataset, eg “latest ETA model run” • Dataset with Service type “Resolver” • On the client, the URI of the logical dataset is sent to the data server • The server finds what is available and returns an actual dataset URL.

  8. Limitations • Catalogs are very general – “data protocol agnostic”. • Automatic metadata extraction needs to “crack” the datasets. • Data Servers must be separately installed, maintained, etc

  9. THREDDS Data Server • Add the data access! • 100% Java – single thredds.war • requirement, not goal • Data input is NetCDF Java 2.2 library • Data output: • OpenDAP • WCS (in progress) • FTP, others (future)

  10. THREDDS Data Server HTTP Tomcat Server Catalog.xml • THREDDS Server • OPeNDAP • WCS Application NJ22 library hostname.edu Datasets IDD Data

  11. THREDDS OpenDAP Server • Catalogs instead of dods_dir • Next version of C++ library will also have THREDDS catalogs • Catalogs automatically generated • Based on DODS-Java library and the Aggregation Server. • Aggregation probably done at nj22 level

  12. THREDDS WCS Server • WCS 1.0 specification • Proposing to add NetCDF as “blessed” binary file return type • OGC interoperability experiment • Difficult to find clients to work with • Working with CADCorp client • Liping Di (George Mason) • NASA GSFC ‘Earth Science Gateway’ • Looking for others

  13. THREDDS WCS Server • Return formats • Geotiff floating point • NetCDF binary • No reprojections for now • Must have Grid coordinate system • Issues • Vertical coordinates • Time coordinates • Large number of coverages (granularity)

  14. NetCDF-4 NetCDF-Java 2.2

  15. NetCDF-4 • Project funded by NASA to create new version of netCDF using the HDF5 file format. • “Extend and merge” netCDF and HDF5: • Widespread use and simplicity of netCDF-3 • Generality and performance of HDF5 • Specifically, we are funded to create netCDF-4 C library API, using HDF5 library underneath. • Russ Rew (PI), Ed Hartnett (Unidata), Mike Folk (co-PI), Quincey Koziol (NCSA)

  16. netCDF-3 Interface netCDF-4 Library HDF5 Library NetCDF-4 Architecture NetCDF-4 C Library

  17. NetCDF-4 and Java • 100% Java library for netCDF-4 files possible? • NetCDF-Java 2.1 already a 100% Java library for netCDF-3 files (and OPeNDAP) • NetCDF-Java 2.2: read HDF5 to determine what netCDF-4 data model should be • “Common Data Model” emerged for netCDF, HDF5 and OpenDAP. • OpenDAP 4.0 protocol being worked on • The 3 groups are actively discussing

  18. NetCDF-Java 2.2 (nj22) • 100% Java library for netCDF, HDF5, OpenDAP, and more • THREDDS integrated • Under development • Alpha release: Nov 2004 • Beta release: May 2005 • Release: summer 2005

  19. THREDDS Catalog.xml Application Scientific Datatypes NetCDF-Java version 2.2 architecture Image Trajectory Grid Station Radial NetcdfDataset NetcdfFile OpenDAP ADDE HDF5 I/O service provider NetCDF-3 NetCDF-4 GRIB NIDS GINI Nexrad … DSMP

  20. I/O Service Provider Implement this interface: public interface IOServiceProvider { boolean isValidFile( RandomAccessFile raf); void open( RandomAccessFile raf, NetcdfFile ncfile); Array readData( Variable v2, List section); // only if you use Structures Array readNestedData( Variable v2, List section); }

  21. CDM Goal: N + M instead of N * M things on your TODO List File Format #1 Visualization &Analysis NetCDF file File Format #2 OpenDAP Server File Format #N WCS Service

  22. Common Data Model (CDM)

  23. Coordinate Systems • Same underlying mathematics as VisAD, ASCII

  24. Scientific Data Types • Point / Station (metars) • Unconnected • DQCs used with IDD station/radar data • Trajectory (floating buoys) • 1D connected • Radial (NEXRAD) • elevation ,azimuth, distance • Grid (Model output) • cartesian x, y, z • WCS

  25. THREDDS Data Server HTTP Tomcat Server Catalog.xml • THREDDS Server • OPeNDAP • WCS Application NJ22 UI library NJ22 library NJ22 library hostname.edu Datasets IDD Data

  26. Summary • THREDDS data server combines Catalog Services and data servers • As nj22 matures, data server will have more functionality • OpenDAP, WCS • NetCDF-4: simplicity of netCDF and flexibility of HDF5

More Related