PIRATA data and products available through a DODS/OPeNDAP Python server

1. PIRATA data and products available through a DODS/OPeNDAP Python server Guilherme P. Castelão & Roberto A. F. De Almeida • Why PIRATA? • Why DODS/OPeNDAP? • Why Python? • Future work. Guilherme P. Castelão <guilherme@oceanografia.org> Roberto A. F. De Almeida <roberto@oceanografia.org> http://opendap.oceanografia.org

2. Why PIRATA? Or better: what is PIRATA? • PIlot Research moored Array in the Tropical Atlantic (PIRATA) maintains a set of 12 ATLAS moorings in the Tropical Atlantic. • Sensors measure sub-surface temperature and salinity from 1 to 500 m depth, near surface temperature and humidity of the air, incident short wave radiation, direction and magnitude of the wind and rainfall. • Deployments started on late 1997, composing a daily mean time series that covers now more than 6 years on most positions. • Ocean surface heat fluxes can be estimated by bulk formulae, whose dependencies are satisfied by the PIRATA dataset. • There is no precedent dataset with such characteristics at the considered region, which makes possible studies unfeasible until now. PIRATA data and products available through a DODS/OPeNDAP Python server – 2/9

3. Why PIRATA? Buoys' locations PIRATA data and products available through a DODS/OPeNDAP Python server – 3/9

4. Why DODS/OPeNDAP? If PIRATA is the question, DODS is the answer. • PIRATA data has been made available only on the Pacific Marine Environmental Laboratory (PMEL) web site through electronic forms. • It is an easy process to download one or a few datasets, but it's not efficient for more frequent or automatic updates. • DODS/OPeNDAP solves this problem in a way that is easy and transparent to the user. In the best-case scenarios, no additional software or training is necessary. PIRATA data and products available through a DODS/OPeNDAP Python server – 4/9

5. Why DODS/OPeNDAP? What is DODS/OPeNDAP? • Fall 1993 - Initial workshop held at the URI to discuss the design and construction of a system that would: • facilitate access to oceanographic datasets • allow users to use their preferred analysis package • Project funded by NASA, NOAA, NOPP, with 23 partners. • DODS is an architectural framework to allow the user to easily access data over the network in a consistent fashion. DODS provides a basic structure to move data over the network. • OPeNDAP (Open source Project for a Network Data Access Protocol) is a non-profit corporation established to develop and promote software that facilitates access to data via the network. PIRATA data and products available through a DODS/OPeNDAP Python server – 5/9

6. Why DODS/OPeNDAP? What does this mean in practice? • Users can access data transparently through the network. • Ferret, e.g., is a DODS client: • set data “sst.nc” • set data “http://opendap.oceanografia.org/data/sst.nc“ • Data is downloaded automatically only for the sections in space-time that are actually used, being bandwidth and memory efficient. • There are plenty of DODS clients: Ferret, GrADS, IDL, Matlab, ncview, ..., and libraries available for C++, Java and Python. • DODS servers support a variety of data formats: NetCDF, HDF, GrADS, Matlab, etc. Data format doesn't matter for the client! PIRATA data and products available through a DODS/OPeNDAP Python server – 6/9

7. Why DODS/OPeNDAP? The PIRATA DODS server. • We developed a DODS server using the Python scripting language. The server has initial support for arbitrary NetCDF datasets, and is easily extendable through plugins. • PIRATA data is downloaded automatically from the PMEL web site and converted to NetCDF. Heat fluxes are then estimated from the data, and both are made available on the DODS server. • Initially only data for the buoy at 0°23°W is being processed: • http://opendap.oceanografia.org/data/pirata_data_0023.nc • http://opendap.oceanografia.org/data/pirata_flux_0023.nc PIRATA data and products available through a DODS/OPeNDAP Python server – 7/9

8. Why Python? Simple is better than complex. • Python is a free, open-source, general-purpose, high-level, interpreted, object-oriented, introspective, extensible, dynamic, robust, cross-plataform, collaborative programming language. • It was also created by a dutch, Guido van Rossum, who named it after the Monty Python's Flying Circus. • Python use for science is gradually increasing. It's easy to learn, fast to program and deploy language. It has a matricial framework similar to Matlab. • CDAT (Climate Data Analysis Tools) and PyClimate are two nice examples of Python being used for scientifics, climate related, applications. PIRATA data and products available through a DODS/OPeNDAP Python server – 8/9

9. Future Work How can the server be improved? • The obvious next step is to process the data from all 12 buoys, and make it available through the DODS server. • The intermediate step of storing the data and products as NetCDF is not really necessary, and adds to the already costy processing time. A planned solution is to store and serve the data using a free relational database (PostgreSQL). PIRATA data and products available through a DODS/OPeNDAP Python server – 9/9