1 / 24

HYCOM Data Service www.hycom.org/dataserver

HYCOM Data Service www.hycom.org/dataserver. Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI). An overview .

bambi
Download Presentation

HYCOM Data Service www.hycom.org/dataserver

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HYCOMDataServicewww.hycom.org/dataserver Ashwanth Srinivasan, (FSU) Steve Hankin (NOAA/PMEL) Contributors: Jon Callahan (Mazama Consulting) Roland Schweitzer (Weathertop Consulting), Ansley Manke (NOAA/PMEL) Jeremy Malczyk (UW/JISAO) Peter Cornillon (URI) An overview

  2. HYCOM Data Service: History and Current Status • Data Service was launched in 2002 in Miami and served HYCOM Outputs via FTP, LAS and OPeNDAP servers • Started Serving Near Real Time Atlantic data in 2003 • Hosted about 3 TB of Data until Dec, 2006 in Miami. • Jan -2007 – service moved to FSU and hosted on a 100 TB SAN • Currently serving approximately 60 TB of data

  3. New Hardware • 100 TB Fibre Channel SAN • Three 8 CPU machines • 32 GB RAM/machine • Software • Red Hat Linux • Red Hat cluster suite and Global File System • Apache Web Server • THREDDS • LAS Server • Vsftpd server

  4. Operational Enhancements • All components (LAS,OPENDAP,FTP) co-located in the same physical machine. • Subset of files are uncompressed for faster access • In memory, decompression by using RAM scratch disks • High-availability architecture – minimal downtime during updates

  5. Currently Available Datasets • Daily outputs from the real time Global HYCOM+NCODA analysis (Apr 2007 –current) • Global HYCOM+NCODA analysis (2004-2005 available now; 2005-2007 will be available shortly) • Four years of Global Non-Assimilative Simulation (2003-2007) • Near real-time 1/12 Atlantic Ocean prediction system output (June 2003 – Present) • Monthly mean 1/12 Pacific Ocean simulation output (1978-2003)

  6. Global HYCOM Data Serving Challenges • Very large data/file sizes – each day of output is 12-19 G – consists of 26 (15+9) 2d and 3d variables (4500 x 3298 x 32 pts) • Curvilinear, staggered and hybrid vertical coordinates makes analysis and visualization on large files resource intensive and time consuming • Need to handle model products in non-standard native HYCOM format and in standard format like NetCDF (B.Cs are needed on native grids and in both NetCDF and HYCOM IO formats)

  7. HYCOM Ocean Prediction SystemData Flow 50 G pulled to FSU ~ 2-3 hrs Conversion to NetCDF and re-gridding to Z levels Global run at NAVO daily – 5 day hindcast and 5 day forecast in curvilinear and hybrid coordinates THREDDS and LAS Update *Approximately 4-5 hrs of data processing daily

  8. Means of Data Access • THREDDS Provides a dataset catalog and OPeNDAP access to all datasets. • Live Access Server Visualization and downloads in different formats, batch access etc. • OPeNDAP enabled clients Choice of common applications or user written programs to access data • FTP NetCDF files for each variable

  9. HYCOM LAS: http://hycom.coaps.fsu.edu/las/ • Provides NetCDF data in both native HYCOM (staggered) and regular (non-staggered) grids • On-the-fly conversion to fixed Z levels (but limited to pre-configured Z levels) • tested HYCOM format IO as a product –but needs better integration • Full resolution global outputs not available currently via LAS

  10. Access to native coordinates and regridded fields

  11. Hybrid-Z regridding fixed-Z native-Z

  12. HYDAE model intercomparison

  13. Synthetic Data Used in the Experiments

  14. Truth and Forecasts on Day 50

  15. RMS Error in 1-Day Forecasts

  16. SSH 1-Day Forecast Error Distribution

  17. HYCOM specific Matlab tools from the OPeNDAP Developers • The Matlab GUI developed for the1/12o North Atlantic Model Runs has been significantly modified. • Access the GUI from: http://opendap.org/download/ml-toolbox.html • The OPeNDAP-IDL client is now robust and can be used to access HYCOM data (no GUI yet). • Many modifications to the netCDF-OPeNDAP libraries. • Server4 - supporting GridFTP is now available.

  18. The HYCOM Matlab GUI

  19. Access the OPeNDAP Matlab GUIs from the OPeNDAP download webpage http://opendap.org/download Or go directly to the GUI web page http://opendap.org/download/ml-toolbox.html

  20. FTP – AccessFTP://www.hycom.org/datasets

  21. Access Details for June 2007

  22. Community Collaborations • Working with John Caron at Unidata on testing the Forecast Aggregation server with large operational datasets • Building a HYCOM .[ab] format IO service provider (ISOP) to be incorporated into the JAVA NetCDF 2.2 library • Partnership with IOOS data management Caucus

  23. Other Addition/Improvements • Forcing data • Reference observational data sets • A completely redesigned website is being planned along the lines of GODAE server • Searchable mailing list, FAQ’s • Links to other data repositories

More Related