1 / 21

Envisat Validation Workshop Atmospheric Chemistry Validation Team

Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database. Terje.Krognes , tk@nilu.no Aasmund.Fahre.Vik, afv@nilu.no Norwegian Institute for Air Research (NILU). Presentation Outline. Data Centre Status

raiden
Download Presentation

Envisat Validation Workshop Atmospheric Chemistry Validation Team

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database Terje.Krognes, tk@nilu.no Aasmund.Fahre.Vik, afv@nilu.no Norwegian Institute for Air Research (NILU)

  2. PresentationOutline • Data Centre Status • Tools • Local end-user tools • WWW search tools • Commissioning Phase Activities

  3. nadir web DOCS DATA ORIGINATOR DATA table.dat ASC2HDF HDF2ASC FILES http or ftp DATABASE Error email OK? http SEARCH DATA USER ftp to zardoz Data flow • Signed protocol required for access to tools, docs and files • ECMWF data requires additional protocol

  4. spare Internet nadir http Storage Firewall zardoz ftp,telnet Data Centre Status (1/3): Hardware • The machines have been swapped between production and spare servers • The hardware has not been changed

  5. Data Centre Status (2/3): Software • Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal) • File server (zardoz) runs Solaris 7, as before • Nusphere MySQL • Apache 1.3.22 (1.3.14) • MySQL 3.23.44 (3.23.32) • PHP 4.0.6 (4.0.4pl1) • IDL 5.4, as before • HDF 4.1r3, as before • No re-boot on nadir since April 2002 • Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)

  6. Data Centre Status (3/3): Metadata • Metadata Guidelines updated by metadata board on 24 April 2002 • During commissioning phase, updates of various metadata table entries, without affecting structure: • Project-related information • Actual descriptors • Guidelines still applicable, but more clarifications to be added.

  7. New user tools • Major re-write of the ASC2HDF file generation and checking tool around launch • 6 smaller upgrades of the ASC2HDF package after launch • Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules • IDL-tool to generate HDF data files (without error checking): IDL2HDF • IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally • IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)

  8. New web functionality - ECMWF • ECMWF plots (NH 2,5º*2,5º and T106) • ECMWF 10-day back trajectory plots • T106 extractions to HDF files

  9. New web functionality - search and plot • Use of defaults • Facility to save and retrieve a search • Provision for downloading multiple files in tarball via web interface • Improved on-line plotting (also for multidimensional data) • Corrected display of comments to variables

  10. File searching (1/4) • Defaults are marked with *** • Page initially opens without defaults • Page may be reset or set to saved defaults bys selecting a previously saved query • Files may be filtered by a large number of metadata parameters

  11. File searching (2/4) • The default criteria page lists all criteria sets that this user has previously saved • Entries may easily be removed • The main search page will be loaded with the selected defaults when launched from this page • Defaults may include 4D space-time parameters and variable name wild-cards

  12. File searching (3/4) • The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page • Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists. • Variable name and variable mode accept multiple selections • 4D space-time criteria may be entered as a single point, or as a max/min box • All files that include or touch the 4D criteria are listed as matches

  13. File searching (4/4) • Files may be selected by posting date • Files updated after a given date may be selected • The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID • If no criteria are selected, the entire file list will be requested

  14. File List • The file list displays the current selection criteria • A single file may be downloaded immediately • Files may be marked and downloaded in a compressed tarball • Search criteria may be saved for later re-use (type a meaningful description before saving) • The Variables link brings up a variable list and other documentation contained in the file

  15. Variable List • The variable list page displays all parameters contained in a file • Variable name, mode and descriptor are displayed • The measurement unit and corresponding SI unit are displayed • All documentation fields attached to the file are displayed • Additional documentation fields attached to a single variable may be displayed through the Comments link • Variables may be selected to generate an on-line plot (next slide) • Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots

  16. Variable plotting (on-line browse)

  17. Administrative tools • Web tools to report uploads/downloads • Web tools to archive/retrieve outdated data files • Web tools to check and maintain index database consistency and integrity

  18. User activities • Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation • After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files • From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived). • From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included) • There are currently 4741 ACVT files open to users (22 active AOIDs) • There are currently 462 MAVT files open to users (5 active AOIDs)

  19. User activities: Up and Downloads The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz

  20. Nadir-team activities • 300+ interactions with users (of which 50% during Commissioning Phase) • 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase) • One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs • Maintained and improved shell scripts for automatic file checking and storage • Created additional IDL and FORTRAN tools • Performed extensive web program development • Performed user administration updates • Added metadata entries and updated the table.dat file 23 times since launch • Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web • Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)

  21. Metadata issues Although many quality checks are performed on the data upon submission to the database: • The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units • Some components and units may still be inconsistently named or mis-used Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.

More Related