1 / 20

Introduction to Exchange Formats First Data Management Training Workshop

Introduction to Exchange Formats First Data Management Training Workshop 12-17 February 2007, Oostende, Belgium Sissy IONA. Exchange Data Formats in SeaDataNet. CF-compliant NetCDF binary format (for gridded fields and multi-dimensional data types)

lea
Download Presentation

Introduction to Exchange Formats First Data Management Training Workshop

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to Exchange Formats First Data Management Training Workshop 12-17 February 2007, Oostende, Belgium Sissy IONA First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  2. Exchange Data Formats in SeaDataNet CF-compliant NetCDF binary format (for gridded fields and multi-dimensional data types) ODV ASCII spreadsheet format (for profiles, time series) MEDATLAS ASCII format (optional) First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  3. NetCDF data format • NetCDF (network Common Data Form) is an interface to create, access, and share array-oriented data in a form that is self-describing and portable. "Self-describing" means that a dataset includes information defining the data it contains. "Portable" means that the data in a dataset is represented in a form that can be accessed by computers with different ways of storing integers, characters, and floating-point numbers. Using the netCDF interface for creating new datasets makes the data portable. Using the netCDF interface in software for data access, management, analysis, and display can make the software more generally useful. • The netCDF software includes C, Fortran, C++, Java and other interfaces for accessing netCDF data. These libraries are available for many common computing platforms.” • (NetCDF Users Guide, V3.6.1, 2006, http://www.unidata.ucar.edu/software/netcdf) • Principally, designed for gridded data but extended to other observational data. • NetCDF software was developed at the Unidata Program Center in Boulder, Colorado. It is freeley available at the above UCAR’s website. First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  4. CF metadata Convention The Climate and Forecast (CF) Conventions is a standard designed to promote the processing and sharing of NetCDF files. The conventions locate data in space-time and as a function of other independent variables, facilitate processing and graphics. Identify data sufficiently to enable users of data from different sources to decide what is comparable and to distinguish variables in archives. The CF conventions generalize and extend the COARDS conventions. The CF conventions can be contained in other formats like XML. The CF conventions includes features that describe: Grid-cell boundaries Horizontal grids other than latitude-longitude Recording common statistical operations Standardised identification of physical quantities Non-spatiotemporal axes Climatological statistics Data compression Full documentation available on http://www.cgd.ucar.edu/cms/eaton/cf-metadata/ First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  5. NetCDF data file structure and main CF features A NetCDF data file has four basic components: Dimensions: gives the dimension information for the variables. Dimensions can be spatial, temporal or any other quantity. The dimensions can contain regularly or irregularly space steps and there is no limit in the number of dimensions. The ‘UNLIMITED’ dimension occurs when the dimension is ‘time’ and data are still being taken. Variables: contain information on the actual variables as well as on each dimension. Coordinates also contained in variables. The variables should be named using the CF Standard Names, otherwise ‘long_name’ attribute should be used to describe the variable. CF requires all variables to have units unless they contain dimensionless numbers. Global attributes: must include as much information as possible. CF conventions provide some basic “discovery” metadata in global attributes: title : description of what is in the dataset Institution : that produces the data source : the method of production of the original data - model, observational history : modifications to the original data references : publications or web references Comment : any other information Data: stores the data. CF conventions suggest the ‘_FillValue’ attribute for missing data and defined as the same type of the variable that replaces. First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  6. Example of gridded data PCMDI Sea Surface temperature data set in NetCDF format following the CF conventions: ------------------------------------------------------------------------------------------------------------------------------------------- netcdf tos_O1_2001-2002 { filename must have “.nc” extension dimensions: lon = 180 ; lat = 170 ; time = UNLIMITED ; // (24 currently)bnds = 2 ; number of verticesvariables: double lon(lon); coordinate variable of 1 dimensionlon:standard_name = "longitude”; legalvalue from CF Standard_Name table lon:long_name = "longitude" ; lon:units = "degrees_east”; compliant with CF/COARDS/Udunits lon:axis = "X”; legal value of axis are X,Y,Z and T lon:bounds = "lon_bnds“; represent the cell lon:original_units = "degrees_east" ;double lon_bnds(lon, bnds); define the extent of the cell and the grid point locationdouble lat(lat) ; lat:standard_name = "latitude" ; lat:long_name = "latitude" ; lat:units = "degrees_north" ; lat:axis = "Y" ; lat:bounds = "lat_bnds" ; lat:original_units = "degrees_north" ; double lat_bnds(lat, bnds) ; First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  7. double time(time) ; time:standard_name = "time" ; time:long_name = "time" ; time:units = "days since 2001-1-1”; the time units must contain a reference time time:axis = "T" ; time:calendar = "360_day“; defines the permitted values of year,month,day time:bounds = "time_bnds" ; time:original_units = "seconds since 2001-1-1" ; double time_bnds(time, bnds) ; float tos(time, lat, lon) ; tos:standard_name = "sea_surface_temperature" ; tos:long_name = "Sea Surface Temperature" ; tos:units = "K" ; tos:cell_methods = "time: mean (interval: 30 minutes)" ; defines the variation within the cells tos:_FillValue = 1.e+20f ; tos:missing_value = 1.e+20f ; tos:original_name = "sosstsst" ; tos:original_units = "degC" ; tos:history = " At 16:37:23 on 01/11/2005: CMOR altered the data in the following ways: added 2.73150E+02 to yield output units; Cyclical dimension was output starting at a different lon;" ; Example of gridded data First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  8. Example of gridded data // global attributes: :title = "IPSL model output prepared for IPCC Fourth Assessment SRES A2 experiment" ;:institution = "IPSL (Institut Pierre Simon Laplace, Paris, France)" ; :source = "IPSL-CM4_v1 (2003) : atmosphere : LMDZ (IPSL-CM4_IPCC, 96x71x19) ; ocean ORCA2 (ipsl_cm4_v1_8, 2x2L31); sea ice LIM (ipsl_cm4_v" ;:contact = "Sebastien Denvil, sebastien.denvil@ipsl.jussieu.fr" ; :project_id = "IPCC Fourth Assessment" ; :table_id = "Table O1 (13 November 2004)" ; :experiment_id = "SRES A2 experiment" ; :realization = 1 ; :cmor_version = 0.96f ; :Conventions = "CF-1.0" ; Identification of convention :history = "YYYY/MM/JJ: data generated; YYYY/MM/JJ+1 data transformed At 16:37:23 on 01/11/2005, CMOR rewrote data to comply with CF standards and IPCC Fourth Assessment requirements" ; :references = "Dufresne et al, Journal of Climate, 2015, vol XX, p 136" ; :comment = "Test drive" ;data:lon = 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63, 65, 67, 69, 71, 73, ........................................................................................................................ lon_bnds = -0, 2, 2, 4, 4, 6, ........................................................................................................................ First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  9. Example of multi-dimensional data OceanSITES ADCP time series in NetCDF format compliant with the CF/COARDS standard: ------------------------------------------------------------------------------------------------------------------------------------------- netcdf gebco2005_leg1_150K_1_fhv1 { dimensions: N_DATE_TIME = 390 ; N_LEVEL = 40 ; DATE_TIME = 14 ; CONST1 = 1 ; CONST2 = 2 ; STRING32 = 32 ; STRINGFILT = 30 ; STRINGFILTUSES = 1 ; variables: char REFERENCE_DATE_TIME(DATE_TIME) ; REFERENCE_DATE_TIME:long_name = "Date of reference for Julian days" ; REFERENCE_DATE_TIME:convention = "YYYYMMDDHHMISS" ; REFERENCE_DATE_TIME:_FillValue = " " ; double JULD(N_DATE_TIME) ; JULD:long_name = "Julian days relative to REFERENCE_DATE_TIME" ; JULD:_FillValue = -999999. ; float CAS_DATE_FLAG(N_DATE_TIME) ; CAS_DATE_FLAG:long_name = "Flag on ADCP current Juld" ; CAS_DATE_FLAG:valid_min = 0s ; CAS_DATE_FLAG:valid_max = 10s ; CAS_DATE_FLAG:_FillValue = -99999.f ; double JULD_ADCP(N_DATE_TIME) ; JULD_ADCP:long_name = "Julian days ADCP relative to REFERENCE_DATE_TIME" ; JULD_ADCP:_FillValue = -999999. ; char DATE_TIME_UTC(N_DATE_TIME, DATE_TIME) ; DATE_TIME_UTC:long_name = "ASCII gregorian date and time" ; DATE_TIME_UTC:convention = "YYYYMMDDHHMISS" ; DATE_TIME_UTC:_FillValue = " " ; float LATITUDE(N_DATE_TIME) ; LATITUDE:long_name = "Latitude of each location" ; LATITUDE:units = "degree_north" ; LATITUDE:valid_min = -90.f ; LATITUDE:valid_max = 90.f ; LATITUDE:_FillValue = -999999.f ; float LONGITUDE(N_DATE_TIME) ; LONGITUDE:long_name = "Longitude of each location" ; LONGITUDE:units = "degree_east" ; LONGITUDE:valid_min = -180.f ; LONGITUDE:valid_max = 180.f ; LONGITUDE:_FillValue = -999999.f ; First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  10. float UVEL_SHIP(N_DATE_TIME) ; UVEL_SHIP:units = "meter per second" ; UVEL_SHIP:long_name = "Eastward ship velocity" ; UVEL_SHIP:valid_min = -20.f ; UVEL_SHIP:valid_max = 20.f ; UVEL_SHIP:_FillValue = -999999.f ; float VVEL_SHIP(N_DATE_TIME) ; VVEL_SHIP:long_name = "Northward ship velocity" ; VVEL_SHIP:units = "meter per second" ; VVEL_SHIP:valid_min = -20.f ; VVEL_SHIP:valid_max = 20.f ; VVEL_SHIP:_FillValue = -999999.f ; float DEPH(N_LEVEL) ; DEPH:long_name = "Depth of bin center" ; DEPH:units = "meter" ; DEPH:valid_min = -12000.f ; DEPH:valid_max = 0.f ; DEPH:_FillValue = -999999.f ; float TEMP_ADCP(N_DATE_TIME) ; TEMP_ADCP:long_name = "ADCP transducer temperature" ; TEMP_ADCP:units = "degree_Celsius" ; TEMP_ADCP:valid_min = -5.f ; TEMP_ADCP:valid_max = 45.f ; TEMP_ADCP:_FillValue = -999999.f ; float HDG(N_DATE_TIME) ; HDG:long_name = "Ship heading" ; HDG:units = "degree" ; HDG:valid_min = -360.f ; HDG:valid_max = 360.f ; HDG:_FillValue = -999999.f ; float PTCH(N_DATE_TIME) ; PTCH:long_name = "Ship pitch" ; PTCH:units = "degree" ; PTCH:valid_min = -360.f ; PTCH:valid_max = 360.f ; PTCH:_FillValue = -999999.f ; float ROLL(N_DATE_TIME) ; ROLL:long_name = "Ship roll" ; ROLL:units = "degree" ; ROLL:valid_min = -360.f ; ROLL:valid_max = 360.f ; ROLL:_FillValue = -999999.f ; Example of multi-dimensional data First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  11. float CAS_CURRENT_FLAG(N_DATE_TIME, N_LEVEL) ; CAS_CURRENT_FLAG:long_name = "Flag on ADCP current (U,V,W)" ; CAS_CURRENT_FLAG:valid_min = 0s ; CAS_CURRENT_FLAG:valid_max = 10s ; CAS_CURRENT_FLAG:_FillValue = -99999.f ; float VVEL_ADCP(N_DATE_TIME, N_LEVEL) ; VVEL_ADCP:long_name = "Northward absolute ADCP current velocity" ; VVEL_ADCP:units = "meter per second" ; VVEL_ADCP:valid_min = -20.f ; VVEL_ADCP:valid_max = 20.f ; VVEL_ADCP:_FillValue = -999999.f ; float UVEL_ADCP(N_DATE_TIME, N_LEVEL) ; UVEL_ADCP:long_name = "Eastward absolute ADCP current velocity" ; UVEL_ADCP:units = "meter per second" ; UVEL_ADCP:valid_min = -20.f ; UVEL_ADCP:valid_max = 20.f ; UVEL_ADCP:_FillValue = -999999.f ; float WVEL_ADCP(N_DATE_TIME, N_LEVEL) ; WVEL_ADCP:long_name = "Upward absolute ADCP current velocity" ; WVEL_ADCP:units = "meter per second" ; WVEL_ADCP:valid_min = -20.f ; WVEL_ADCP:valid_max = 20.f ; WVEL_ADCP:_FillValue = -999999.f ; float EVEL_ADCP(N_DATE_TIME, N_LEVEL) ; EVEL_ADCP:long_name = "Absolute ADCP current velocity error" ; EVEL_ADCP:units = "meter per second" ; EVEL_ADCP:valid_min = -20.f ; EVEL_ADCP:valid_max = 20.f ; EVEL_ADCP:_FillValue = -999999.f ; float PGOOD_ADCP_B1(N_DATE_TIME, N_LEVEL) ; PGOOD_ADCP_B1:long_name = "Percent of good data before ensemble averaging" ; PGOOD_ADCP_B1:units = "percent" ; PGOOD_ADCP_B1:comment = "Number of good data = PGOOD * NB_ENS_AVE" ; PGOOD_ADCP_B1:valid_min = 0.f ; PGOOD_ADCP_B1:valid_max = 100.f ; PGOOD_ADCP_B1:_FillValue = -999999.f ; float PGOOD_ADCP_B2(N_DATE_TIME, N_LEVEL) ; PGOOD_ADCP_B2:long_name = "Percent of good data before ensemble averaging" ; PGOOD_ADCP_B2:units = "percent" ; PGOOD_ADCP_B2:comment = "Number of good data = PGOOD * NB_ENS_AVE" ; PGOOD_ADCP_B2:valid_min = 0.f ; PGOOD_ADCP_B2:valid_max = 100.f ; PGOOD_ADCP_B2:_FillValue = -999999.f ; ………………………………………………………………………………………………………………………….. Example of multi-dimensional data First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  12. …………………………………………………………………………………………………………………………..………………………………………………………………………………………………………………………….. Float VVEL_ADCP_CORTIDE(N_DATE_TIME, N_LEVEL) ; VVEL_ADCP_CORTIDE:units = "meter per second" ; VVEL_ADCP_CORTIDE:long_name = "Northward absolute velocity corrected for tide" ; VVEL_ADCP_CORTIDE:valid_min = -20.f ; VVEL_ADCP_CORTIDE:valid_max = 20.f ; VVEL_ADCP_CORTIDE:_FillValue = -999999.f ; char FILT_TYPE(STRINGFILT) ; FILT_TYPE:long_name = "Type of filtering" ; char FILT_FLAGS(STRINGFILTUSES) ; FILT_FLAGS:long_name = "List of flags used before filtering" ; // global attributes: :DATE_CREATION = "26-Oct-2005" ; :SOFTWARE = "CascadeVersion 5.3" ; :ADCP_CONSTRUCTOR = "RDI" ; :ADCP_TYPE = "Ocean Surveyor" ; :CONVENTIONS = "OceanSite dictionary" ; :CRUISE_NAME = "gebco2005" ; :PLATFORM_NUMBER = "FABB" ; :PLATFORM_NAME = "BEAUTEMPS-BEAUPRE" ; :DATE_UPDATE = "26-Oct-2005" ; :DATA_TYPE = "SADCP" ; :FORMAT_VERSION = "1.0" ; :NAVIGATION_REFERENCE = "GPS Navigation" ; :HEADING_REFERENCE = "GPS Heading" ; data: REFERENCE_DATE_TIME = "19500101000000" ; JULD = 20361.5601215279, 20361.5670601851, 20361.5739872684, 20361.5809317129, 20361.5878761574, 20361.5948379631, 20361.6017824076, 20361.6087268516, 20361.6156712961, 20361.6226157406, 20361.6295775464, 20361.6365219909, 20361.6434664354, 20361.6503935186, 20361.6573379631, 20361.6642997684, 20361.6712442129, 20361.6781886574, 20361.6851157406, 20361.6920601851, 20361.6990219909, 20361.7059664354, 20361.7129108794, ………………………………………………………………………………………………………………………….. Example of multi-dimensional data First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  13. ODV Generic Spreadsheet format ODV (Ocean Data View) is a simple, flexible, user friendly ASCII format. It consist of 3 parts: Comment lines Header line Data lines: up to 50 values The ODV QC flags are: 0=good, 1=unknown, 4=questionable, 8=bad It supports files in ASCII and binary NetCDF format: water profile data, benthic core profile data, Surface underway data, Moored instrument time series Developed by Schlitzer, R., http://odv.awi.de First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  14. Example of a CTD vertical profile // Data from: C:/sissy/ODV/tests/New_ODV_Collection (Feb/09/2007 11:59:08) // Extracted: Feb/09/2007 12:00:17 by Iona@SISSY Cruise;Station;Type;mon/day/yr;hh:mm;Longitude [degrees_east];Latitude [degrees_north];Bot. Depth [m];Pressure [decibar];QF;Temperature [degrees Celsius];QF;Salinity [psu];QF;Oxygen [ml/l];QF 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;8;0;19.734;0;38.703;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;9;0;19.683;0;38.702;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;10;0;19.62;0;38.713;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;11;0;19.442;0;38.717;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;12;0;19.052;0;38.728;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;13;0;18.662;0;38.739;0;'9999';8 36AE_GN36199402802;00010;C;6/1/1994;06:30;22.5;36.25;2386;14;0;18.401;0;38.808;0;'9999';8 …………………………………………… 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;1;0;20.924;0;38.556;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;2;0;20.871;0;38.547;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;3;0;20.888;0;38.536;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;4;0;20.836;0;38.541;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;5;0;20.772;0;38.541;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;6;0;20.674;0;38.546;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;7;0;20.597;0;38.539;0;'9999';8 36AE_GN36199402802;00020;C;6/1/1994;12:00;22.5;36;3100;8;0;20.527;0;38.538;0;'9999';8 …………………………………………… First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  15. Example of a Current Meter time series / Data from: C:/sissy/ODV/tests/New_ODV_Collection2 (Feb/09/2007 12:12:51) // Extracted: Feb/09/2007 12:14:15 by Iona@SISSY Cruise;;;Type;mon/day/yr;hh:mm;Longitude [degrees_east];Latitude [degrees_north];Bot. Depth [m];Time [years since 1900-01-01 00:00];QF;Year;QF;Month;QF;Day;QF;Time [hhmmss];QF;Current East Component [m/s];QF;Current North Component [m/s];QF;Horizontal Current Speed [m/s];QF;Direction rel. true North [degree];QF 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2228;1;1997;0;3;0;23;0;80000;0;0.009;0;-0.015;0;0.017;0;149.4;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2229;1;1997;0;3;0;23;0;83000;0;0.008;0;-0.012;0;0.014;0;146.3;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2229;1;1997;0;3;0;23;0;90000;0;0.013;0;-0.011;0;0.017;0;128.8;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.223;1;1997;0;3;0;23;0;93000;0;0.011;0;-0.009;0;0.014;0;128.8;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2231;1;1997;0;3;0;23;0;100000;0;0.011;0;-0.013;0;0.017;0;137.5;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2231;1;1997;0;3;0;23;0;103000;0;0;0;-0.014;0;0.014;0;180.6;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2232;1;1997;0;3;0;23;0;110000;0;0.005;0;-0.013;0;0.014;0;157.1;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2232;1;1997;0;3;0;23;0;113000;0;0.008;0;-0.012;0;0.014;0;146.9;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2233;1;1997;0;3;0;23;0;120000;0;0.01;0;-0.014;0;0.017;0;144.5;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2233;1;1997;0;3;0;23;0;123000;0;0.011;0;-0.008;0;0.014;0;125.6;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2234;1;1997;0;3;0;23;0;130000;0;0.007;0;-0.012;0;0.014;0;151.5;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2235;1;1997;0;3;0;23;0;133000;0;0.007;0;-0.012;0;0.014;0;148;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2235;1;1997;0;3;0;23;0;140000;0;0.013;0;-0.006;0;0.014;0;116.5;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2236;1;1997;0;3;0;23;0;143000;0;0.022;0;-0.007;0;0.023;0;107;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2236;1;1997;0;3;0;23;0;150000;0;0.027;0;-0.005;0;0.028;0;101.1;0 36AE_GN36199724603;M1N21;C;3/23/1997;08:00;24.6;40.0725;500;97.2237;1;1997;0;3;0;23;0;153000;0;0.03;0;-0.009;0;0.031;0;106;0 First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  16. MEDATLAS format MEDATLAS formatdesigned by the MEDATLAS and MODB consortia in conformity with the ICES/IOC GETADE recommendations in the frame of the EU MAST II programme. It is an auto-descriptive ASCII format. Main characteristics of MEDATLAS format: -facilitate the reading of the data (and not to optimize the data archiving neither to speed up the data processing). is independent of the computer. is flexible and accept (almost) any number of different parameters. keeps track of the history of the data including the data collection and the processing. allows the processing of each profile independently. Therefore the date, time and geographical co-ordinate must be reported on each profile header (and not in separate files) real (floating) numbers must remain in the same way as they have been transmitted, not reformatted into integers. The number of decimals must implicity indicate the accuracy of the measurements First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  17. Data and Metadata Structure A MEDATLAS file includes: - data from the same cruise - data measured with the same instrument (CTD, Bottle, Current Meter, etc) A MEDATLAS file consists of three parts: a cruise header based on the international ROSCOP information a station header including the cruise reference, the originator station reference within the cruise, date, location, list of observed parameters with units the data of the station (example of a time series data file) The sequence ‘station header + data records' is repeated for each profile. The archived parameters are the observed. Derived parameters (eg. density) are not archived. Detailed description of the format and the main fields in the “Medar-Medatlas Protocol, Part I: Exchange format and quality checks for observed profiles” First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  18. MEDATLAS format identifiers Cruise reference (unique): FI35199745003 (String of 13 Characters, No blanks, ‘0’ instead) FI                          data centre code 35                GF3 country code of the data source 1997                year of the beginning of the cruise 45003      assigned to the cruise by the data centre Station reference (unique): FI3519974500300011 (String of 18 Characters, No blanks, ‘0’ instead) FI35199745003 cruise reference 0001 station name 1 cast number First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  19. Reference codes Reference codes are used to describe the metadata: IOC/GF3 Country codes (http://www.ices.dk/ocean/codes/Country.htm) ICES Ship codes (http://www.ices.dk/ocean/codes/shipcodes.htm) Extended GF3 codes for parameters and units (UNESCO/IOC/IODE, 1987 : GF3 A General Formatting System for Geo-Referenced Data - Manual and Guides 17) IOC/ROSCOP codes for data types (http://www.ices.dk/ocean/roscop/par-cod.htm) Mediterranean and Black Seas region from International Hydrographic Bureau definitions (I.H.B.) (http://www.ices.dk/Ocean/codes/ihb.htm) MEDAR/MEDATLAS Data Centres and Confidentiality Codes (MEDAR/MEDATLAS protocol). Quality flags First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

  20. References/links “Climate and forecast (CF) metadata convention” by J. Gregory, Univ. of Reading, Met Office Hadley Centre, UK Coriolis data-center, Volume 2/2, V1.2, 2001 Jonathan Gregory, 2003, “The CF metadata standard” “Medar-Medatlas Protocol, Part I: Exchange format and quality checks for observed profiles”, V3, 2001 NetCDF Users Guide, V3.6.1, 2006 OceanSITES User’s Manual, V1.0, 2006 Schlitzer, R., Ocean Data View, http://odv.awi.de, 2006 http://badc.nerc.ac.uk - http://www.cgd.ucar.edu/cms/eaton/cf-metadata/ http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html http://www.ices.dk/ocean http://www.unidata.ucar.edu/software/netcdf First Data Management Training Workshop, 12-17 February, 2007, Oostende, Belgium

More Related