1 / 29

DATA MANAGEMENT UPDATE

DATA MANAGEMENT UPDATE. Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005. NAME Data Management Timeline. 2003. 2004. 2005. Complete Operations Summary (WWW). Finalize Operations Plan.

jared-kinney
Download Presentation

DATA MANAGEMENT UPDATE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DATA MANAGEMENT UPDATE Steve Williams, Linda Cully, and Scot Loehrer NCAR/Earth Observing Laboratory (EOL) NAME SWG (7.5) Meeting 28 October 2005

  2. NAME Data Management Timeline 2003 2004 2005 Complete Operations Summary (WWW) Finalize Operations Plan Operational Data Staged to CODIAC Finalize Data Management Plan Data Analysis Workshop 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 Final Data Staged Preliminary data staged Integrated Data Staged Draft DM Plan Field Ops and Data Collection Final Specs from PIs QC’d data from PIs OPS Planning Meeting Planning Meeting On-line Catalog Operational (CODIAC)

  3. http://www.joss.ucar.edu/name/dm

  4. NAME Data Flow Delayed Operational Data • Operational • Data Ingest • GTS • Satellite • Radar • Model • Special • Research • Data Ingest • Aircraft • Ships • Profiler • ISS NAME PI Data and Products • Distributed • Data Centers • CICESE • IMTA • NASA/DAACs • NASA/GSFC • NASA/JPL • NASA/LaRC • NOAA/ETL • NOAA/NCDC • NOAA/NODC • NOAA/NSSL • NSIDC • SMN • Univ. of AZ “Preliminary” Data Sets “Preliminary” Products NAME Data Archive Center UCAR (JOSS) Catalog Information NAME Online Field Catalog “Final” PI Research Data SMEX-04 Data Center (NSIDC)

  5. EOL Data Archives Data Sources Research Community EOL Interactive Data System metadata quality control special processing Archive and Distribution

  6. NAME DATA BASE STATUS(as of 26 October 2005) • 239 of 266 Data sets (90%) have been submitted or linked. • 11 Data sets are in progress • Field Catalog (w/reports and “browse” products) is available on-line • Precipitation (hourly and daily) and Upper Air “Composites” are under preparation • Data Access: http://www.joss.ucar.edu/ihop/archive/

  7. http://www.joss.ucar.edu/name/namap2/

  8. www.joss.ucar.edu/name/catalog/ .

  9. NAME FIELD CATALOG SAMPLE PRODUCTS

  10. NAME 2004 Field Catalog Statistics http://www.joss.ucar.edu/name/catalog/ • Reports/Summaries (Status, Mission, and Operations) • 3,785 documents/imagery (0.27 GB) • Research Platform Products (Aircraft, Surface, Upper Air, Ship) • 17,606 Imagery files (1.25 GB) • Operational Products (Satellite, Surface, Upper Air) • 367,092 Imagery files (17.83 GB) • Model Output Imagery (Analysis and Forecast Fields) • 99,387 imagery files (6.76 GB) • TOTALS: 487,870 Files (26.11 GB)

  11. Composite Data Sets at NCAR/EOL A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does NCAR/EOL develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

  12. “Composite” Data Sets Being Developed by NCAR/EOL for NAME • Surface Meteorological – Datasets being converted • Precipitation - Hourly, and Daily (June-Sept 2005) • Soundings - Highest Res & 5 hPa (June – Sept 2005) • Complete Station List for all Composites

  13. NCAR/EOL Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters e.g., u, v, dz/dt Gross Limit Checks Automated Checks QC Flags Error Files Vertical Consistency Checks Visual Examination QC Flags High-resolution and/or Interpolated Vertical Composite e.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

  14. NAME TIER-1 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

  15. NAME TIER-2 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

  16. NAME TIER-3 EOL UPPER AIR “COMPOSITE” DATA Fixed High Resolution (+) Pibal (o) R/V Altair (+) R/V Ulloa (+) R/V El Puma (+)

  17. NAME ARCHIVED SOUNDINGS BY SOURCE Belize - 140 soundings R/V Ulloa - 91 soundings R/V Altair - 154 soundings R/V El Puma - 32 tethersonde SMN - 2468 soundings ISS - 462 soundings Loreto GLASS - 176 soundingsNWS - 16079 soundings Phoenix - 360 soundings San Jose - 324 soundings Yuma - 938 soundings pibal - 3235 pibals Total of 24,459 soundings, pibals, & tethersondes to be included in the Final “composite” data set for Tier 3*. * Need NSSL Tezopaco and Pibal data to complete composite

  18. CSU UPPER AIR COMPOSITE DATASET (V2.0) (1 July - 15 August 2004) • Enhanced sites: • Number Type Number Estimate of percent • of sites of sondes number missing missing • 10 USA-NWS enhanced 1380 17 1.2 • 6 ISS, Loreto,2 ships 815 10 1.2 • 7 SMN enhanced 1057 30* 2.8 • 1 Yuma (4 sites comb.) 377 9 2.3 • 1 San Jose 173 12 6.9 • subtotal enhanced 3802 78 2.1 • Operational sites: • 33 USA-NWS operational 3030 41 1.3 • 8 SMN operational 402 57** 14.2 • 1 Belize 75 17 22.7 • subtotal 3507 115 3.3 • 67 Total 7309 193 2.6 • * 14 missed due to unpaid electric bill at Empalme • ** 11 missed due to broken generator at Isla Socorro

  19. Embedded Accumulative Precipitation

  20. “Preliminary” Tier-1 Hourly Precipitation (872 stations) ALERT (+ 405) Army Range (o 40) AZMET (o 27) ISS (* 3) MADIS (+ 84) METAR (o 18) NCDC Coop (* 58) NCEP (* 59) NERN/SMN/Sonora (+ 119) NMSU (+ 59)

  21. “Preliminary”Tier-2 Hourly Precipitation (3913 stations) ALERT (+ 414) Army Range (o 40) AZMET (o 28) ISS (* 3) MADIS (+ 891) METAR (o 327) NCDC Coop (* 736) NCEP (* 1163) NERN/SMN/Sonora (+ 182) NMSU (+ 129)

  22. “Preliminary”Tier-3 Hourly Precipitation (10328 stations) ALERT (+ 414) Army Range (o 43) AZMET (o 28) ISS (* 3) MADIS (+ 2855) METAR (o 1089) NCDC Coop (* 2328) NCEP (* 3249) NERN/SMN/Sonora (+ 190) NMSU (+ 129)

  23. “Preliminary”Tier-1 Daily Precipitation (2050 stations) GASIR (* 195) Hourly Stations (+ 872) NCDC Daily Coop (+ 208) NCEP (* 557) Simple Raingage (o 218)

  24. “Preliminary”Tier-2 Daily Precipitation (11955 stations) GASIR (* 1072) Hourly Stations (+ 3913) NCDC Daily Coop (+ 2544) NCEP (* 4208) Simple Raingage (o 218)

  25. “Preliminary”Tier-3 Daily Precipitation (30232 stations) GASIR (* 1106) Hourly Stations (+ 10328) NCDC Daily Coop (+ 7326) NCEP (* 11254) Simple Raingage (o 218)

  26. DATA ISSUES FOR DISCUSSION • Mexican Navy SEMAR Data approval letter received; Request to collaborate with data processing. • SMEX-04 Archives are being merged  Priorities? • Are there any datasets not included in NAME Master List? • Are there other “composites” that are needed? • Should U.S. and Mexican Streamflow data be “composited”? Period of Record? • Need re-organization (and cleanup) of NAME web pages. • Need to establish “cutoff” date for datasets to be included in the “composites”. • Others?

More Related