1 / 43

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework. Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008. Outline . NOAA’s Integrated Ocean Observing System Data Integrated Framework Challenge

scout
Download Presentation

NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008

  2. Outline • NOAA’s Integrated Ocean Observing System • Data Integrated Framework • Challenge • Solution - DIF • Goals and Objectives • Approach • Coordination • Web Services • Status • Validation and Verification • Regional Implementation • Future Challenges

  3. NOAA IOOS … • Is modernizing the way NOAA collects, shares, and uses ocean information • Is increasing data interoperability across NOAA and the regions and efficiency of operations • Is expediting access to data for improved decision making • To achieve these objectives, the Program focuses its contributions in two main areas: • Developing the IOOS Data Integration Framework (NOAA DIF-to-National DMAC) • Managing the regional IOOS partnership

  4. NOAA IOOS Program Focus • Standards • Develop and maintain a process to document, coordinate, evaluate and accept data standards • Regions • Invest in Regional Associations efforts that contribute to the definition and implementation of IOOS • Data Integration Framework (DIF) • Conduct a limited scope implementation of IOOS within NOAA as a proof of principle and to gather lessons to inform the larger IOOS effort

  5. NOAAIOOS Data Integration Framework Systems Engineering & Standards Development Hurricane Intensity Model Temperature Salinity Sea Level Currents Color Coastal Inundation Model Harmful Algal Bloom Model Data Access & Exchange Data Standards Integrated Ecosystem Assessment Temperature Salinity Sea Level Currents Color External sources of 5 Core Variables (consistent with NOAA standards) Data Integration Framework (DIF)5-4-12 Months 0-12 Month 18 Month 24 Month 36 Test & Evaluation Benchmarked Product Improvements for Operational Use Integration of 5 IOOS Core Variables Integrated Variable Ingest for Select Data Products NOAA 5 Core Variables Hurricane Intensity Model • Integrated information services for NOAA programs • Identify observation gaps • Validated enhanced data products • NOAA’s Data Integration Framework Enhanced decision support through: • Product Enhancement • Test & Evaluation • Verification & Validation Systems Engineering: Coastal Inundation Model Harmful Algal Bloom Model Integrated Ecosystem Assessment NOAA MISSION OBJECTIVES REGIONAL COASTAL NEEDS • Future State: • Regional-coastal data integration for • Regional scale data and information products and services

  6. IOOS Functional Areas = Data Discovery, Access, Transport (encodings, metadata, catalog services, access services, conversion services) Observing Systems Data Management and Communications (DMAC) Modeling and Analysis = Customer (decision support tool, analysis tool, generic client) = Provider (observing system, data archive)

  7. Integration Challengemultiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

  8. Integration SolutionDIF content and transport standards/services

  9. (Graphic by i3 Aerospace Technologies Pty Ltd– used with permission) What is the Data Integrated Framework? Developing the IOOS Data Integration Framework • First spiral of IOOS development • Core IOOS project • Limited scope • DMAC functions • Common data-sharing infrastructure • Agreed-upon standards • Aggregation services • Web services • Implement at selected provider & customer sites • Focus on several core variables • Standardize on small number of services & encodings • Adopt, adapt, expand, develop • Evaluate in FY 2010

  10. NOAA IOOS Data Integrated Framework Goals and Objectives • Validate premise: data integration to measure improvements • System engineering approach • Functional Requirements from identified customers • Concept of Operations Document • Design Document in progress • Architectural principles • Service oriented architecture • Distributed services operated by data stewards • Adopt existing standards and specifications

  11. DocumentationSystems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 Concept of Operations Version 1.0 4/25/08 DIF As-Is Baseline Systems Document Version 1.0 9/19/07

  12. NOAA IOOS DIF Office of National Environmental Satellite, Data, and Information Services (NESDIS) CoastWatch National Climate Data Center(NCDC) National Ocean Data Center (NODC) National Geophysical Data Center (NGDC) National Coastal Data Development Center (NCDDC) Office of Satellite Data Processing and Distribution (OSDPD) National Marine Fisheries Service Southwest Fisheries Science Center (SWFSC) National Ocean Service Coastal Services Center (CSC) National Centers for Coastal Ocean Science (NCCOS) Center for Operational Oceanographic Products and Services (CO-OPS) National Weather Service National Data Buoy Center (NDBC) National Centers for Environmental Prediction (NCEP) Oceanic and Atmospheric Research Pacific Marine Environmental Laboratory (PMEL)

  13. Functional Requirements (completed) NESDIS/NCDDC NOS/CSC NWS/NDBC NMFS/SWFSC NESDIS/NGDC NWS/NCEP OAR/PMEL NOS/CO-OPS Data Standards (completed) NESDIS/NCDDC NESDIS/NGDC OAR/PMEL NOS/CSC CONOPS (completed) NESDIS/ORA NMFS/SWFSC NWS/NDBC NOS/CSDL NOS/CSC NWS/NCEP NWS/EMC Integrated Products Team Chair – Charles Alexander NOAA IOOS Program National Weather Service (6) National Environmental Satellite Data and Information Service (6 + 1) National Marine Fisheries Service (2) National Ocean Service (9 + 3) Oceanic and Atmospheric Research (1 + 1) Design NOS NWS NESDIS NOS NMFS Harmful Algal Blooms NOS/NCCOS NOS/CO-OPS NWS/NDBC Coastal Inundation NWS/TPC NOS/CO-OPS Web Services and Data Encodings Chair - Jeff de La Beaujardiere NOAA IOOS Program NESDIS (5) NWS (2) NOS (4) OAR (1) NMFS (1) RA’s (5) Image Matters METADATA NESDIS/NGDC NESDIS/NCDDC IOOS Data Integration Framework Coordination

  14. DIF Implementation Schedule Schema and Integration Final Report May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb 2008 2009 2010 OSMC – Observing System Monitoring Center XML – Entensible Markup Language DCS – Data Content Standard WMS – Web Map Standard 10/31/2008 Integrated PORTS Currents (profile) CO-OPS 8/18/2008 Integrated Sea Level Temperature, Salinity, Winds, & Waves CO-OPS & NDBC 12/31/2008 PORTS Survey Currents (profile) CO-OPS Global Profile Obs OSMC 6/12/2008 Integrated Currents CO-OPS & NDBC 9/25/2008 Integrated Color CoastWatch 11/30/2008 WMS Services Available NDBC 6/26/2008 DCS Color NCDDC 9/30/2008 Integrated HF Radar NDBC Global and Coastal Point Obs OSMC 5/12/2008 Deliver XML Schema NOAA IOOS Program May Jun Jul Aug Sep Oct Nov Dec Schema and Integration Developing the National IOOS Data Integration Framework

  15. IOOS Data Integration Framework Data Sources Gridded Data Imagery Data KULA01 CWAO 051800 CREX++ T000101 A008 D09040++ 71 917 EUREKA 7598 -08593 00010 18 1998 04 29 23 18 061 019 //// //// 375 0082 0000 400 10137 030 0000 200 10000 030 0001 002 09687 037 0002 002 09366 033 0004 002 08831 037 0005 200 08500 036 0007 002 08013 043 0007 002 07881 047 0008 002 07646 037 0009 002 07442 042 0011 200 07000 031 0012 002 06849 027 0013 002 06710 036 0015 002 06291 029 0022 200 05000 028 0025 002 04557 027 0029 002 04065 024 0029 200 04000 020 0032 002 03626 025 0038 002 03000 020 0040 002 02890 021 0040 002 02829 065 0041 002 02726 105 0043 002 02576 118 0044 200 02500 135 0048 002 02218 165 0049 002 02147 161 0050 002 02104 171 0051 002 02031 153 0051 002 02010 159 0051 200 02000 171 0052 002 01941 188 0054 002 01854 198 0056 002 01744 187 0056 002 01717 194 0057 002 01683 191 0058 002 01640 161 0058 002 01623 159 0059 002 01585 168 0059 002 01576 185 0060 002 01545 197 0061 002 01500 202 In-Situ Data Need for three services! Open Geospatial Consortium (OSG) provides them.

  16. Recommended Web Services and Data Encodings Data Type Web Service Encoding In-situ data (buoys, piers, towed sensors) OGC Sensor Observation Service (SOS) XML based on OGC Observations and Measurements (O&M) Gridded data (model outputs, satellite) OpenDAP and/or OGC Web Coverage Service (WCS) NetCDF using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc.-possibly with standardized styles OGC: Open Geospatial Consortium XML: Extensible Markup Language OPeNDAP: Open-source Project for a Network Data Access Protocol NetCDF: Network Common Data Format GeoTIFF: Geospatial Tagged Image File Format PNG: Portable Network Graphics

  17. XML Encoding of In-Situ Data(SOS GetObservation Result) XML Extensible Markup LanguageGeneric method for structuring text data OGC GMLOpen Geospatial Consortium Geography Markup LanguageXML that can represent any geospatial feature specializedby OGC O&MObservations and Measurements ModelGML that describes the act of measuring real-world phenomena and the result of the measurement specializedby Data providers currently beta-testing this specification(http://www.csc.noaa.gov/ioos/schema/IOOS-DIF/)

  18. IOOS Data Integration Framework Sensor Observation Service NDBC is implementing SOS on a map interface page [ http://sdf.ndbc.noaa.gov/] Data may be retrieved as: Comma separated values (CSV) NetCDF XML Data available: Sea level Currents Salinity Sea Temperature Winds & Waves

  19. IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

  20. DIF Data Provider Status Data Integrated from One Provider Web Coverage Service & OPeNDAP Sensor Observation Service NDBC Data Content & Encoding Standards High-Frequency Radar (HFR) surface currents NDBC DAC- buoys, C-MAN, VOS, NOS NERRS, etc. IOOS DAC- Regional observations TAO DAC- Tropical Atmosphere Ocean DART DAC- Deep-Ocean Assessment and Reporting of Tsunamis NDBC Data Assembly Centers (DACs) User Requests Data Integrated from One Provider Sensor Observation Service CO-OPS Data Content & Encoding Standards National Water Level Observation Network PORTS® Web Coverage Service & OPeNDAP Data Integrated from One Provider CoastWatch Data Content & Encoding Standards IOOS Variables: temperature, salinity, currents,water level, winds, waves, ocean color (chlorophyll) Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS) Expected by end CY2008

  21. Additional Providers & Services SOS Single User Request WCS/OPeNDAP NDBC SOS CO-OPS Registry/Catalog Service WCS/OPeNDAP CoastWatch SOS Data Archives(NODC?) Data Integrated from Multiple Providers SOS Region 1 WCS/OPeNDAP Data Integration Service Region 2 WCS/OPeNDAP Agency 1 IOOS Variables + other variables t.b.d. SOS Agency 2

  22. Verificationand Validation • Service Function • Valid HTTP response on success and error • Server load test • Controlling Documents • OGC SOS Specification • OGC O&M Specification • Abstract Data Content Standard • XML schema • XML dictionaries • Feature Type Record Definitions • Service Content • XML is well-formed (obeys generic XML rules) • XML is valid (obeys schema rules) • XML matches Data Record Definition • Service Operations • Service can receive data queries • Service can send data queries • Data Validation • Data returned matches query parameters • Data complies with Abstract Data Content Standard • Data is easily combined/integrated • expressions of geographic position are consistent • expressions of temporal position are consistent • expressions of measured parameters are consistent

  23. DIF Service Layers NOAA DSTs COTS/ GIS General Public Other Customers NOAA Regional Partners Other Data Providers Target for FY2010 Registry/Catalog Other Services Integration Translation Web Map Service Sensor Observation Service Access Services Web Coverage Service OpenDAP

  24. FY 08 FY 09 FY 10 – FY 14 Initial Concept Studies & Works of others Define capability requirements High - level Requirement Use Cases Conops Analyze the options and determine the optimal approach to fielding the capability Functional & Technical Requirements Resources for Alternatives Requirements Analysis of Alternatives Gather Data ( AoA ) For Existing & Planned DIF Development Effort Economic Resources Analysis of Preferred Existing & Planned Analysis of Alternative Approaches Initiatives Cost Capability Data Development Strategy RFIs We Are Here KDP - 1 Implement Selected Strategy Pathway to a National DMAC Developing the National IOOS Data Integration Framework

  25. NOAA IOOS DIF Accomplishments Developing the IOOS Data Integration Framework • 7 IOOS variables integrated by end of 2008 • Focus on major NOAA data providers for greatest impact • Example: 83% of NOAA’s ocean temperature integrated and accessible in initial DIF implementation • DIF Integrated Products Team (IPT) developing effective, cross-NOAA integration solution • DIF implementation software and protocols will be made available to regional partners • NDBC provides 39 Regional temperature observations in August 2008 • Regions to begin implementation in FY09

  26. Regional Implementation

  27. Regional Potential(GCOOS, SECOORA, Federal, Industry)

  28. NOAA IOOS Data Integrated Framework Future Challenges • Very short schedule driven by externals • NDBC and CO-OPS are breaking new ground • Limited IOOS-specific reference material • Simultaneous implementation, testing, revision • Sensor metadata descriptions (SensorML) not ready • Process needed for working with OGC, SSML, etc. to revise/harmonize/promote in FY09 • Need for better documentation, tools, converters, clients, reference implementation, test suites • Need more customers/users

  29. Questions? • http://ioos.noaa.gov

  30. Back-up Slides

  31. NOAA Coordination • Integrated Products Team 11 Chair, Charles Alexander – NOAA IOOS Program • National Weather Service • National Env. Satellite Data & Informtn. Service • National Marine Fisheries Service • National Ocean Service • Oceanic and Atmospheric Research Chris Taylor Walter Smith Hendrick Tolman Steve Baig Avichal Mehra Bill Burnett Ken Casey Kent Hughes Lewis McCulloch Julie Bosch Jennifer Fry Phil Keegstra Tess Brandon Ken McDonald Paul Digiacomo Ted Haberman Roy Mendelssohn Jim Sargent Dave Foley Katie Fisher John Ulmer Daniel Martin Andrea Hardy Rebecca Love Katie Fisher Shelley Tomlinson Whitley Saumweber Rich Patchen John Dandy Josh Pederson Jack Harlan Anne Ball (CSC/Ocean.US) Steve Hankin Gustavo Goni Derrick Snowden

  32. % of NOAA Integrated FY08 83% 100% FY09 (w/ funding) Under investigation Temperature Integration Build

  33. High-Level Functional Requirements Concept of Operations Implementation Decision (~late FY2010) National DMAC Development Path(based on NOAA, NASA DoD Decision Processes) 26 Initial Concept Studies & Work of Others Define requirements Use Cases Functional & Technical Requirements Analyze options and determine optimal approach Analysis of Alternatives (AoA) Industry RFI Economic Analysis of Preferred Alternative NOAA Data Management Projects Data Integration Framework (DIF) Capability Development Strategy Analysis of Existing Approaches NSF Ocean Observing Initiative Regional Coastal Ocean Observing Systems Federal SOA projects

  34. IOOS Data Integration FrameworkWeb Map Service (WMS) for Images of Data Map is a georeferenced “picture” of data GetCapabilities operation results in a “table of contents” in standardized format GetMap operation results in image of data customized according to: Variables of interest User-specified bounding box User-specified time Image size File format PNG, GetTIFF, JPEG, GIF May contain layers of data

  35. IOOS Data Integration FrameworkWeb Coverage Service for Gridded Data Coverage is an array of gridded data values Simplified viewpoint for this discussion, can be more complex GetCapabilities operation results in a “table of contents” GetCoverage operation results in a data file containing header and array(s) of numbers customized for: Variable of interest User-specified bounding box User-specified time File format netCDF, HDF, floating-point TIFF DescribeCoverage operation results in metadata about a specific dataset

  36. IOOS Data Integration FrameworkSensor Observation Service for in-situ Data Sampling feature discrete locations of measurements Point, vertical, or horizontal profile, trajectory … and time series or collections thereof GetCapabilities operation results in a “table of contents” GetObservation operation results in an XML data file containing observation values for desired: Variables of interest Bounding box or named geographic feature of interest or a single sensor Time DescribeSensor operation results in XML providing detailed information about a specific sensor

  37. IOOS Data Integration FrameworkSensor Observation Service Sensor Ocean Service (SOS) One of a family of standards and specifications that constitute the Open Geospatial Consortium (OGC), Inc. Sensor Web Enablement (SWE) activity An application programming interface (API) for providing sensor and observation data via extensible markup language (XML). Supports: GetCapabilities – This function returns metadata about this service DescribeSensor – The function returns detailed sensor characteristics GetObservation – This function returns observation data of the specified type for the stations within the specified station and date/time range. A bounding box will be added in the future. Currently, can get: Most recent observation Observation at a specific time Observation for a time range

  38. IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

  39. IOOS Data Integration FrameworkSensor Observation Service – GetObservation for ocean current vertical profile

  40. Integration Challengemultiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

  41. Integration SolutionDIF content and transport standards/services DIF content and transport standards/services

  42. Documentation:Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 Concept of Operations Version 1.0 4/25/08 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 16

  43. Data Integration Framework (DIF) NOAAIOOS Data Integration Framework Systems Engineering & Standards Development Hurricane Intensity Model Temperature Salinity Sea Level Currents Color Coastal Inundation Model Harmful Algal Bloom Model Data Access & Exchange Data Standards Integrated Ecosystem Assessment Temperature Salinity Sea Level Currents Color External sources of 5 Core Variables (consistent with NOAA standards) Months 0-12 Month 18 Month 24 Month 36 Test & Evaluation Benchmarked Product Improvements for Operational Use Integration of 5 IOOS Core Variables Integrated Variable Ingest for Select Data Products NOAA 5 Core Variables Hurricane Intensity Model • Integrated information services for NOAA programs • Identify observation gaps • Validated enhanced data products • NOAA’s Data Integration Framework Enhanced decision support through: • Product Enhancement • Test & Evaluation • Verification & Validation Systems Engineering: Coastal Inundation Model Harmful Algal Bloom Model Integrated Ecosystem Assessment NOAA MISSION OBJECTIVES REGIONAL COASTAL NEEDS • Future State: • Regional-coastal data integration for • Regional scale data and information products and services

More Related