AIRS Meeting GSFC, February 1, 2002 ECS Data Pool

1. AIRS MeetingGSFC, February 1, 2002ECS Data Pool Gregory Leptoukh

2. SW dev SSI&T Hydrology Hydrology Hydrology Hydrology MODIS Atm. Dyn. Data Support Team Data Support Team Data Support Team Data Support Team Data Support Team Data Support Team AIRS Mission Support at the GDAAC Mission Support AIRS Mission Support SCIENCE Gregory Leptoukh

3. Outline • What is Data Pool? • How does Data Pool help the distribution? • What is the Data Pool project status? • WHOM for Data Pool • Future plans Gregory Leptoukh

4. EOS Data Pools Concept Gregory Leptoukh

5. Gregory Leptoukh

6. Data Pool Goals • Data Pool • Support increased electronic distribution • Reduce need to pull data from archive • Reduce need for order submissions • Give science and applications users timely, direct access to data, including machine access • Allow users to tailor their data views to more quickly locate the data they need Gregory Leptoukh

7. Data Pool goals, cont. • Spatial Subscription Server • Data Pool is populated by subscriptions • Subscriptions set product importance and expiration priority • Allow subscriptions to correctly qualify data on extended set of metadata attributes, especially spatial and temporal attributes • Increase subscription processing capacity • Design server to be extensible for new action types Gregory Leptoukh

8. Initial Data Pool Parameters • Data Pool Sizing • GSFC: 14 TB + (23 TB to help with data transfer to MODIS L2+ processing) • Contents • Data files • XML-based metadata files; consistent with BMGT DTD • Browse files in jpeg format (via hdf2jpg) • Directory Structure • /datapool/<mode>/<collectgrp>/<shortname.versionid>/<acq date> • e.g., /datapool/OPS/ASTT/AST_L1B.001/1999.12.31 Gregory Leptoukh

9. Gateways to Data Pool • Web: • GES DAAC Search&Order Interface (WHOM) – working on it • Earth Data Gateway (EDG) – no plans yet • ECS/Landover WHOM-like interface – delayed • Anonymous FTP: Gregory Leptoukh

10. GUIs and Utilities • Spatial Subscription Server GUI - allows operators to place qualified (spatially, temporally, or by parameter) subscriptions for: 1) email notification of ECS insert/delete/metadata update events ; or 2) distribution from the ECS archive • Data Pool Maintenance GUI - allows operators to set or change Data Pool configuration parameters, including which data types are eligible for insertion, and to monitor Data Pool insert processes • Update Granule Expiration utility - command line utility which allows operators to update the expiration date and optionally the retention priority of specific Data Pool granules • Cleanup utility - a command line utility which allows operators to cleanup expired granules from the Data Pool database and disks • Access Statistics utility - a command line utility which collects Data Pool access statistics from the FTP and Web Server logs, and summarizes this information by subscription ID in the Data Pool database • Data Pool User Web Pages - allows end users to navigate data pool contents, and browse and download data products Gregory Leptoukh

11. Current GDAAC DP status • Hardware on the floor • 14 TB of disk space mounted on REID • Spatial Subscription Server has been delivered • Data Pool insertion s/w + database to be delivered in February 2002 • WHOM-like web interface delivery delayed • GDAAC starts developing WHOM extension to Data Pool Gregory Leptoukh

12. Initial capabilities • Initial Data Pool capabilities: • 1 Gigabit Ethernet • One extra “X” distribution capability • Approximately 2 weeks of data • On-line data access via Web and FTP • DAACs control Data Pool contents and data residency • Initial Spatial Subscription Server capabilities: • Spatial, temporal and PSA qualifiers • Electronic (push and pull) data delivery Gregory Leptoukh

13. Data Pool and DAAC • DAAC does support the very DP idea • Data Pool management tools too powerful – it easy to delete important (or even all) data • Data Pool design was not coordinated with DAACs, and it would be difficult to use it as delivered. • DAAC Unique Extensions (DUE) needed • DAAC wants to use Data Pool to strengthen and optimize L1 data transfer to MODAPS Gregory Leptoukh

14. WHOM for Data Pool Utilize existing WHOM paradigm, structure and software: • Hierarchical Structure and Navigation: • simple point & click • flexible, alternative hierarchies (views) • Pages Generated from Templates: • consistent look and feel throughout interface • configuration management made easier due to content being separate from software • Modular Structure: • Easy to add new products • Fast search • Query Preview paradigm – no zero hits! • Convenient Calendar View • Geographical presentation of granule coverage • Attribute filtering Gregory Leptoukh

15. WHOM for Data Pool, cont. Several release approach: • Short term (2 months): have a usable prototype at the GDAAC • Slightly longer term (3 - 4 months): DP view prototype (in parallel) • Long term (6 months): have a version of WHOM with ECS archive and DP holdings integrated. Gregory Leptoukh

16. Future GDAAC Plans for DP • Port Level 1 browse to DP • Port Level 2+ browse from SIPS to DP • Tie user subscriptions with DP • Tie subsetting capability to DP • Tie data mining and similar activities to DP Gregory Leptoukh

17. Down The Road? Data-driven Processing and Distribution (internal and external) View Access User-specific Navigation Clients Data Providers Expanded Data Pool Data Items ECHO Data Producers Metadata Export Other Data Pools Data Services Data Service Developers OGC Access Hierarchical Storage Management (if required) Archive Gregory Leptoukh