AMSR-E SIPS Processing Status

1. AMSR-E SIPS Processing Status Kathryn Regner Information Technology and Systems Center at the University of Alabama in Huntsville kregner@itsc.uah.eduwww.itsc.uah.edu 256.961.7791 Image provided by Matt Smith

2. Outline What’s New at the SIPS Data Flow Review New System Configuration Update Processing Status Near real time Reprocessing AMSR Midori Field Campaign Support Important Reminders

3. SIPS Highlights • All processing (near real time and reprocessing) transitioned from ARIEL to AMSR cluster. • If you currently pull data from Ariel, please verify login to teamaqua@amsr1 • Reprocessing all products to ESDT Version 2 • product maturity codes are transitional or validated

4. NASAEDOS JAXA-EOC L1A Product Generation Ground Stations NSIDC DAAC Data Archive and Distribution AMSR-E Data Flow RBD & PDS Level-0 NASAFDS GBAD Level-1 Level-1 PO.DAAC Level-1 SIPS-RSS TLSCF Level-2 and Level-3 L2A algorithm L2A Product Generation Science Software I&T L2 and L3 algorithms Product Delivery Record Server (PDRS) SIPS-GHCC L2B and L3 Algorithms Algorithm Teams L2B and L3 Product Generation (science, metadata, qa, ph, subsets, browse) Rain Snow Data Ocean Sea Ice Land

5. New System Status (1) Dell hardware configuration is complete • Storage Area Network (SAN) sized to hold all Level-2A products from beginning of mission through 2010 • evaluating procurement options for next increment • Cluster Suite and Global File System enable much greater flexibility for processing streams of data.

6. New System Status (2) Software configuration • Recently upgraded operating system to Red Hat Enterprise Linux 5; minor software adjustments were required • All processing automation scripts were ported to Linux and tested with science software • QA files being sent to web server on “weather” • All algorithms are ESDT Version Collection 2

7. Dell PowerEdge 2850 2 x 3.4 GHz Processors 4 GB RAM, RHEL 5 AMSR 1-4 8 TB Dell EMC CX300 Storage Area Network Array Forward Processing “late” processing Reprocessing Reprocessing Development, Integration & Test Restricted FTP Server NASA Network (100 Mb/sec) 2 GB/sec Fibre Channel Switch Near line storage of all Level-2A and most recent Level-2B & Level-3 SIPS-GHCC HardwareOperational Configuration Multiple processing environments across four machines results in much greater flexibility for processing and reprocessing.

8. Near Real Time Ingest from RSS Level-2A Brightness Temperatures Level-2A Brightness Temperatures (with Metadata and QA files) Product Generation Scheduling ------------- ~ 29 times per day L2B Ocean (1/2-orbit) L2B Rain (1/2-orbit) L2B Land (1/2-orbit) L3 Ocean (Daily) L3 Sea Ice (Daily, 3 Res) L3 Snow (Daily) L3 Land (Daily) Once per day L3 Snow (Pentad) Once every five days L3 Ocean (Weekly) Once per week L3 Ocean (Monthly) L3 Snow (Monthly) L3 Rain (Monthly) Once per month Software Architecture(Routine Processing )

9. Algorithm Versionsand Product Maturity Codes Algorithm Short Name Current Versions (July 2008) L2A Tb L2B Land L2B Ocean L2B Rain L3 Land L3 Ocean L3 Rain L3 Sea Ice L3 Snow V09 T08 V06 V10 T05 V03 V06 T08 T08 Product Maturity Codes T=transitional V=validated

10. Forward Processing Routine forward processing is running very smoothly automated to run 24 x 7, unattended The L2A files typically arrive at GHCC 10-12 hours after observation Routinely dealing with the occasional straggler or replacement Level-2A files, which requires regeneration of the composite products, replacement files being sent to the DAAC for archive

11. Reprocessing Status Reprocessing standard Level-2B products began in March, using validated (V09) Level-2A files generated at RSS and ported versions of Level-2B algorithms Achieving rates of 11-12x standard Level-3 products (land, ocean, rain and snow) began in May using validated (V09) Level-2A files generated at RSS, reprocessed Level-2B and ported versions of Level-3 algorithms Achieving rates of ~25x Standard Level-3 Sea Ice began last week (algorithm updated to remove sea ice temperature parameter) Achieving rates >25x The reprocessed data is available to the science and validation teams on amsr1 and for public distribution at the NSIDC DAAC.

12. 2002 2003 2004 L2B products with L2A V08 2002 2003 2004 2005 2006 L2B products with L2A V09 ‘04 ‘02 ‘03 L3 Land, Ocean, Rain, Snow products with L2A V09 and L2B ‘02 ‘04 L3 Sea Ice products with L2A V09 Reprocessing Timeline 12/07 01/08 02/08 03/08 04/08 05/08 06/08 07/08 08/08 09/08 10/08 11/08 Available at the DAAC In progress at the SIPS

13. AMSR-Midori Transferred the 7 months of AMSR-Midori Level-2A products from RSS to GHCC in Jan 08 • Transferred a few new test files last month for TLSCF metadata/qa development and science software integration and test • SIPS will begin transferring full dataset to replace previous version in the near future

14. Field Campaign Support Real Time Mission Monitor - interactive field experiment asset management and data visualization tool http://rtmm.nsstc.nasa.gov/ Near Real Time AMSR-E data requested to assist with flight planning during three field campaigns this year • Snow depth product for ARCTAS (Arctic Research of the Composition of the Troposphere from Aircraft and Satellites) spring and summer http://www.espo.nasa.gov/arctas/ • working with sea ice team to supply data in support of an Arctic experiment in August. “The near real-time AMSR-E data helped us immensely with our flight planning during our last field campaign in 2006.” - Don Cavalieri

15. Important Reminders • Products Retention Plan • Original Hardware • Ariel to be decommissioned soon • Teamaqua Login Information

16. SIPS-GHCC Products Retention Plan All latest version Level-2A files are kept on the Storage Area Network to facilitate reprocessing. Approximately six weeks of the latest Level-2B and Level-3 daily science files are kept on line for science team QC and to facilitate product regeneration due to late arriving L2A files The 5-day, weekly and monthly science files are kept on line for approximately 6 months

17. sgi 2100 sgi 2100 Ariel Melody Original SIPS-GHCC HardwareSoon to be Decommissioned SGI Origin 2100 4 x 350Mhz Processors 2GB RAM SGI Origin 2100 4 x 250Mhz Processors 2GB RAM 2 TB RAID file system Network File System Interface Fibre Channel Interface Restricted FTP Server • Used for: • Reprocessing • Special Processing • Development • Integration & test • Used for: • Routine processing • Late L2A processing On line storage of most recent six weeks of Level-2 and Level-3 daily products for - transfer to DAAC - science team quality control

18. teamaqua Login Update sftp teamaqua@amsr1.nsstc.nasa.gov password: same as before (call me if you need it) Directory structure is a little different: • cd ops/science/data (for near real time) • cd repro/science/data (for reprocessed)