Overview of NOAA/NESDIS GOES-R Hyperspectral Sounder Data Compression Study

1. Overview of NOAA/NESDIS GOES-R Hyperspectral Sounder Data Compression Study Bormin Huang, Allen Huang, Alok Ahuja Cooperative Institute for Meteorological Satellite Studies University of Wisconsin-Madison 4th MURI Workshop, April 27-28

2. What is hyperspectral sounding data? • It is generated from an interferometer (e.g. HIS, AERI, CrIS, • IASI) or a grating sounder (e.g. AIRS). • It consists of several thousand spectral channels that span the • infrared region on the order of one wavenumber or less • What is hyperspectral sounding data for? • to retrieve • - atmospheric temperature, water vapor, and trace gases profiles; • - cloud & aerosol properties, • - surface temperature, emissivities, etc., • to derive wind from radiance or retrieved water vapor fields, • for better weather and climate prediction. • Why does it require compression? • Unprecedented volume of 3D data that consists of one spectral • and two spatial dimensions (~10-100 GB per day) ; • Beneficial to efficient data transfer and archive. • What is new for the data compression society? • High correlation among remote disjoint channels due to the • absorption of the same absorbing gases. • Lossy compression needs retrieval impact studies, i.e. • interdisciplinary knowledge in data compression and • remote sensing is needed! • Why is lossless or near-lossless compression desired? • Physical retrieval of atmospheric temperature and • absorbing gases is a mathematically ill-posed problem, • i.e. sensitive to data error and noise!

3. Lossless Compression Study • 2D Wavelet-Based Compression Scheme JPEG2000: 2D IWT → Bitplane Coding → Entropy Coding • 3D Wavelet-Based Compression Schemes 3D IWT →3D EZW→ Entropy Coding 3D IWT →3D SPIHT→ Entropy Coding 3D IWT → 1D BWT → Entropy Coding • 2D Predictor-Based Compression Schemes CALIC: 2D Gradient-adjusted Prediction → Entropy Coding JPEG-LS: 2D Nonlinear Prediction → Entropy Coding

4. 2D Wavelet Transform Integer Wavelet Transform (Lifting Scheme)

5. Wavelet based Schemes • JPEG2000 • A new ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) compression standard. • Successor to the DCT (discrete cosine transform)-based JPEG algorithm. IWT with 3 stages (Taubman et. al. 2000)

6. 3D Wavelet Tree Coding 3D EZW: It uses the spatial hierarchical tree relationship of the wavelet transform coefficients for efficient compression. 3D SPIHT: Refinement of the EZW scheme that provides better compression while having faster encoding and decoding times. Parent-child interband relationship and locations for EZW and SPIHT coding

7. i nn nne nw n ne ww w ? j Predictor-Based Schemes • 2D Context-based Adaptive Lossless Image Codec (CALIC) • Among the nine proposals in the initial ISO/JPEG evaluation in July 1995, CALIC was ranked first. • It is considered the benchmark for lossless compression of continuous-tone images. Neighboring pixels used in prediction (Wu et. al. 1997) Schematic description of the CALIC encoder

8. c b d a x • 2D JPEG-LS • Published in 1999 as a lossless compression standard of the ISO/IEC. Neighborhood of JPEG-LS used in prediction Schematic description of the JPEG-LS encoder

9. Burrows Wheeler Transform • Block-sorting compression scheme [Burrows et al, 1994] • Rearranges the positions of the data such that the few distinct values under the same previous context are grouped together in position. tennessee*tennessee* ennessee*t          *tennessee nnessee*te          ssee*tenne nessee*ten          e*tennesse essee*tenn          nnessee*te ssee*tenne          nessee*ten see*tennes          essee*tenn ee*tenness          see*tennes e*tennesse          ee*tenness *tennessee          ennessee*t An example of the Burrows-Wheeler transform. bwt(tennessee*) = t*sennesee. The matrix on the right is obtained by sorting the rows of the left matrix in right-to-left lexicographic order. * denotes end of the data block and can be considered as the smallest symbol.

10. Ten selected AIRS granules on Sept. 6, 2002 AIRS radiance field at wavenumber 900.3cm-1 for the selected granules

11. AIRS radiance field at wavenumber 900.3cm-1 for the selected granules

12. Compression ratios of different algorithms for the 10 selected AIRS granules

13. Bias-Adjusted Reordering (BAR)* Scheme • for Data Preprocessing • Hyperspectral sounder data features strong correlations in disjoint spectral regions affected by the same type of absorbing gases at various altitudes. • The Bias-Adjusted Reordering (BAR) scheme is used for exploring the correlation among remote disjoint channels. • The technique can be used to improve the compression ratio of any existing scheme. • The BAR scheme paper is accepted to be published in Optical Engineering. • We are in the process of patent application.

14. Effect of the BAR scheme on various compression algorithms for the 10 selected AIRS granules

15. Summary • In support of the NOAA/NESDIS GOES-R data processing studies, we investigated lossless compression of 3D hyperspectral sounding data using wavelet-based schemes (3D EZW, 3D SPIHT, JPEG2000) and predictor-based schemes (CALIC, JPEG-LS). • The performance rank from best to worst in terms of compression ratios before the BAR scheme is given in the order of JPEG-LS, 3D SPIHT, JPEG2000, CALIC, BWT and 3D EZW. • The performance rank from best to worst in terms of compression ratios after the BAR scheme is given in the order of JPEG-LS, JPEG2000, CALIC, 3D SPIHT, BWT and 3D EZW. • To take advantage of the spectral correlations, we applied the BAR scheme to significantly improve the compression performance of all the compression algorithms. Acknowledgement: This research is supported by NOAA NESDIS OSD under grant NA07EC0676.