1 / 23

Thoughts on Data Compression:

Thoughts on Data Compression: Data compression of high spectral IR data is very important. Others think so too (ORA, ARL, etc). Data compression can have applications in a number of areas: downlink rebroadcast distribution archive

vcastillo
Download Presentation

Thoughts on Data Compression:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Thoughts on Data Compression: • Data compression of high spectral IR data is very important. Others think so too (ORA, ARL, etc). • Data compression can have applications in a number of areas: • downlink • rebroadcast • distribution • archive • The nature of the data must be understood to best compress and analyze the impact on products • Due to the natural redundancy in the spectra, lossy data compression should be considered (if needed). • UW/CIMSS is uniquely qualified to do the impact of lossy compression on a number of derived products (retrievals, clouds, etc).

  2. Hyperspectral IR Data Compression Where we were then… Where we are now… Where we are going…

  3. Hyperspectral IR Data Compression • Where we were then… • Guessed at lossy and lossless values for hyperspectral data, hoped for 10X and 2X • Only had experience with (time consuming) principle component work • No standard datasets for comparisons of various methods

  4. Hyperspectral IR Data Compression • Where we were then… • Knew there were several potentail uses: • Downlink • Rebroadcast • Archive (both short and long-term) • No optimized algorithms for hyperspectral IR sounder data • No studies done to demonstrate compression impacts on products

  5. Hyperspectral IR Data Compression • Where we are now… • Several optimized algorithms for hyperspectral IR sounder data • Begins of studies demonstrating compression impacts on products • Developed and have shared standard datasets for comparisons of various methods • Can compare to and more experience with principle component work

  6. Hyperspectral IR Data Compression • Where we are now… • Preliminary estimates for lossless values for hyperspectral data via a number of methods, showed that at least 2X should be very possible • Preliminary estimates for lossy values for hyperspectral data via a number of methods, showed that 5-6X should be very possible • Modified wavelet-based algorithms so we can compress on “irregular” (non-2^n) grids

  7. Hyperspectral IR Data Compression • Where we are now… • Understand more of how the compression ratio is a function of bit depth • Developed a new pre-processing step to improve the compression ratio of any existing scheme. • Began impact of lossy compressions on products

  8. Hyperspectral IR Data Compression • Where we are now… • Investigated several state-of-the art algorothms on hyperspectral data (ISO image compression standards – JPEG2000, JPEGLS) • Adopted and tested several lifting schemes: • 45 in 1-D • 9 of which were also converted to 3-D • Working with a new Cooperative Institute: CREST

  9. Hyperspectral Data Compression • Where we are going… • Continue to study the impact of lossy compressions on data and core (and new) products • Validate results on near global observations • Process in instrument “count-space”, not just the radiance space (eg, “raw data”) • Process on interferometric data, not just grating-type. • Reach out to others with similar interests/needs.

  10. Hyperspectral Data Compression • Where we are going… • Trade-off studies (CPU, memory, Compression Ratio, etc) to recommend what types of compression algorithms are strong candidates for GOES-R • Continue PCA work

  11. NOAA/NESDIS GOES-R Hyperspectral Sounder Data Compression Studies Bormin Huang, Allen Huang, Alok Ahuja and Kevin Baggett Cooperative Institute for Meteorological Satellite Studies University of Wisconsin-Madison March 2004 UW-Madison

  12. Satellite Satellite “Archive” 2:1 10:1 Users GRB SYSTEM “Downlink” 2:1? “Re-broadcast” 6:1? SOCC/ Backup Any GRB site “Data Pool” “Data pool” of full GOES (the compete level 1b data) would not be compressed, nor rebroadcast. Full GOES would be available for ground-based transfers.

  13. How might we move from 100 to 24 Mbps for a rebroadcast? • Maintain a "data pool" concept for land access and archive of • uncompressed sensor data • Users could then "test what is not being sent“ for their own application(s) • One method going from ~100 Mbps to 24 Mbps: • 10X for the 0.5 km visible band • 6X for the "1km” bands • 2X for the IR bands of the ABI • 6X for the HES-IR • Means approximately half of the band width would be used by the imager and half by the sounder; each image could be sent out

  14. Selected AIRS granules on Sept. 6, 2002 GOES will not see data from two of the most extreme (polar) datasets

  15. AIRS Radiance Field Five of the sample AIRS granules (IR window)

  16. AIRS Radiance Field Five of the sample AIRS granules (IR window)

  17. Lossless CompressionComparison of Methodologies JPEG-LS and JPEG-LS give the best compression ratios

  18. Lossless Compression JPEG-LS at various bit-depths The compression ratio is a function of the bit-depth

  19. Lossy Compression JPEG-LS RMS error A compression ratio of 5 is less than the instrument noise. A pre-processing step improves the performance.

  20. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

  21. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

  22. JPEG-LS Lossy Compression Brightness Temperature Error (K) for AIRS Granule 82 Longwave at 893 cm-1, Sept. 6, 2002 The noise does not appear to be spatially correlated.

  23. Summary • What has been done • Generic Data cubes prepared for HES compression studies • Various 3D wavelet transforms tested on real data • Various 3D wavelet tree coding developed and tested • Regular-sized vs. irregular-sized wavelet tree encoders/decoder have been implemented • Lossless and Lossy compression implemented • Predictor based encoders tested on real data • PVQ implemented and investigated on real data • Results 1. The performance rank from best to worst in terms on compression ratios is given in the order of JPEG-LS, JPEG2000,3D SPIHT, CALIC, BWT and 3D EZW. 2. JPEG-LS and JPEG2000 are applied for lossy compression and both have rms errors below the noise threshold.

More Related