Technical lossless / near lossless data compression

The talk will start by defining what we mean by lossless and near-lossless compression in the context of a hyperspectral sounder such as IASI. Standard techniques for lossless file compression will be reviewed – including bzip2 and BUFR, both of which are widely used in EUMETCast. Next we will look at imagery compression, including some published studies of compression in the context of hyperspectral imagery from the AVIRIS aircraft sensor.

The main part of the talk will discuss compression of IASI data. A 2004 study by Tony Lee recommended the use of PC scores plus residuals, using noise-normalised simulated spectra, and claimed near-lossless compression to 23% of the volume of the 16-bit scaled radiances. Using real IASI level 1c and level 1b data I will compare various compression schemes. The BUFR compression used operationally for level 1c achieves data volumes 67% of the original. Lossless compression can reduce this to 44%, whilst near-lossless compression using Lee’s recommended method can reduce further to 30% (which is a little higher than Lee’s estimate). However, care must be taken not to degrade the information in the spectra – since the use of apodised radiances in the 1c datasets means that random noise is much lower for rapidly-varying spectral components (high optical path difference in the interferogram) than for slowly-varying components. The implications for future sensors, such as IASI-NG and MTG-IRS will be discussed.