More efficient satellite data transmission

Hyperspectral satellites collect imagery in hundreds of spectral bands simultaneously (over wavelengths that can range from the near-UV to the short-wave IR regime) and are capable of providing direct identification of surface materials. They are used in a wide diversity of remote-sensing applications, including geology, oceans, soils, vegetation, atmosphere, and snow/ice. They generate enormous data volumes (which may exceed the downlink capacity) and quickly exhaust the onboard storage capacity. To deal with this problem, we normally reduce the amount of onboard data by limiting the length and/or swath of image acquisition and reducing spatial and/or spectral resolution. Onboard lossy data compression, which discards some data to achieve higher compression, may mitigate this problem more effectively if we could control possible information losses due to compression.

Many lossy data-compression algorithms have been developed. However, these approaches distort hyperspectral imagery when the compressed data is reconstructed on the ground, since they were not designed to prevent information loss. We have, therefore, developed novel compression algorithms that restrict the errors introduced in the compression process below the floor of the original data's intrinsic noise. We call this ‘near-lossless’ compression, since it is expected to have negligible impact on the ultimate applications.

We have developed two vector-quantization (VQ)-based onboard data-compression techniques, successive-approximation multistage and hierarchical self-organizing cluster vector quantization (SAMVQ and HSOCVQ, respectively).^1,2 A VQ compression algorithm consists of two phases, a training phase, in which similar vectors in a sequence are grouped into partitions and each partition is assigned to a single representative vector (a ‘code vector’), and a coding phase, in which each vector is encoded by replacing it with the nearest code vector referenced by a simple partition index.

The SAMVQ compresses data in multiple stages using extremely small codebooks to successively approach a given threshold, while the HSOCVQ works by adaptively clustering vectors and splitting the clusters hierarchically so that each vector is encoded with an error less than a given threshold. Near-lossless compression is achieved when the threshold is set to smaller than the floor of the original data's intrinsic noise.

We have examined the near-lossless properties of the SAMVQ and HSOCVQ by comparing the compression errors with the intrinsic noise of the original data.³ Our experimental results show that compression errors are smaller than the intrinsic noise (see Figure 1). We have carried out a multidisciplinary user-acceptability study, which involved eleven users covering a wide range of application areas and a variety of hyperspectral sensors, to assess the impact of compression on Earth-observation applications. The users examined the compressed data qualitatively and quantitatively using well-understood data and predefined evaluation criteria and ranked the compressed data according to the impact on the applications using a double-blind-test approach. Most accepted the compressed data at a compression ratio between 10:1 and 50:1.⁴

Figure 1. Intrinsic noise of the original data, compression error (using SAMVQ at a compression ratio of 20:1), and intrinsic noise with compression error as a function of band image. SAMVQ: Successive-approximation multi-stage vector quantization. DN: Digital number (raw data after digitization).

We built two versions of hardware compressor prototypes that implement the SAMVQ and HSOCVQ techniques for onboard processing. The first was targeted for real-time application, while the second was for nonreal-time use (see Figure 2).⁵

Figure 2. Prototype compressor board (including four compression engines and a network switch, each of which uses a field-programmable gate-array chip).

The Consultative Committee for Space Data Systems (CCSDS)⁶ is developing new international standards for satellite multi- and hyperspectral data compression. The SAMVQ has been selected as a candidate. We are currently working with the CCSDS on developing the international standards. The preliminary evaluation results show that SAMVQ produces competitive rate-distortion performance on the CCSDS test images.⁷ Our next step will be to apply these techniques to a new Canadian mission—Polar Communication and Weather—and possibly to synthetic-aperture-radar satellite images.