A family of schemes has been devised for organizing the output of an algorithm for predictive data compression of hyperspectral imagery so as to allow efficient parallelization in both the compressor and decompressor. In these schemes, the compressor performs a number of iterations, during each of which a portion of the data is compressed via parallel threads operating on independent portions of the data. The general idea is that for each iteration it is predetermined how much compressed data will be produced from each thread.
A simple version of this technique is applicable when the image is divided into “pieces” that are compressed independently. As an example, for a compressor that does not make use of interband correlation, a piece could be defined to be an individual spectral band, or a fixed number of bands. In the technique, the compressed output for a piece is comprised of multiple “chunks.” The concatenated chunks for a given piece form the compressed output for the piece. Most of the compressed image is produced in multiple iterations, where during a given iteration, one chunk is produced for each piece. Prior to the start of an iteration, chunk sizes are calculated for each piece. The chunks can be produced or decompressed in parallel. It is noted that it is not specified how much of the image data will go into a chunk, and in fact a chunk may contain incomplete portions of encoded samples (at the chunk’s start or end). The compressor iterates the process of deciding on chunk sizes and producing chunks for each piece of the requested size, until compression of each piece is almost finished. At that point, the remainder of the pieces is compressed serially without a target chunk size.
Typically, the chunk size calculation should seek to balance the progress through each piece, i.e., to leave equal numbers of samples remaining in each piece; a suggested procedure has this aim. A key requirement on the chunk size calculation is that reasonable chunk sizes must be decided on based only on information from the compressed data available at a given point in the process. Similarly, from previous data, it must be possible to evaluate when to switch from the parallel chunk compression to the serial process that completes compression of each piece.
A more general technique accommodates pieces that are not compressed independently, allowing compressors such as the Fast Lossless (FL) to more fully exploit dependencies between spectral bands, which generally allows a higher compression factor to be achieved.
This work was done by Matthew A. Klimesh and Aaron B. Kiely of Caltech for NASA’s Jet Propulsion Laboratory. NPO-48521
This Brief includes a Technical Support Package (TSP).
Organizing Compression of Hyperspectral Imagery to Allow Efficient Parallel Decompression
(reference NPO-48521) is currently available for download from the TSP library.
Don't have an account?
Overview
The document titled "Organizing Compression of Hyperspectral Imagery to Allow Efficient Parallel Decompression" by Matthew Klimesh and Aaron Kiely from NASA's Jet Propulsion Laboratory discusses innovative methods for compressing hyperspectral imagery. The primary focus is on predictive compression techniques that reduce the data burden on downlink resources and storage systems, particularly for images acquired by spacecraft or aircraft.
The authors introduce the Fast Lossless (FL) compressor, which is central to their approach and serves as the basis for the emerging Consultative Committee for Space Data Systems (CCSDS) Lossless Multispectral & Hyperspectral Image Compression standard. The FL compression method operates by estimating individual sample values from previous samples and encoding the differences, which forms the bulk of the compressed output. However, the compression process within a spectral band is inherently serial, as the predictor state must be updated with each sample.
To address the limitations of serial processing, the authors propose organizing the output of the compression algorithm to facilitate parallel implementations. This organization allows different threads to compress separate spectral bands simultaneously, enhancing efficiency. The document outlines a general procedure that reduces the reliance on serial mode, thereby increasing the effective parallelization factor and minimizing buffer requirements during compression and decompression.
The authors also discuss the challenges associated with indicating the sizes or starting points of compressed bands, which can lead to redundant information and increased file sizes. They suggest interleaving data from different bands as a solution, although this requires careful management of length indicators to avoid significant overhead.
The document includes schematic depictions of the proposed organization methods, illustrating how hyperspectral images can be divided into pieces for independent compression. Each piece can be processed in parallel, with chunk sizes calculated prior to iterations, allowing for efficient handling of the compressed output.
Overall, the research highlights the importance of efficient compression techniques for hyperspectral imagery, emphasizing the potential for broader technological, scientific, and commercial applications. The findings are particularly relevant for aerospace-related developments, showcasing how advancements in data compression can enhance the capabilities of imaging systems used in space exploration and other fields.
