Light from an object contains continuous various colors, the spectrum of light, that result from the interaction between light and the object. Spectral measurement is thus the basis of remote sensing, allowing for highly accurate material analysis and image recognition. Although the world is full of colors, human being and standard color cameras receive light through their eyes/sensors and perceive it as only three primary colors of red (R), green (G), and blue (B). Hyperspectral (HS) imaging is a technology that splits and detects light into more colors than humans and color cameras can. The richer spectral information of HS image is promising for machine vision to provide more information than human eyes or color cameras in visual inspection of foods, industrial products, and so on.
Overview and Challenges of Hyperspectral Imaging
Since its first appearance in the 1980s, various types of HS imaging techniques have been developed. Commercially available HS imagers can be classified into three representative methods: spatial-scanning, spectral-scanning and snapshot. The spatial-scanning method, also known as the “push-broom scanner”, provides HS image with an extremely high spectral resolution of 1 nm or even subnm. One drawback is that it requires a scanning process in the spatial dimension, resulting in a low frame rate owing to a long shooting time. The spectral-scanning method provides HS image preserving the spatial resolution of the image sensor used; it naturally achieves a higher spatial resolution than that of other methods. Similar to the spatial-scanning method, it requires a scanning process in the spectral dimension, resulting in a low frame rate. The other is the snapshot method, in which the HS image is acquired through a pixel-sized bandpass filter built onto an image sensor. A very high frame rate is possible without scanning process, but it requires pixel convolution to acquire HS image, resulting in a lower spatial resolution. In addition to these drawbacks on the frame rate or spatial resolution, conventional approaches face the same physical limits in terms of sensitivity. There, each pixel of the image sensor detects a small portion of the incoming light split by dispersive optics; roughly speaking, the sensitivity is inversely proportional to the number of spectral bands (colors). Such limited performance results in poor usability and may be the reason why HS imaging is not yet widespread.
Concept of Compressed Sensing-Based HS Imaging
To break through the physical limits of sensitivity, compressed sensing (CS) is applied to HS imaging. CS is a technique to efficiently acquire and reconstruct signals from under-sampled measurements and has been applied in several fields such as medical inspection and deep space exploration. An example application in the field of optics includes a miniaturized spectroscopic device, overcoming the physical limitations of device size with CS. Here, CS is used to efficiently acquire the spatial and spectral 3D information of HS image using a 2D image sensor, enabling high sensitivity HS imaging. Compared with the conventional HS imaging approaches that detect one color per pixel, the developed method detects multiple colors per pixel. Furthermore, by randomly modulating the light intensity for each wavelength at each pixel, thereby spectrally and spatially encoding light, the spatial and spectral 3D information can be reconstructed from the captured 2D image. The effective sensitivity is then determined by the average transmittance resulting from the random modulation. The effective sensitivity ideally becomes 50 percent regardless of the number of spectral bands, as the average of 0–100 percent random modulation.
To correctly reconstruct the HS image from the captured 2D image, the incoming light must be randomly modulated in the spatial and spectral dimensions, i.e., spatially and spectrally random sampling. Spatial randomness can be evaluated from the histogram of transmission patterns at each wavelength, and spectral randomness can be evaluated by calculating the 2D correlation coefficient between transmission patterns (rij) at different wavelengths. Quantitatively, the reconstruction error due to the spatial randomness is less than 2 percent, same level as shot-noise, when the standard deviation (σ) divided by the average transmittance (μ) is higher than 0.1. For reconstruction errors due to spectral randomness, rij should be less than 0.9 to keep the reconstruction error below 2 percent.
Even if our CS-based approach is theoretically possible, the challenge is how to achieve both spatially and spectrally random light transmission in the system. In the present case, it was achieved by a coded mask designed as a random array of Fabry-Pérot filters. By optimizing the structure of the Fabry-Pérot filter at each pixel, the light transmission can be properly designed to obtain the required randomness. Considering the complexity of the fabrication process and the randomness of the light transmission, 64 different Fabry-Pérot filters are fabricated and arranged randomly.
Through the Fabry-Pérot filters, the image sensor detects spatially and spectrally encoded light and outputs a single monochrome image. Image reconstruction is then carried out, generating a number of spectral images corresponding to the number of wavelengths from the compressed image. This reconstruction process leads to more outputs than inputs leading to an underdetermined system problem is needed to be solved. Many algorithms and regularized terms have been studied to solve such an underdetermined system problem, and one set of them, two-step iterative shrinkage thresholding and total variation (TV), was applied in the present case.
Improvement of sensitivity is accomplished by using a Fabry-Pérot filter that randomly modulates the incoming light in the spatial and spectral dimensions. Specifically, the averaged transmittance of a developed filter is about 45 percent for the visible light. This sensitivity is about 10 times higher than that of conventional technology (light use efficiency less than 5 percent), making it the world’s highest sensitivity for HS imaging. For example, the developed technology can clearly capture an object under room illumination (550 lux), but it is difficult for the conventional system to recognize it. The developed HS imaging technology enables clear shooting without extremely bright lighting which has been required in conventional HS imaging.
Acquisition of spectral information was experimentally demonstrated by dividing the wavelength range from 450 nm to 650 nm into 20 wavelengths. As shown in an example using color samples the spectral information is correctly obtained via random modulation followed by image reconstruction. As a result of the image reconstruction, all pixels acquire spectral information, and the number of pixels is preserved in the CS-based HS imaging. However, this does not guarantee that each pixel is spatially separated. To experimentally obtain the effective spatial resolution, a ladder resolution chart is photographed and evaluated by calculating modulation transfer function (MTF) curves. The MTF curves show that the spatial resolution of CS-based HS imaging is comparable to that of RGB cameras with 3 dB contrast. With a mm/pixel conversion ratio of 1,150 pixels/130 mm, a spatial resolution of 3 cycles per mm indicates that 3 pixels are required to separate adjacent lines with 3 dB contrast. This result is reasonable because edge smoothing occurs in the CS-based HS imaging during the image reconstruction process, whereas the RGB camera with Bayer filter mosaic also loses spatial information due to 2 × 2 pixel convolution and inter-pixel crosstalk.
Together with the sensitivity and resolution, a high frame rate is an essential factor for high usability. A low frame rate makes it difficult to perform shooting procedures smoothly, for example, focusing, adjusting lighting conditions, and moving objects into proper positions. From a usability perspective, video rate operation (>30 fps) is an important goal for HS imaging. The operation speed of a camera is mainly limited by a shutter speed and image processing time. The CS-based HS imaging enables a fast shutter speed because of the high sensitivity. High-speed image processing is also achieved by using GPU parallel computing of the image reconstruction. As a result, 32 fps HS imaging was achieved with VGA (640 × 480 pixels) resolution.
The high usability of the CS-based HS imaging brings HS imaging in various scenarios, including not only industrial applications, but also consumer applications such as smartphones, drones, and Internet of Things (IoT) devices.
This article was written by Yako Motoki, Researcher, Panasonic Applied Materials Technology Center. For more information, contact