Hyperspectral imaging is an exciting and promising new technology. When combined with modern machine learning software and actuators such as robots, air-jets, or flippers, hyperspectral imaging has the capability to revolutionize industrial sorting, enabling machines to perform complex tasks previously limited to humans.
What Exactly Is It?
Hyperspectral imaging is measuring high-resolution spectral data at every pixel in a two-dimensional image. Standard cameras provide three spectral data points at each pixel: red, green, and blue (RGB). The viewer's brain inputs the mixture of these three and interprets them as a unique color. In a hyperspectral image, the materials’ reflectance spectrum is a continuous curve of wavelengths with hundreds of spectral data points at each pixel. It is really a form of spectroscopy. A standard spectrometer provides only one “pixel” per measurement — there is no imaging. “Multispectral” imaging refers to a camera with each pixel delivering a few spectral data points, typically between four and twelve. The term hyperspectral refers to many spectral data points producing a near continuous spectrum, typically hundreds of data points at each pixel. However, designing and manufacturing hyperspectral cameras is significantly more complex than for point spectrometers, as imaging all wavelengths across an entire image plane while minimizing distortions is a difficult optical problem.
Figure 1 shows a false-color RGB image from aerial hyperspectral data of a farm, along with reflectance spectra from three regions in the scene. Figure 2 shows the same spectral data, but reduced to a multispectral signal with four bands. These four bands are similar to what the early Landsat satellites measured. It is clear from Figures 1 and 2 that much more information is contained in the hyperspectral data, enabling precise and reliable classification of objects in a scene.
Where is the Technology Now?
Hyperspectral imaging is becoming popular for three primary reasons: the cameras are becoming inexpensive and operate at high speeds; compact computers provide sufficient computing capability to handle high data volumes; and modern machine learning algorithms can use hyperspectral data to accurately identify materials, even those that appear quite similar. The technology is advancing rapidly in both research and industrial applications. Its usefulness is based on the fact that different materials have different reflectance spectra.
Applications — Agriculture
In agriculture, different plant species have different spectra, as do the same plant species under different conditions. For example, hyperspectral data can be used to alert farmers to early signs of crop stress or noxious weed infestations, as well as to determine ripeness. In precision agriculture and environmental monitoring the technology is increasingly viable for commercial applications, and will expand as additional obstacles are overcome. Determining accurate relationships between spectral features and plant physiologies, for example, is a focus of much research worldwide. This will enable site-specific applications of herbicides, pesticides, and fertilizers. Ultimately, such targeted crop management techniques will allow farmers to use fewer chemicals more effectively and produce greater yields. Thus, there is strong institutional support for hyper-spectral technologies, especially in Asia where food supply and environmental protection are critical issues.
Applications — Industry
Because factory environments can be controlled better than outdoor farmlands and forests, hyperspectral imaging is gaining traction for industrial sorting applications. Conventional vision systems often fail to sort items that have similar colors or appearance. Two examples of difficult applications for traditional color sorting systems, are foods with similar-colored shells or peels, and materials whose composition is unrelated to their visible color, such as recycled plastics. When standard vision systems fail, these sorting tasks fall to humans, processes that are costly, slow, and prone to mistakes.
Not only can hyperspectral data be used to distinguish differences between similarly colored materials, they can even access information outside the visible range — both infrared and ultraviolet. Real-time machine learning algorithms can process the data in seconds and then transmit the information to actuators such as robotic arms, air-jets, and mechanical flippers to complete the sort.
While designing and manufacturing the cameras is technically challenging, producing reliable hyperspectral analysis software for industrial sorting is both a barrier and an opportunity. Because widely used simple algorithms exist alongside a wealth of knowledge about deep machine learning, hyperspectral machine vision software development is an exciting field.
Traditional methods of analyzing spectral imaging data use the concept of “vegetation indices” or more generally just “indices,” colloquially called “band math”. These indices are scalar quantities associated with each pixel, calculated via simple algebraic combinations of measured intensities of a few specific spectral bands. For example, a historically venerable vegetation index is the Normalized Difference Vegetation Index (NDVI), which dates from the early days of multispectral remote sensing. Given a full reflectance spectrum at each pixel, the value of NDVI can be calculated at each pixel as
where “NIR” is the reflectance at an infrared band (around 750 nm), “Red” is the reflectance of a red band (around 650 nm), and NDVI is a number between -1.0 and 1.0. The magnitude of NDVI is a rough indicator of plant health. NDVI was well suited to early multispectral cameras that recorded four broad-band data points at each pixel (usually blue, green, red, and near-infrared). This is what was measured in the early Landsat satellites.
NDVI can give, at best, a rough estimate of plant health. Since the advent of airborne spectrometers and hyperspectral sensors, many more precise indices have been invented, with new indices invented regularly. An example of a common vegetative index that measures chlorophyll, the Modified Chlorophyll Absorption Reflectance Index (MCARI), takes the form
while an index that measures anthocy-anins in plants, the Anthocyanin Reflectance Index #2 (ARI2) takes the form:
In these equations, ρx is the measured reflectance at wavelength x.
Figure 3 shows a map of MCARI for the farmland in Figure 1. Bright locations in the map indicate high values of MCARI, while dark regions indicate low values. The interpretation of MCARI is that higher values imply higher chlorophyll activity.
Originally developed for agriculture and environmental monitoring, indices have now expanded to other fields, such as water quality, biology, and geology. In medicine, indices related to water and oxygen are used to determine properties of human muscle tissue during surgery.
Although indices are useful tools, they only incorporate a very small portion of the information available in broad continuous spectra. Hyperspectral data carries much more information than a few selected spectral bands and can yield all indices upon request. However, employing the entire spectrum along with statistical methods yields much more reliable classifications — this is exactly what is used in advanced machine vision applications.
Machine learning algorithms that utilize the entire spectrum along with rigorous statistics can more accurately classify plants and other materials. Techniques such as Support Vector Machines (SVM) and Convolutional Neural Networks (CNN) have proven successful in hyperspectral sorting applications. When combined with spatial recognition algorithms, hyperspectral machine vision can make for very smart computers with very sensitive eyes. This allows assembly-line machines to identify a wide range of materials, patterns, coatings, defects, and contaminants.
A Bright Future
Despite the clear advantage of providing more detailed data than conventional imaging systems, hyperspectral imaging is still in its infancy. Speed limitations for data acquisition, processing speed, and computational complexity have only recently been overcome for many real-world applications. Moreover, the market is understandably skeptical, as production facilities are not the place for incomplete technology. As the technology continues to improve, and as the market comes to understand the capabilities of hyperspectral imaging, previously intractable problems will be solved on large scales. Thus, the future of this exciting technology appears bright.