Image processing algorithms can (clockwise, from top left) brighten or darken for dusk and dawn viewing; dehaze images for fog, smoke, and sand storm environments; clarify images with contrast enhancement that continuously adapts to changing brightness and contrast; and identify anomalous shapes and highlight details.
A new concept, the Any-Image-Anywhere (AIA) System combines image enhancing algorithms, a super-fast field-programmable gate array (FPGA) parallel processing platform, and a high-speed video switching matrix in a flexible, expandable, open architecture. The AIA System accommodates multiple video input streams and routes them to any combination of attached displays or network connections. Multiple image streams can be presented on an individual LCD display simultaneously: for example, four cameras could be supported using one as the primary image in full screen and three other streams as picture-in-picture (PIP) windows. Operators can turn image functions on or off, or swap the primary and PIP windows using a touchscreen user interface. The AIA System executes image enhancement algorithms in real time for live video surveillance feeds such as those applicable to UASs, and offers the performance, flexibility, and ruggedness necessary for mission-critical applications.

For example, in the field, the AIA System could be installed at a UAS Ground Control Station (GCS) to apply image enhancement and edge detection algorithms to incoming video streams. The image enhancement algorithms will bring out the detail from images degraded by poor visibility or atmospheric interference. The edge detection algorithm will identify anomalous shapes and highlight details for surveillance and bomb damage assessment (BDA).

The Origins of Image Enhancement

While real-time enhanced video capability is new, many of the algorithms involved have been routinely used for still image processing in software applications such as VITec Electronic Light Table (ELT) or Adobe Photoshop. These algorithms are computationally intensive, and applications like Photoshop often push the limits of the general-purpose platforms they run on just to post-process a single image. The ability to process FMV in real time requires an extraordinarily fast processor capable of processing multiple input streams simultaneously.

New generation FPGAs make it possible to meet the demanding performance requirements for enhanced FMV. The advantage of FPGAs compared to other processors is their unmatched capacity for parallel processing. FPGAs benefit from an arbitrary number of data paths and operations, up to the limit of the device capacity. Larger FPGA devices are capable of performing hundreds or even thousands of multiple operations simultaneously on data of varying widths. Even with lower internal clock frequencies compared to dedicated processors, FPGAs provide better performance due to the high degree of parallelization that is achievable.

As fast as FPGAs are, achieving the goal of zero latency required for real-time FMV requires some ingenuity. The trick is to calculate the necessary adjustment based on the first frame, but apply it to the following frame, and so on. Calculations for the next frame are done in parallel with processing of the current frame, so no latency is introduced. Because the differences between successive frames are small, the adjustments are still applicable.

Image processing algorithms are the key technology for improving FMV image clarity and usefulness. The AIA System hosts a collection of image enhancing algorithms. A large number of algorithms that run on other platforms can be ported to the AIA System as well. An open architecture system, all necessary interfaces for creating and installing algorithms are published. The architecture allows new algorithms to be installed without disrupting the system. In addition to image enhancement algorithms, there are also rectification algorithms that correct for camera angles; mosaic or stitching algorithms that combine multiple images into a single, unified picture; and encoding/decoding and compress/decompress algorithms that make image data transmission and storage more efficient.

Inside the AIA System

Designed to meet the needs of deployed military applications, the AIA System brings a new level of performance and capability to real-time FMV. Sophisticated parallel processing of image streams, on-demand video matrix switching, and the ability to present and manipulate multiple streams on a single display provide the high quality and reliable imagery necessary for mission success.

The AIA System consists of separate hot-swappable modules for video input (VIP), video output (VOP), and algorithm processing that plug into a high-speed switching fabric contained in a compact and lightweight docking bay. The docking bay, with a less than 4U form factor, can accommodate combinations of up to 18 modules, and provides dedicated slots for two hot-swappable and redundant load-sharing power supplies. All modules are frontloading, and the entire system, including the fabric switch, can be configured for failover.