Modern datasets consisting of retrievals from space-based missions have target results, but often are accompanied by hundreds or thousands of other retrieved parameters or facts regarding a particular retrieval (e.g., pressure, temperature, spectral intensities). Many of the retrieval attempts fail due to complex or contaminated soundings, wasting precious computational time. This algorithm generates a filter based on all available metadata regarding a run that predicts whether it will converge or not.
Modern missions will generate so much data that only 6% of the record is planned on being processed by the existing slow, CPU-intensive retrieval algorithm. This algorithm generates a filter that permits “sounding selection” to avoid attempted retrievals that would inevitably fail and thus waste CPU cycles.
Unlike linear regressions, Fischer analysis, or other standard machine learning techniques that examine the “bulk” of the data to create a “fit,” this method utilizes a genetic algorithm that establishes upper and lower thresholds for each input feature. These thresholds are then optimized with a training dataset and reduced to the smallest identical set of rules that generates the same filter output. This increases scientific interpretability later as to the mechanics of the filter’s operation.
This work was done by Lukas Mandrake of Caltech for NASA’s Jet Propulsion Laboratory.
This software is available for commercial licensing. Please contact Dan Broderick at
This Brief includes a Technical Support Package (TSP).
Automated Generation of Adaptive Filter Using a Genetic Algorithm and Cyclic Rule Reduction
(reference NPO48254) is currently available for download from the TSP library.
Don't have an account?
Overview
The document is a Technical Support Package from NASA's Jet Propulsion Laboratory (JPL), specifically identified as NPO-48254, which focuses on the "Automated Generation of Adaptive Filter Using a Genetic Algorithm and Cyclic Rule Reduction." It is part of NASA Tech Briefs, aimed at disseminating results from aerospace-related developments that have potential applications across various technological, scientific, and commercial fields.
The primary objective of the document is to present methodologies for improving data retrieval and processing through the use of adaptive filters. These filters are designed to enhance the quality of data analysis by automatically adjusting their parameters based on the input data characteristics. The use of genetic algorithms in this context allows for the optimization of filter performance, enabling the system to evolve and adapt over time to better meet the needs of specific applications.
Key features of the proposed system include:
-
Retrieval Quality Estimation (RQE): The document outlines a framework for RQE that aims to perform comparably to expert systems, such as those developed by Chris O'Dell. This system is designed to provide transparency in data handling, allowing users to adjust the amount of data processed based on their requirements.
-
Identification of Key Parameters: The RQE framework identifies critical parameters that correlate with the quality of data retrieval, which is essential for ensuring that the most relevant and useful data is prioritized.
-
Product Development: The document discusses the creation of a new product that sorts soundings (data points) by their likely utility, enhancing the decision-making process for users.
-
Geographic and Temporal Neutrality: The system is designed to be unbiased, not favoring specific geographic regions or timespans, which is crucial for applications that require a broad and equitable analysis of data.
-
Incorporation of Truth Metrics: The framework integrates both TCCON (Total Carbon Column Observing Network) and SHA (Surface Heat Anomaly) as truth metrics to validate the quality of the data being processed.
The document also notes that the RQE has been completed for specific data types (Land H-gain and Glint) but not for M-gain due to insufficient data.
Overall, this Technical Support Package serves as a resource for researchers and developers interested in advanced data processing techniques, particularly in the context of aerospace applications, and provides contact information for further inquiries related to the research and technology discussed.
