Traditionally, quality flags provided a binary yes/no estimation of a datapoint’s utility. However, in modern instrumentation, significant auxiliary information for each datapoint can be obtained. This permits prediction of more than a binary estimate of good or bad data. Further, the physical confounding forces that obscure an observation’s utility are themselves rarely binary, such as the example of clouds with varying thickness from insignificant to entirely opaque. In this method, many different increasingly stringent filters are created allowing less and less data through, while attempting to minimize an error metric. This metric can be compared with select “truth” systems such as ground observations or regions of the Earth where the truth is believed to be predictable and known. For each sounding, the number of these filters that reject the observation in question becomes an estimate of its data quality: larger values mean most filters reject the sounding, while smaller values mean most filters accept the sounding. This integer, ranging from 0 to 19, is called the Warn Level. Instead of a binary yes/no data quality flag, this instead provides a data ordering paradigm with “better” and “worse” data. Warn Levels can be developed for any metadata-rich datasource with a functional error metric to help guide researchers to superior, tunable data filtration.

The software described is not a Warn Level implementation, but rather an entire method to construct new warn levels for any metadata-rich data source. It is a large genetic algorithm for use on a supercomputer that explores the large dimensional space of all possible filters and combinations of filters to yield the best-performing singleton, pair, triple, etc. filters. These are then folded into a specific Warn Level implementation specific to the problem at hand.

Previous sounding selection and data quality estimations were entirely binary and created ad-hoc. There have been no automated methods for generation of quality estimation based on a goal metric. If a user applied a good/bad data flag to their observations, it might pass only 20% of the data. If this data volume wasn’t high enough for the specific investigation, there was no guide as to the “next most reliable” data to admit for an extra 10% if needed. Warn Levels permit the investigator to gradually add data, starting with the most reliable and then admitting less and less quality data, while being assured that they are keeping their error metric as low as possible. This permits different users to decide how high a Warn Level to include in their analysis, given their data magnitude needs and their sensitivity to data quality.

All instruments that generate data but are too complex to deduce precise posterior error require some kind of “good/bad” data filter to guide users. This product replaces that concept with a tunable dial in terms of percentage of the data required, with the “best” data provided first and the worst offenders withheld until the end.

This work was done by Lukas Mandrake of Caltech for NASA’s Jet Propulsion Laboratory.

This software is available for commercial licensing. Please contact Dan Broderick at This email address is being protected from spambots. You need JavaScript enabled to view it.. Refer to NPO-49483.



This Brief includes a Technical Support Package (TSP).
Document cover
An Empirical Metric of Individual Datapoint Utility Given Ample Metadata as Applied to the OCO-2 Flight System

(reference NPO49483) is currently available for download from the TSP library.

Don't have an account? Sign up here.