A computer program denoted OODT CAS has been transformed to enable performance of larger tasks that involve greatly increased data volumes and increasingly intensive processing of data on heterogeneous, geographically dispersed computers. Prior to the transformation, OODT CAS (also alternatively denoted, simply, "CAS") [wherein "OODT" signifies "Object-Oriented Data Technology" and "CAS" signifies "Catalog and Archive Service"] was a proven software component used to manage scientific data from space-flight missions. In the transformation, CAS was split into two separate components representing its canonical capabilities: file management and workflow management. In addition, CAS was augmented by addition of a resource-management component. This third component enables CAS to manage heterogeneous computing by use of diverse resources, including high-performance clusters of computers, commodity computing hardware, and grid computing infrastructures.
CAS is now more easily maintainable, evolvable, and reusable. These components can be used separately or, taking advantage of synergies, can be used together. Other elements of the transformation included addition of a separate Web presentation layer that supports distribution of data products via Really Simple Syndication (RSS) feeds, and provision for full Resource Description Framework (RDF) exports of metadata.
This work was done by Chris Mattmann, Dana Freeborn, Daniel Crichton, John Hughes, Paul Ramirez, Sean Hardman, and David Woollard of Caltech and Sean Kelly of Northrop Grumman Information Technology for NASA's Jet Propulsion Laboratory.
This software is available for commercial licensing. Please contact Karina Edmonds of the California Institute of Technology at (626) 395-2322. Refer to NPO-44883.
This Brief includes a Technical Support Package (TSP).
Transformation of OODT CAS To Perform Larger Tasks
(reference NPO-44883) is currently available for download from the TSP library.
Don't have an account?
Overview
The document outlines the transformation of the Object Oriented Data Technology (OODT) Catalog and Archive Service (CAS) at NASA's Jet Propulsion Laboratory (JPL) to enhance its capabilities for handling larger tasks and increasing data demands in scientific missions. The re-architecture of CAS was motivated by the need to support high throughput job processing, terabyte-scale data management, and computationally intensive processing required by next-generation NASA missions, such as the Orbiting Carbon Observatory (OCO) and the National Polar-orbiting Partnership (NPP) Sounder.
The new CAS implementation focuses on scalability and flexibility, allowing it to manage massive data volumes and support distributed computing environments. The CAS has been split into three main components: file management, workflow management, and resource management. This modular approach enables users to leverage each component individually or collectively, enhancing maintainability and reusability.
The file management component improves upon legacy systems by supporting dynamic metadata, flexible policies for data persistence, and advanced search capabilities. It allows for the separation of metadata catalogs and repository management, facilitating better data ingestion and distribution.
The workflow management component introduces graph-based models for parallel execution of scientific processes, enabling dynamic task reallocation and improved analysis. It supports various workflow patterns and allows for the persistence of workflow instances in multiple backend stores, enhancing the overall efficiency of data processing.
The resource management component is designed to manage heterogeneous computing resources, including high-performance clusters and grid computing infrastructures. It provides capabilities for dynamic node profiling, job scheduling, and integration with existing batch management and resource monitoring systems.
The document also highlights the use of open-source technologies from the Apache Software Foundation and Sun’s Java software to upgrade the CAS implementation. When necessary, custom glue code was developed to integrate these components into a cohesive system.
Overall, the transformation of the OODT CAS represents a significant advancement in NASA's ability to manage and process large volumes of scientific data, supporting ongoing and future aeronautical and space activities. The new CAS is positioned to meet the evolving demands of scientific research and data management in the coming years.
