The Science Processing and Data Management (SPDM) system’s primary functions are to control and monitor science processes and data products for Soil Moisture Active Passive (SMAP) Science Data System (SDS). SPDM adapts the Apache/OODT Framework 0.3 as the basic infrastructure. Although the OODT core framework is software-extensible, it provides limited, minimal, and rough functionalities. The adaptation effort has been extensive to fulfill SMAP SDS requirements including the integration of 26 processors with complex interactions among processors and files (≈70 static ancillary data types and ≈30 external dynamic ancillary types), pipeline automation to generate multiple products (LO-3) with large data volumes within a few hours to a few days of radar and radiometer instrument acquisition, monitoring and control of the data system, and interfacing with two NASA Data Active Archive Centers (DAACs).
The SPDM system has adapted the open source Apache/OODT 0.3. The key components selected for adaptation include File Manager, Workflow Manager, Resource Manager, Crawler, PGE Task Wrapper, and Balance Framework. These OODT components were re-architected, re-designed, customized, and extended to meet the SMAP SDS requirements.
SPDM Core contains a set of tools to provide the operation interface to the control, monitor, search, and report of SPDM services, processing, resources, and products. SPDM SIPS Interface reports the science data products’ readiness for publishing and distribution automatically.
SPDM Web Services is responsible for providing the search results to the SPDM Web Portal for tracking, monitoring, and reporting purposes. SPDM Web Portal Application provides the on-line Web monitoring of SPDM services, processing, resources, and products.
This work was done by Cynthia L. Wong, Michael D. Cayanan, Hyun H. Lee, Nga T. Quach, Gabriel M. Resneck, Alice Stanboli, Vivian Tang, David T. Cuddy, and Jennifer W. Cruz of Caltech for NASA’s Jet Propulsion Laboratory. This software is available for commercial licensing. Please contact Dan Broderick at
This Brief includes a Technical Support Package (TSP).
SMAP SDS Science Processing and Data Management System
(reference NPO49700) is currently available for download from the TSP library.
Don't have an account?
Overview
The document outlines the SMAP (Soil Moisture Active Passive) Science Data System (SDS) developed by NASA's Jet Propulsion Laboratory (JPL) for processing and managing data related to soil moisture and its freeze/thaw state. The primary objective of the SMAP mission is to provide global, high-resolution mapping of soil moisture, linking terrestrial water, energy, and carbon-cycle processes, estimating global water and energy fluxes, quantifying net carbon flux in boreal landscapes, and enhancing weather and climate forecasts.
The SMAP SDS consists of several subsystems, including the Science Processing and Data Management (SPDM) subsystem, which manages workflows, resources, and data products. It processes raw radar telemetry and radiometer data into validated products, which are then archived and distributed. The system includes Level 1 to Level 4 products, with varying processing times: Level 1 products are processed within 12 hours, Level 2 within 24 hours, Level 3 within 50 hours, and Level 4 products take 7 to 14 days.
The hardware architecture of the SDS features commodity boxes, high-speed switches, and a robust storage system capable of nearly a petabyte of data, including ancillary data and multiple reprocessing campaigns. The system operates on a Linux (CentOS) platform and utilizes a 10 Gigabit backbone for efficient data transfer.
The SDS functionalities include processing management, which automates workflows, and a file manager that tracks and stores files for data accessibility. The system comprises 19 processors and 9 pre-processors that work together to generate standard products and prepare inputs for processing. The Operations Interface allows for monitoring and controlling the processing system, generating reports, and providing access to science data products.
The mission is a collaboration between JPL and NASA's Goddard Space Flight Center (GSFC), utilizing an L-band synthetic aperture radar (SAR) and radiometer. Launched on January 29, 2015, the mission is designed to last three years and is recognized as a tier one mission in the NRC Earth Science Decadal Survey.
Overall, the document serves as a technical support package, detailing the architecture, functionalities, and objectives of the SMAP SDS, emphasizing its role in advancing our understanding of soil moisture dynamics and its implications for climate and environmental monitoring.
