Framework for Integrating Science Data Processing Algorithms Into Process Control Systems
- Created on Thursday, 01 December 2011
This technique can be used for data processing and management systems.
A software framework called PCS Task Wrapper is responsible for standardizing the setup, process initiation, execution, and file management tasks surrounding the execution of science data algorithms, which are referred to by NASA as Product Generation Executives (PGEs). PGEs codify a scientific algorithm, some step in the overall scientific process involved in a mission science workflow.
The PCS Task Wrapper provides a stable
operating environment to the
underlying PGE during its execution
lifecycle. If the PGE requires a file, or
metadata regarding the file, the PCS
Task Wrapper is responsible for delivering
that information to the PGE in a
manner that meets its requirements. If
the PGE requires knowledge of
upstream or downstream PGEs in a
sequence of executions, that information
is also made available. Finally, if
information regarding disk space, or
node information such as CPU availability,
etc., is required, the PCS Task
Wrapper provides this information to
the underlying PGE.
After this information is collected, the PGE is executed, and its output Product file and Metadata generation is managed via the PCS Task Wrapper framework. The innovation is responsible for marshalling output Products and Metadata back to a PCS File Management component for use in downstream data processing and pedigree. In support of this, the PCS Task Wrapper leverages the PCS Crawler Framework to ingest (during pipeline processing) the output Product files and Metadata produced by the PGE.
The architectural components of the PCS Task Wrapper framework include PGE Task Instance, PGE Config File Builder, Config File Property Adder, Science PGE Config File Writer, and PCS Met file Writer. This innovative framework is really the unifying bridge between the execution of a step in the overall processing pipeline, and the available PCS component services as well as the information that they collectively manage.