The Constellation Program and the Exploration Technology Development Program (ETDP) funded the development of diagnostic models using the TEAMS (Testability Engineering and Maintenance System) tool for the Ares, Orion, and Ground Operations Projects to demonstrate operational uses for ground processing and launch operations. These models were found useful not only for operational pre-launch checkout, but also for analysis of failure effects, failure detection coverage, and fault isolation effectiveness. TEAMS, a commercial model-based tool from Qualtech Systems, Inc. (East Hartford, CT), performs fault diagnostics (isolation and identification). Fault isolation means identifying the location of the fault (cause) that is compromising system functions. Fault identification means identifying the failure mode (mechanism) that is causing system failure. Diagnostics refers to both fault isolation and identification functions.

Translating design schematics into a TEAMS directed graph model.

The TEAMS tool provides the capability for engineers to model a system architecture using directed graphs in which nodes represent components, and directed arcs (lines with arrows) represent the connections between components. Failure modes are modeled as the lowest-level internal elements of a component, and failure effects are modeled as functions associated with each component. The locations of the components for these functions define the failure effect propagation paths. Sensors and measurements are associated with test points, and “tests” at those points define which functions are observable. Changes to the system configuration are controlled through switch states, which represent component power, mechanical, and software switches. Failure effects are modeled with effect nodes or pseudo sensors. With TEAMS, engineers can analyze the capability of the measurement suite to detect failures and isolate faults by forward- and backward-chaining logic, and to perform operational diagnostics to determine the locations and mechanisms of failure causes. The TEAMS model so developed is a “diagnostics model.”

Effective Failure Detection

TEAMS enables a variety of useful and important failure analyses. Its ability to trace from a failure mode to all of its effects (whether sensed or not), and from a particular effect to all possible failure mode causes, is useful for a host of applications. Tracing backwards to all possible causes of a failure effect is important for caution and warning, launch commit criteria, fault trees, and probabilistic risk analyses. Conversely, tracing forward to all possible effects is necessary for the understanding of failure scenarios and all mechanisms in which these effects are observed. Combining forward and backward traces is the basis for assessment of failure detection coverage and fault isolation effectiveness. The former is needed for analysis and verification of failure mitigation mechanisms, and the latter for assessment and development of repair strategies.

The process of building and verifying the diagnostic model involves face-to-face meetings in which subsystem designers, safety and failure model analysts, systems engineers, and modelers formally trace the failure effect propagation paths through the system schematics. Doing so significantly improves the quality of the failure modes and effects analyses by providing accurate failure effects and detection mechanisms. If fault tree nodes are modeled, then the directed graph model enables formal connection of the fault trees to the Failure Modes and Effects Analyses (FMEAs) to the system architecture, providing a means to uncover gaps and overlaps. The model assists with assessment of the Time to Criticality for failures by defining the precise paths along which failure effects propagate, including the specific physics at each step. Finally, the model is delivered to operations personnel to provide systems diagnostics during ground processing and launch operations.

An architectural concept for fault detection, isolation, and recovery (FDIR) was formulated under the ETDP Integrated System Health Management Project to integrate vehicle and ground fault models, as well as other health management tools and techniques. This FDIR architecture was tested during execution of the Ares I-X Ground Diagnostic Prototype (GDP). The Ares I-X GDP demonstrated anomaly detection (detecting unexpected events, generally different from what has previously been observed), failure detection, and fault diagnostics for the Ares I-X First Stage Thrust Vector Control, and for the associated ground hydraulics while the vehicle was in the Vehicle Assembly Building at Kennedy Space Center (KSC) and while it was on the launch pad.

The GDP combines three existing tools. The first tool is TEAMS, described above. The second tool, SHINE (Spacecraft Health Inference Engine), is a rule-based expert system that was developed at the NASA Jet Propulsion Laboratory. SHINE rules were developed for failure detection and mode identification, and SHINE outputs served as inputs to TEAMS. The third tool, IMS (Inductive Monitoring System), is an anomaly detection tool developed at NASA Ames Research Center. The GDP was deployed to KSC and monitored live data during the prelaunch period leading up to the October 28, 2009 launch of Ares I-X. Ares I-X did not have any failures in the systems monitored by the prototype. The GDP had a small number of false alarms, largely due to differences between the Ares I-X data and the historical Space Shuttle data on which IMS was trained. The prototype successfully demonstrated the feasibility of integrating three very different failure detection and fault diagnostic methods, and of integrating diagnosis of the vehicle with diagnosis of the ground systems.

Using lessons from the Ares I-X GDP, an FDIR prototype application for the Constellation liquid hydrogen subsystem was developed to provide the initial operating capability for KSC’s launch control system. It included TEAMS for diagnostics and IMS for anomaly detection. The FDIR prototype used a liquid hydrogen system diagnostics model, an Ares I Main Propulsion System diagnostic model, and a ground power system model to formulate preliminary processes and interface requirements for ground-vehicle and ground-ground diagnostic model integration and validation. The FDIR Architecture also investigated the feasibility of integrating prognostics (failure prediction) capabilities.

Benefits of Automated Diagnostics

Ares I-X Launch

The automation of pre-launch diagnostics for launch vehicles offers three potential benefits: improving safety, increasing launch availability, and reducing cost. In today’s launch processing environment, fault isolation is conducted on a subsystem-by-subsystem basis. Launch support personnel identify and respond to anomalies, faults, and failures by conducting complex design analyses, tracing failure effect propagation paths, and correctly identifying suspected or bad components in real time, without benefit of automation to assist with integrated analysis. Without such automation, the complexity of ground and vehicle systems and their interactions requires a large, highly skilled workforce for safe operations.

Integrated diagnostic models developed by system designers provide accurate and rapid (on the order of seconds instead of minutes, hours, or days when not automated) information on locations and identities of potential causes of the observed failure effects. Faster diagnosis decreases recovery time and increases the launch availability of ground and vehicle systems. Integrated diagnostic models also support flight rationale assessments. Encapsulating design knowledge within the diagnostic models reduces operations personnel workload and enables more efficient launch operations. Anomaly detection and prognostics applications also increase launch systems availability and decrease workforce requirements by alerting operators to anomalous conditions and impending failures. Both techniques enable condition-based maintenance, which prevents future system damage and reduces remediation time and cost.

The FDIR Architecture and diagnostic tool suite will be matured in follow-on technology development efforts to perform integrated diagnostics for ground-vehicle systems, and improve troubleshooting and recovery by providing operators with recommendations for mitigation and/or recovery from anomalous or failed conditions. Diagnostic models, in their analytic and operational uses, provide significant benefits to NASA programs, improving safety, reliability, and availability. If the models are developed in design and re-used in operations, their development costs are significantly decreased compared to current NASA methods, and they provide significant cost benefits in operations by reducing diagnostic and decision-making times, and reducing operator workloads. These analytic and operational diagnostic capabilities are expected to be certified to support the 21st Century Space Launch Complex, and are applicable to other applications beyond launch vehicles, including surface systems, crewed spacecraft, robotic spacecraft, and aircraft.

For more information, contact Stephen B. Johnson (stephen.b.johnson @nasa.gov), Mark Schwabacher (This email address is being protected from spambots. You need JavaScript enabled to view it.), or Barbara Brown (This email address is being protected from spambots. You need JavaScript enabled to view it.).