A generalized framework has been developed for systems validation that can be applied to both traditional and autonomous systems. The framework consists of an automated test case generation and execution system called Nemesis that rapidly and thoroughly identifies flaws or vulnerabilities within a system. By applying genetic optimization and goal-seeking algorithms on the test equipment side, a “war game” is conducted between a system and its complementary nemesis. The end result of the war games is a collection of scenarios that reveals any undesirable behaviors of the system under test.
The software provides a reusable framework to evolve test scenarios using genetic algorithms using an operation model of the system under test. It can automatically generate and execute test cases that reveal flaws in behaviorally complex systems. Genetic algorithms focus the exploration of tests on the set of test cases that most effectively reveals the flaws and vulnerabilities of the system under test. It leverages advances in state- and model-based engineering, which are essential in defining the behavior of autonomous systems. It also uses goal networks to describe test scenarios.
This work was done by Kevin J. Barltrop, Cin-Young Lee, Gregory A. Horvath, and Bradley J. Clement of Caltech for NASA’s Jet Propulsion Laboratory. This software is available for commercial licensing. Please contact Daniel Broderick of the California Institute of Technology at
This Brief includes a Technical Support Package (TSP).
Nemesis Autonomous Test System
(reference NPO-47596) is currently available for download from the TSP library.
Don't have an account?
Overview
The document presents an overview of the Nemesis Autonomous Test System, developed by NASA's Jet Propulsion Laboratory (JPL) to enhance the testing of complex systems, particularly in aerospace applications. The primary focus of the project is to address the challenges of exhaustive testing, which has become increasingly infeasible due to the vast number of possible inputs and interactions in modern systems.
At the core of the Nemesis framework is the use of Genetic Algorithms (GAs), which are heuristic search techniques inspired by Darwinian evolution. These algorithms efficiently navigate complex search spaces to generate and select effective test cases for system verification and validation (SVV). The project aims to improve upon traditional testing methods, which often rely on random sampling or expert-driven approaches that may be either too broad or too narrow in scope.
The document outlines the project's objectives, which include developing a generalized framework for automated test case generation, execution, and evaluation. The Nemesis system interfaces with the Dawn testbed, allowing for iterative testing and evaluation of flight systems. Key components of the system include a supervisor that manages the genetic algorithm processes, a telemetry system, and a fitness function that evaluates test cases based on their ability to uncover system flaws.
Results from the project indicate that Nemesis significantly reduces the number of test iterations required to identify effective test cases compared to random testing—approximately four to five times fewer iterations. This efficiency translates to reduced time and costs associated with testing, making it a more intelligent approach for NASA and JPL.
Additionally, the document highlights the integration of the ASPEN model, which generates valid test scenarios based on various parameters such as fault types and power conditions. This model enhances the testing process by allowing for quick validation of test scenarios before execution.
Overall, the Nemesis Autonomous Test System represents a significant advancement in automated testing methodologies, promising to deliver faster, more effective, and cost-efficient testing solutions for complex aerospace systems. The project showcases the potential of genetic algorithms in improving the reliability and safety of flight systems, ultimately benefiting NASA's mission objectives.
