This test automation software reduces the burden of testing artificial-intelligence (AI) planning software written in the Heuristic Scheduling Testbed System (HSTS) [a software architecture for planning and scheduling]. It automates what would otherwise be the labor intensive and infeasibly time-consuming process of determining whether the plans produced by the planner in response to test case inputs are consistent with the specifications. Automating this process is particularly important for verifying planning systems due to the large numbers of test cases and the complexity of both the plans and the specifications. This program automatically translates constraints written in the HSTS constraint language into automatically executable queries in a database language. As a result, plans generated by HSTS planning software can be automatically verified for satisfaction of all the constraints. Moreover, the constraints can be validated by manually expressing their intents directly in the database query language, then plans generated by the HSTS planning software can be automatically verified through the responses to those queries. Redundant information contained within plans is automatically cross-checked, thus increasing confidence in the correct operation of the planning software and of the plan-checkers themselves. This program generates a pass/fail report of the responses of the planning software, as well as detailed and summary reports as to how each constraint was or was not satisfied. It also generates simple metrics concerning the constraints and requirements exercised in testing.
This program was written by Benjamin Smith and Martin Feather of Caltech for NASA's Jet Propulsion Laboratory. For further information, access the Technical Support Package (TSP) free on-line at www.nasatech.com/tsp under the Software category
This software is available for commercial licensing. Please contact Don Hart of the California Institute of Technology at (818) 393-3425.
Refer to NPO-21118
This Brief includes a Technical Support Package (TSP).
Test Automation Software for AI Planning Software
(reference NPO-21118) is currently available for download from the TSP library.
Don't have an account?
Overview
The document discusses a software tool developed for the automated testing of artificial intelligence (AI) planning systems, specifically those utilizing the Heuristic Scheduling Testbed System (HSTS). This software addresses the challenges associated with verifying the plans generated by AI planners, which can be both complex and numerous, making manual testing labor-intensive and time-consuming.
The primary function of the software is to automate the verification process, ensuring that the plans produced by the HSTS align with specified constraints. It translates constraints written in the HSTS constraint language into executable queries in a database language, allowing for efficient validation of the generated plans. This automation is crucial for verifying planning systems, as it significantly reduces the workload involved in testing and increases the reliability of the results.
The software not only verifies the satisfaction of constraints but also allows for manual expression of constraint intents in the database query language. This dual approach enhances the verification process, as it enables the automatic checking of plans against the responses generated by the queries. Additionally, the software cross-checks redundant information within plans, which bolsters confidence in the accuracy of both the planning software and the verification tool itself.
Upon completion of the testing, the software generates comprehensive reports, including pass/fail results and detailed summaries of how each constraint was satisfied or not. It also provides metrics related to the constraints and requirements exercised during testing, offering valuable insights into the performance of the planning software.
The development of this test automation software was carried out by Benjamin Smith and Martin Feather from the California Institute of Technology (Caltech) for NASA’s Jet Propulsion Laboratory (JPL). The software is available for commercial licensing, and interested parties are directed to contact Don Hart at Caltech for further information.
The document emphasizes that references to specific commercial products or services do not imply endorsement by the U.S. Government or JPL. It also clarifies that the work was conducted under a contract with NASA, highlighting the collaborative nature of the project between government and academic institutions.
In summary, this document outlines a significant advancement in the field of AI planning software testing, showcasing a tool that enhances efficiency, reliability, and confidence in the verification of complex AI-generated plans.
