Integrated Planning for Telepresence With Time Delays

NASA’s Jet Propulsion Laboratory

A conceptual “intelligent assistant” and an artificial-intelligence computer program that implements the intelligent assistant have been developed to improve control exerted by a human supervisor over a robot that is so distant that communication between the human and the robot involves significant signal-propagation delays. The goal of the effort is not only to help the human supervisor monitor and control the state of the robot, but also to improve the efficiency of the robot by allowing the supervisor to “work ahead.” The intelligent assistant is an integrated combination of an artificial-intelligence planner and a monitor of states of both the human supervisor and the remote robot. The novelty of the system lies in the way it uses the planner to reason about the states at both ends of the time delay.

To enable the human supervisor to work ahead of the robot, the planner and executive parts of the artificial-intelligence system must comprehend that execution of a task becomes split into two stages: that of the “leader” (the human supervisor) and that of the “follower” (the robot). Although a task is not truly complete until done by both the leader and the follower, it is nevertheless essential for the planner and executive parts of the artificial-intelligence system to work on the assumption that the follower will indeed follow, until this assumption is violated. Violations can occur (1) when the human supervisor intentionally or unintentionally deviates from activities previously planned in coordination with robot activities directed toward a goal, (2) the robot fails to execute a command as directed or fails to do anything else required of it within a maximum allowable time, or (3) the robot environment changes or is progressively revealed to be significantly different from what was previously assumed.

The planner part of the intelligent assistant must respond gracefully to such violations and notify the human supervisor. Graceful response must include re-planning, for which it is necessary to cause the state model to revert to the most recent known state of the robot. In re-planning, it is also necessary to recognize which goals have been reached so as not to again expand and schedule the constituent tasks involved in reaching those goals.

The purpose served by the assistant is to provide advice to the human supervisor about current and future activities, derived from a sequence of high-level goals to be achieved. To do this, the assistant must simultaneously monitor and react to various data sources, including (1) actions taken by the supervisor, including commands being issued by the supervisor to the robot; (2) actions taken by the robot as reported with delay; (3) the environment of the robot as currently perceived with time delay; and (4) the current sequence of goals. As any of these change, the assistant must respond appropriately, detecting both normal completion of tasks and exceptional conditions.

This work was done by Mark Johnston and Kenneth Rabe of Caltech for NASA’s Jet Propulsion Laboratory.

The software used in this innovation is available for commercial licensing. Please contact Karina Edmonds of the California Institute of Technology at (626) 395-2322. Refer to NPO-43520.