Computer Vision Lends Precision to Robotic Grappling

Johnson Space Center, Houston, TX

A target is acquired by the vision system. The parameters [X, Y, Z] (cm) are calculated and delta commands are fed to the human operator to move the robotic arm end-effector to grapple a moving target. (Image: NASA)

Innovators at NASA’s Johnson Space Center (JSC) have developed computer vision software that derives target posture determinations quickly and then instructs an operator how to properly align a robotic end-effector with a target that they are trying to grapple.

As an added benefit, the software’s object identification capability can also help detect physical defects on targets. This technology was originally created to aid robotic arm operators aboard the International Space Station (ISS) that relied more heavily upon grappling instructional maneuvers derived from flight controllers on the ground at JSCs Mission Control Center (MCC).

Despite the aid of computer-based models to predict the alignment of both robotic arm and target, iterative realignment procedures were often required to correct botched grapple operations, costing valuable time. To solve this problem, NASA’s computer vision software analyzes the live camera feed from the robotic arms single borescope camera and provides the operator with the delta commands required for an ideal grasp operation.

This process is aided by a machine learning component that monitors the camera feed for any of the ISS’s potential target fixtures. Once a target fixture is identified, proper camera and target parameters are automatically sequenced to prepare for grasping operations.

The software includes a machine learning component that uses a trained regional Convolutional Neural Network (r-CNN) to provide the capability to analyze a live camera feed to determine ISS fixture targets a robotic arm operator can interact with on orbit.

This feature is intended to increase the grappling operational range of ISS’s main robotic arm from a previous maximum of 0.5 m for certain target types, to greater than 1.5 m, while significantly reducing computation times for grasping operations.

Industrial automation and robotics applications that rely on computer vision solutions may find value in this software’s capabilities. A wide range of emerging terrestrial robotic applications, outside of controlled environments, may also find value in the dynamic object recognition and state determination capabilities of this technology as successfully demonstrated by NASA on-orbit.

NASA is actively seeking licensees to commercialize this technology. Please contact NASA’s Licensing Concierge at This email address is being protected from spambots. You need JavaScript enabled to view it. or call at 202-358-7432 to initiate licensing discussions. For more information, visit here .