© Copyright 2015 by RE2, Inc.

As ground robotics moves towards autonomous and semi-autonomous operations, the need to have full control over manipulators is still required for complex situations, or when a user feels the need to take control of the system. Repetitive tasks tend to be the tasks in which humans begin to feel fatigued and make mistakes. Automation is often great in these situations. Complex tasks keep a human alert and thinking as long as the task can be performed in a short amount of time. Automated systems tend to have a harder time completing these complex tasks accurately and in a timely manner. Research has shown that when con - trolling a robot outside autonomous mode, a good control device needs to give the user full control of the system while enabling the mission to be completed in a quick, accurate, and efficient manner. Additional research shows that limitations in a control device can often reduce the usefulness of the robot.

In recent years, manipulators have become more capable and more dexterous through additional degrees of freedom. As a result, new control techniques and new control devices are required. Techniques such as “flying the end effector” require little cognitive load when working in environments with a minimal amount of obstacles, but users do not have control of all the joints and links with this control method. This can potentially result in unwanted configurations of the robot or collisions with obstacles. Current methods for having full control over all of the joints often require moving one or two joints at a time through the use of knobs or buttons, resulting in an overall system that is hard to control, very slow, and is tedious for the operator.

A wide range of existing control devices was studied with the goal of determining which were the most intuitive to use, and resulted in the fastest and highest success rate for common tasks. This research also looked into combining devices to utilize the positive aspects of the devices while canceling out the negative aspects. The project looked at existing products and techniques to determine their strengths and weaknesses in terms of ease of control and their extensibility to additional degrees of freedom, among other factors. This research identified several approaches to manipulation control of high-degree-of-freedom arms. In-depth testing and analysis was performed of the most promising devices with a wide range of users. These prototypes and concepts led to further research and development of the most promising solution — a puppet controller —through a contract with the U.S. Navy. This new control device, called the Imitative Controller (IC), allows the user to move a scaled model, sometimes referred to as a puppet, of the robot’s manipulators. This controller has been shown to effectively control a highly dexterous two-arm system, consisting of two 7-degrees-of-freedom arms and a 2-degrees-of-freedom torso (16 total degrees-of-freedom), called the Highly Dexterous Manipulation System (HDMS).


Figure 1. The Imitative Controller (top) in a configuration set to be a scaled-down match of the joints and links of the HDMS robot (bottom).

A puppet control device is often comprised of joints and links that are to the scale of the robot that the device is controlling. The joints have encoders in them to determine the position and orientation of the device, which is then translated to movement and position commands for the robot. In the case of IC (Figure 1 top), the user holds onto a handle and moves their hand around, which moves all of the joints and links between the handle and the base of the device. Given that the puppet device is a model of the robot, the robot will attempt to match the position and orientation in a master/slave relationship. This point of interaction between the human and the control device is similar to that used for flying the end effector.

In this respect, the user is using the device in a similar manner, but with a puppet device, they get a physical confirmation that the overall state of the robot will consist of a set of specific positions and orientations. Flying the end effector relies on mathematical formulas that can produce multiple solutions to the same desired position and orientation, resulting in an unknown final state of the robot, even when the position and orientation of the end effector is known.


Figure 2. A grip with a handle that is at an angle (left) offers better ergonomics to the user when compared to straight handles (right).

Recent research has centered around handle types, handle locations, and the overall size, or scaling factor, of puppet systems in order to optimize the user’s experience and ensure the most accurate and intuitive control of the robot. Handle types refer to the style of grip that is intended as the connection point between the user and the control device. It was found that the best handles are those designed to fit in the user’s hand in such a manner that it is at an angle to the piece of the control device being controlled, such as the grip on the lefthand side of Figure 2.

The curvature of the grip between the thumb and forefinger also appears to aid in better control and more comfort for the user. This handle style also allows for several easy-to-reach buttons to be added. Straight bicycle handlebar-style handles cause extra strain on the user’s wrists, and are especially uncomfortable when buttons are added since the user’s fingers need to stretch and assert force at an unnatural angle.

The handle works best when it is located close to the end point of the control device. This allows the user to intuitively control the robotic arms since the user’s hands are in similar locations to the robot’s end effectors. This becomes important when controlling a two-armed system and coordinated movement between the two arms is required, such as when handing an object from one end effector to another, unscrewing an object while also holding the base of that object, or performing complex, multiend-effector manipulation such as tying a knot. When the location of the handle was moved further up the control device (away from the end point), the user took longer to line up the manipulator to the object that was being manipulated, and had a higher “miss” rate where the user misjudged the location of the end effector compared to the object they intended to manipulate. Handle locations are also important since humans can only bend and twist their own bodies in certain directions and to certain extents, often proving to be the limiting factor for the range of motion in the system. Proper handle locations ensure full control of the device while providing an intuitive method for doing any given task.

Links and Joints

A proper scaling factor for a puppet system is important to ensure users can comfortably control the manipulators using the natural motion of their arms and hands. Link lengths that are too long result in the user stretching to reach certain positions, and the user will become fatigued faster. If the link lengths are too short, the user’s workspace gets smaller and can increase the difficulty of controlling the robot. A good scaling factor needs to be determined for each individual robot and controller pair. This becomes a hard problem to solve as users may have arms of different lengths, making it hard to find a good fit for all potential users. It is also important to keep the same scale factor throughout the puppet device. Failing to do so will violate the puppet nature of the controller, and the end effector position of the manipulator will not match the position of the controller.

The design of the robot is also a factor when choosing a scale factor. If several small joints are clustered in one area of the robot, and then the rest of the robot is large and sparse between joints, a scaled down puppet device may be hard to achieve due to the small size needed to keep the proper scaling near the cluster of joints. To further ensure the puppet system matches the capabilities of the robot, the joint limits on the puppet device should match those of the robot. In addition to ensuring the range of motion is the same between the puppet and the robot, joint limits act as a physical feedback mechanism to the user to let them know they have reached the joint limits.


The base of the puppet can take many different forms. The one that is best is dependent on the application as well as how and where the user intends to use the puppet. Fast movements call for a stable base, such as on a tripod or otherwise attached to a heavy, stable object (such as a table or the chair the user is sitting in). Slower movements can be less demanding on the base of the control device; however, it still requires a base that is heavy enough or to be otherwise attached to something sturdy to stay in place when the joints are being moved.


Feedback to the user enables the user to better understand the interaction between the robot and the environment around the robot. Feedback can inform a user how tightly the robot is grasping an object, if the robot has bumped into something, or to simply give the user a status update. Feedback can be expressed in several different ways, such as light, sound, or force.

Light feedback can also be further broken down into several different modalities. A simple light, such as a single LED, can change colors to indicate feedback. It can blink and change the speed at which it blinks. If the light source is on a movable platform or if there are several lights spread out across a specified area, the location of the lights can also offer feedback. For example, imagine a row of four LEDs located on the handle of a control device. These four LEDs could indicate the location of the robot’s gripper in terms of how open or closed the gripper is. A separate LED could change color from green to yellow to red to indicate how hard the gripper is grasping an object. This same LED can start to blink if the robot detects that the object is starting to slip out of its grasp. This simple visual feedback will offer the user valuable information that can increase the user’s ability to successfully complete the task at hand.

Similar to light-based feedback, sound can offer simple feedback to the user. A user is often using their sight to control a robot. Asking the user to divert their gaze to an LED, even for a fraction of a second, could result in a failed mission. Pulsing sounds, such as a “Geiger counter” noise, can allow the user to keep their visual attention on the task while adding to the user’s knowledge by this repetition of noise. An example of the use of sound is a repeating sound that repeats faster as the robot’s gripper gets closer to an object, and slower as the gripper moves away from the object.

Early research found haptic feedback to be the most useful of the feedback types when manipulating objects. Haptic feedback can come in a variety of types, including vibration and through resistance by pushing or pulling against the user. Vibration feedback can be accomplished through small, vibrating motors such as those found in modern cellphones. The placement and intensity of these motors is important to ensure the user feels the device vibrating, but also to ensure the device does not vibrate so much as to distract the user from the task.

Force feedback can be accomplished by giving the puppet device motors in the joints or in the portion of the device that controls the robot’s grippers. The motors can attempt to hold a specific position or push back in a certain direction once a signal from the robot is received, indicating the robot is grasping an object, one of the joints cannot move in a specific direction, or for any other number of reasons. This resistance prevents further movement in the wrong direction, and informs the user that further movement in that direction is not possible.

Pros and Cons

A puppet control device, consisting of a scaled model of a robot, allows the user to have very specific control over all the joints of that robot while providing an interface that is intuitive and easy to use. Through this method of manipulation, a control device can contain more degrees of freedom than the user has on their own arm. Although this sounds counter-intuitive, the additional degrees of freedom are controlled through the help of biased movements of the joints. For example, a bending joint may be biased to bend in one direction more so than in the other direction. This bias can be easily overcome by the user if they wish a joint to bend in a particular manner, but otherwise the decision on which way to bend is pre-determined. So even though the operator can only actively control the six degrees of freedom of the handle of the controller, the biasing helps control the additional degrees of freedom. This is, in part, what solves the drawbacks with the “flying the end effector” design, and can contribute towards making a puppet control device more intuitive to use.

There are drawbacks to using a puppet device. Often, an entirely new control device is needed that is specific to the robot under control. Specializing controllers in this manner often means higher cost. By having a modular architecture, similar to the one developed for IC, links can be cut to size and joints are non-specific. This lowers the cost of creating a novel configuration for a given robot, and makes it more available to a wider range of customers and applications.

When done properly, puppet systems have the benefits of the “flying the end effector” technique, but maintain full control over all the joints and links of the robot. This combination allows the user to intuitively and easily adjust all of the joints of a manipulator, whether it is 3, 7, or any other number of joints on a single or dual-arm system. Puppet systems, though, have the down side of only being useful for the robot that the puppet is designed to control. To counteract this shortcoming, these control devices can be designed to be easily reconfigurable for changes in the number of joints, link lengths, and offsets between joints.

This allows the device to match several different models of manipulators with minimal effort. The simplicity of puppet devices offers the ability to control complex manipulators, and has sparked the interest of various programs and fields. In a project for the U.S. Navy, a two-armed underwater robot with 7 DOF for each arm is being developed. The ability to control such a system would be nearly impossible with techniques such as individual joint control. Bomb squads also have shown interest in using a puppet control device to control existing robots. Home healthcare is also an industry where this type of control shows potential. A simple control mechanism, such as IC, can enable a person in a wheelchair to control a complex robotic arm to help lift items off the floor, reach up high into cabinets, open doors, or carry heavy items.

This article was written by David Rusbarsky, Jim Grebinoski, and Andrew B. Mor of RE2 Robotics, Pittsburgh, PA; and Jeremy P. Gray of the U.S. Army TARDEC, Warren, MI. For more information, Click Here .

Motion Control & Automation Technology Magazine

This article first appeared in the October, 2015 issue of Motion Control & Automation Technology Magazine.

Read more articles from this issue here.

Read more articles from the archives here.