Teaching Robots to Move by Sketching Trajectories

Motion Design INSIDER

Robotics Institute researchers have developed a way to use sketches to show robots how to move. (Image: Courtesy of the researchers)

Getting robots to perform even a simple task requires tons of behind-the-scenes work. Part of the challenge is planning and executing movements, everything from turning wheels to lifting a robotic arm. To make this happen, roboticists collaborate with programmers to develop a set of trajectories — or pathways — that are clear of obstacles and doable for the robot.

William Zhi, a postdoctoral fellow in Carnegie Mellon University's Robotics Institute (RI), working with Ph.D. student Tianyi Zhang and Robotics Institute Director Matthew Johnson-Roberson are creating new ways to chart these trajectories by using sketches to show robots how to move.

"Traditional approaches to generate robot motion trajectories require specific programming of the robot," said Zhi. "Humans can infer complex instructions through sketches. We seek to empower robots to do the same."

There's been preliminary work that studies using natural language to control robots, but researchers have primarily been testing different ways to teach robots to learn through demonstration. There are two main ways to go about this. One method relies on kinesthetic teaching, where a human records where the robot goes and then physically adjusts the robot to place its joints into the desired positions. The other approach is teleoperation, where the user manipulates the robot with a specialized remote controller or a joystick and then records the demonstration for the robot to copy.

Both methods have their drawbacks. Kinesthetic teaching requires the user to be in the same space as the robot. It's also difficult to manually adjust some robots — and that difficulty only increases with mobile robots, like a quadruped robot with an arm attached to it. And teleoperation requires the user to have precise control, so it takes time to put a robot through its paces.

The RI team's approach for sketching trajectories teaches the robot how to move without the drawbacks of kinesthetic teaching or teleoperation. In this novel method, the robot learns from movements drawn on an image of the environment where it will work.

To capture the image of the environment, the team placed cameras at two locations to take pictures from different perspectives. They then sketched the trajectory of the robot's desired movement on the image and converted the 2D images into 3D models the robot could understand. They did this conversion through a technique called ray tracing, which uses light and shadows on objects to estimate their distance from the camera.

Once the team generated the 3D models, they gave them to the robot to follow. For a quadruped robot equipped with a robotic arm, the researchers sketched three motion trajectories on each of the photographs they took, demonstrating how the arm should move. They converted the images to 3D models using ray tracing and the arm then learned to follow these trajectories in the real world.

Using this technique, the team trained its quadruped robot to close drawers, sketch the letter "B," tip over a box and more. They have also programmed it to open its gripper at the end of certain trajectories, which allows it to drop objects into boxes or cups. Moreover, they can generalize the movements that they teach the robot across many separate tasks.

"We're able to teach the robot to do something and then switch it to a different starting position, and it can take the same action," said Zhi. "We can get quite precise results."

Currently, this method only works on robots with rigid joints and not soft robots, because it needs to account for joint angles and how they correspond to various points in space. But dealing with the hardware comes with its own set of challenges. During their experiments, the quadruped would sometimes lose its balance after executing a motion like extending its arm to close a drawer. That's one of the parameters the team is working on for the program's next iteration.

"People in the field have focused more on algorithms for better motions from a demonstration. Our research, on the other hand, is the inception of using trajectory sketches to instruct robots," said Zhi. "We envision future manufacturing settings, where someone unskilled at programming robots, could sketch on an iPad and then do collaborative work with a robot."

Source