USC Viterbi Computer Science PhD student Heramb Nemlekar (left) and Assistant Professor Stefanos Nikolaidis aim to teach robots how to predict human preferences in assembly tasks. (Image: Keith Wang)

Humans have a knack for understanding each other’s goals, desires, and beliefs — a skill needed to anticipate people's actions. This “theory of mind” comes easily to us humans. But to robots? Not so much. Now, a group of USC Viterbi computer science researchers aims to change that.

They want to teach robots how to predict human preferences in assembly tasks, so they can one day help out on everything from building a satellite to setting a table.

“When working with people, a robot needs to constantly guess what the person will do next,” said lead author Heramb Nemlekar, a USC computer science PhD student working under the supervision of Stefanos Nikolaidis, assistant professor of computer science. “For example, if the robot thinks the person will need a screwdriver to assemble the next part, it can get the screwdriver ahead of time so that the person does not have to wait. This way the robot can help people finish the assembly much faster.”

Most of the current techniques require people to show the robot how they would like to perform the assembly, but this takes time and effort and can defeat the purpose. “Imagine having to assemble an entire airplane just to teach the robot your preferences,” said Nemlekar.

However, the researchers found similarities in how an individual will assemble different products. So, instead of showing the robot their preferences in a complex task, they created a small assembly task they called a “canonical” task that people can easily and quickly perform — in this case, putting together parts of a simple model airplane, such as the wings, tail and, propeller.

The robot observed the human complete the task using a camera placed directly above the assembly area, looking down. To detect the parts operated by the human, the system used AprilTags — like QR codes — attached to the parts. Then, the system used machine learning to learn a person’s preferences based on their sequence of actions in the canonical task.

In the researchers’ user study, the system was able to predict the actions that humans will take with around 82 percent accuracy.

“We hope that our research can make it easier for people to show robots what they prefer,” said Nemlekar. “By helping each person in their preferred way, robots can reduce their work, save time, and even build trust with them.”

The team’s aim is not to replace humans on the factory floor. Instead, they hope this research will lead to significant improvements in the safety and productivity of assembly workers in human-robot hybrid factories and eventually improve quality of life for everyone.

“I viewed this project as an example of a human-centered robotics where we’ll teach robots to not just do the optimal thing but to adapt to our preferences and help us,” said Nikolaidis in an exclusive Tech Briefs interview, the entirety of which can be seen below. “So, I would like to encourage people to do research on technology that is focused on improving our quality of life.”

The researchers next plan to develop a method to automatically design canonical tasks for different types of assembly tasks.

Here is the Tech Briefs interview with Nikolaidis, edited for clarity and length.

Tech Briefs: What inspired your research?

Nikolaidis: When I started grad school at MIT, my graduate advisor wanted some furniture in the lab. So, we got some and started to assemble it. My colleague Matthew has a mechanical engineering background, so he was super skilled, but I was not; I would take my time, try different things, and wouldn’t follow the directions. At the end, we both ended up with a chair, but we followed very different processes to get there.

I remember I thought at the time, ‘Wouldn’t it be great to have a robot or a robotic assistant who can give me a tool when I need it and help the assembly process?’ That was the starting point; when I started at USC, I wanted to implement this capability.

Tech Briefs: What was the biggest technical challenge you faced throughout your work?

Nikolaidis: The idea is that we want robots to learn about human preferences to assist us, but every single one of us is different. So, when you build a robot, you do not want it to learn the preferences of a single person, right? Or, you do not want a system to work for only one person; you want something that works for an entire population. This meant distilling what factors people consider when they select actions for an assembly.

We had to go back and study papers on the economy of human movement and on cognitive science to find out what are exactly the factors that people consider — which are prioritizing physical and mental workload — and build a system that allows us to infer a user’s preferences based on these factors. So, what we did was we designed a canonical task that enables us to infer how users prioritize physical workload or mental workload, and then the robot can use that consistently.

Tech Briefs: You said your next steps include a plan to develop a method to benefit personal assistance in homes. How is that coming along? What’s your next move?

Nikolaidis: The idea is that we can have a canonical task and we can use it to infer the user’s preference, and we design the canonical task manually. This means that we use domain expertise to design this kind of canonical task that needs to be expressed.

What we’re trying to do now is automate the process. In other words, we’re trying to come up with metrics that assess how expressive a canonical task is and how effective it is for distilling a user’s preference. What we are trying now to do is build a system where you pass this input, the actual assembly task, and then it outputs a much shorter canonical task that if you do that, the robot will understand your preferences. It’s still a work in progress; it’s a very challenging problem, but we’re trying to automate part of the process here.

Tech Briefs: Do you have any advice for engineers aiming to bring their ideas to fruition?

Nikolaidis: I would like to give a shoutout to my PhD student here, Heramb Nemlekar, who led this work. He has been very fearless and has run user studies with hundreds of participants — some of them actually during the pandemic! So, I would encourage engineers to be fearless and target the hardest problems, but also understand that this is a trial-and-error process. We had to run more than 10 studies, and each study lasted about one or two months until we got this right. So, it’s not going to work the very first time, but I think that’s what makes the problem interesting; if it worked the first time then the problem would not be interesting, nor would it be challenging.

Tech Briefs: Is there anything else you’d like to add?

Nikolaidis: I viewed this project as an example of a human-centered robotics where we’ll teach robots to not just do the optimal thing but to adapt to our preferences and help us. So, I would like to encourage people to do research on technology that is focused on improving our quality of life.