Giving Autonomous Vehicles the Ability to Reason

University of Virginia, Charlottesville, VA

Yen-Ling Kuo, an Assistant Professor of Computer Science, is building a driving simulator, similar to this one in UVA Engineering’s Link Lab, to collect data on driving behavior. Kuo use the data to enable a robot’s AI to associate the meaning of words with what it sees by watching how humans interact with the environment or by its own interactions with the environment. (Image: Graeme Jenvey/University of Virginia School of Engineering and Applied Science)

Self-driving cars are coming, but will you really be okay sitting passively while a 2,000-pound autonomous robot motors you and your family around town?

Would you feel more secure if, while autonomous technology is perfected over the next few years, your semiautonomous car could explain to you what it’s doing — for example, why it suddenly braked when you didn’t?

Better yet, what if it could help your teenager not only learn to drive, but to drive more safely?

Yen-Ling Kuo, the Anita Jones Faculty Fellow and Assistant Professor of Computer Science at the University of Virginia School of Engineering and Applied Science, is training machines to use human language and reasoning to be capable of doing all of that and more. The work is funded by a two-year Young Faculty Researcher grant from the Toyota Research Institute.

“This project is about how artificial intelligence can understand the meaning of drivers’ actions through language modeling and use this understanding to augment our human capabilities,” Kuo said.

“By themselves, robots aren’t perfect, and neither are we. We don’t necessarily want machines to take over for us, but we can work with them for better outcomes.”

To reach that level of cooperation, you need machine learning models that imbue robots with generalizable reasoning skills.

That’s “as opposed to collecting large datasets to train for every scenario, which will be expensive, if not impossible,” Kuo said.

Kuo is collaborating with a team at the Toyota Research Institute to build language representations of driving behavior that enable a robot to associate the meaning of words with what it sees by watching how humans interact with the environment or by its own interactions with the environment.

Let’s say you’re an inexperienced driver, or maybe you grew up in Miami and moved to Boston. A car that helps you drive on icy roads would be handy, right?

This new intelligence will be especially important for handling out-of-the-ordinary circumstances, such as helping inexperienced drivers adjust to road conditions or guiding them through challenging situations.

“We would like to apply the learned representations in shared autonomy. For example, the AI can describe a high-level intention of turning right without skidding and give guidance to slow to a certain speed while turning right,” Kuo said. “If the driver doesn’t slow enough, the AI will adjust the speed further, or if the driver’s turn is too sharp, the AI will correct for it.”

Kuo will develop the language representations from a variety of data sources, including from a driving simulator she is building for her lab this summer.

Kuo’s proposal closely aligns with the Toyota Research Institute’s goals for advancing human-centered AI, interactive driving, and robotics.

For more information, contact Jennifer McManamay at This email address is being protected from spambots. You need JavaScript enabled to view it.; 540-241-4002.