NTB: When you talk about dialog between robots and humans, part of that involves speech recognition and speech synthesis technology. How sophisticated has that technology become in the last few years?

Fong: It’s made huge advances. In this project, we really were trying to not only use speech recognition and speech synthesis, but we were focusing on “spatial dialogue.” That’s the idea that if a human speaks to a robot and says, “Hey, shine a light to my left,” the robot not only knows what the human is talking about with a light, but it can actually figure out what “my left” means. So there has to be some spatial reasoning based on the robot understanding where it is and the relationship between the two so that it can point the light at the right place. When you say, “my left,” what does “my left” mean? It depends. Are you facing me? Are you next to me? Are you behind me? Being able to reason in terms of spatial language, spatial location, and orientation is something that we think is certainly important for tasks that NASA is going to want to conduct on the lunar surface.

NTB: What types of machine vision systems are you designing into these robotic systems?

Fong: The Intelligent Robotics Group does a broad range of machine vision. We do vision that’s onboard for looking at the world and determining where there are obstacles so the robot can avoid them. We also do vision from off-board, whether that’s aerial or orbital imagery for doing 3D modeling of the terrain, so you can determine areas where you don’t want to send the robot because it’s too steep or too rough, as well as being able to reconstruct and build models you can use; for example, lighting simulation or a wide variety of things.

Ames has actually had a long history of developing computer vision tools. We recently released a software package called Vision Workbench, which is an open-source tool for doing computer vision, and it incorporates a broad range of things including stereovision and texture matching. We use it for a variety of applications. One is the Ames Stereo Pipeline, which is used for building 3D models for orbital imagery. And a new piece of software called the Terrain Pipeline, which produces digital elevation models over a very wide scale. So if you really want to produce a model of a site that’s going to be used for a number of purposes, such as robot planning for science, or things like illumination modeling where you start off with orbital images, you process those into terrain models and then you provide essentially a server that can provide that data to different users and clients.

NTB: Do you see any potential commercial applications for the robotic technology your group is developing?

Fong: Sure. I think that systematic survey – robotic survey – clearly is something that would be of interest to, say, the oil and gas industry. Our work right now is focused primarily on wheeled mobile robots and planetary rover type robots, but there’s no reason the software couldn’t be used by underwater robots if you’re going to try to map and prospect for resources. So clearly, I think, it would be useful from a commercial standpoint.

I think also the computer vision tools that we have been developing would be of interest to organizations that care about doing 3D modeling, whether they’re scientific organizations or anywhere vision is going to go out and look to better understand a site that you want to operate in. The oil and gas industry is a perfect example, and perhaps construction. Perhaps, people who care about long-term environmental changes might be interested in being able to look at how the surface of an area changes over time.

For more information, contact Terry Fong at This e-mail address is being protected from spam bots, you need JavaScript enabled to view it