Although driverless forklifts have been around for about 30 years, it's only in the last 10 or so that they've been free to maneuver anywhere around their environment. In the early days, the machines followed a current in a wire buried in a concrete floor. There have since been a handful of different approaches, but all are variants on that theme. They went from wires in the floor to paint on the floor, to tape on the floor, to magnets in the floor, to reflectors on the walls, to QR codes on the ground, but they were all just different types of landmarks. So, the vehicles were confined to a fixed path — quite predictable, but extremely inflexible.

Figure 1. The primary navigation sensors are stereoscopic cameras, 10 of which are mounted in pairs on top of the vehicle.

The infrastructure-free navigation that has commercially matured in the past 10 years enables fully self-driving industrial vehicles that adapt to their environment based on sensor input. There is no need to depend upon explicit landmark infrastructure.

Infrastructure-Free Navigation

Figure 2. A driverless forklift can easily navigate around a corner.

Lidar is the standard out-of-the box solution for infrastructure-free navigation because it's relatively easy. Unfortunately, it performs quite poorly in real-world production settings, where environments are dynamic. In factories and warehouses, everything is moving, including pedestrians, so the machines have to navigate carefully and reliably. People don't buy automated forklifts to have them work some of the time, they're buying them so they never again have to worry about moving pallets 24/7.

Seegrid Corporation (Pittsburgh, PA) produces self-driving forklifts and tow tractors that use cameras for navigation. They're fully self-driving: they sense an environment and make decisions entirely onboard the vehicle. An array of cameras has the advantage over lidar because it affords wide coverage of an environment. With lidar, any individual range measurement is highly precise, but its coverage is so limited it's almost blind. A lidar unit is like a laser pointer on a turntable, it's just spinning around and will give a precise measurement of how far a wall is from you, but it tells you absolutely nothing about that wall ½ inch below or ½ inch above the beam.

Seegrid vehicles use the cameras in pairs to capture information in stereo and use the disparity between the two cameras to obtain three-dimensional spatial data. The five pairs “see” a full hemisphere over and around the vehicle as it is driving. With such a very rich environment of data available, the navigation is extremely reliable in real world conditions. Wide, accurate coverage is more important than having high precision but with only a small number of points. The cameras are used as data sensors — they're not doing landscape photography.

Programming the Forklift

Figure 3. The 360-degree views of the environment are processed by software to identify and locate thousands of unique features on every snapshot.

Programming the vehicles is very straightforward. Because Seegrid's forklifts are built on a manual chassis, you can drive them like a regular truck. In order to program it, you just put it into record mode and drive the route you want it to take. While you are driving, all 10 cameras — the five stereo pairs — are taking pictures continuously. As they find points of contrast in the images, the disparity between the pairs of cameras provides a dense 3D point cloud that covers the full space around the truck. Those images are captured every few centimeters as you drive. Every time the vehicle navigates that path thereafter, the system makes mathematical and geometric probabilistic comparisons against what it “sees” now versus what it saw before. The system then makes small navigation adjustments every fraction of a second to compensate for any differences. Since there's much more information that is persistent compared to what is dynamic, there's more than enough to navigate safely and reliably. It's similar to the teach/repeat model used elsewhere in robotics. Seegrid calls its patented system Evidence Grid technology, which uses probabilities of occupancy for each point in space. Applying probabilities to the data collected by the sensors compensates for the uncertainty in the performance of sensors and in the environment itself.

Putting the Driverless Forklifts to Work

Figure 4. Seegrid's vision technology enables its vehicles to autonomously transport materials throughout facilities for customers like Whirlpool.

I asked Jeff Christensen, Seegrid's VP of Product, whether these forklifts are best suited for repetitive tasks. “Imagine for a simple case in manufacturing that a vehicle does a loop around the assembly line all day long bringing parts to the line and taking others away. That's the most repetitive you might get and obviously it would work for something like that,” he said. However, “something maybe less obvious would be a distribution warehouse where you have pallets at an inbound dock. Each pallet is going to a different location — that's the other extreme,” he said. “For that, we create a ‘route network,’ which is a fully-connected graph of all the places it's been. Then when you tell the vehicle, ‘move this pallet from here to there,’ it will search its entire route network and find the shortest path for that particular task. You can think of it as sort of a dot-to-dot connection, that goes all over your plant or warehouse. Then, when I'm sending a truck somewhere, it can pick the optimal path to go anywhere within that space. Note that you don't want it to move just anywhere in a plant — that's like a delivery truck driving over your lawn to avoid a stop sign — but it should it pick the most productive path that is safe and predictable.”

There are many different ways to input instructions for a specific task, depending on the customer's application. In the simplest case you can put a start and end location on the vehicle itself by punching in a couple of numbers. In a more sophisticated application, it can be fully directed by Seegrid Supervisor, a software system that controls entire fleets of vehicles. Supervisor sends instructions to a specific vehicle, telling it what to pick up and where it's supposed to go. In that case, no operator actually touches it at all. Alternatively, Supervisor has a tablet interface that acts as more of a pull system — there are lots of different operational modes.

The Cameras

I then asked Seegrid VP of Engineering Sean Stetson about the cameras. “We use stereo cameras, which have two imaging channels — two separate sensors and two separate lenses. The cameras are automotive-grade imaging sensors. We prefer global shutters because they produce better stereo quality than rolling shutters. We design the cameras to produce concurrent exposures on both cameras on a given board, so that you're capturing the scene at the same time. From there, there's a lot of math to understand the parallax and the cyclical shifts between specific features, between the two cameras,” he said.

I then asked him how the data is used to control the vehicle. “You can imagine all the other different sensors we use — all sorts of encoders and the other things that you typically have on vehicles to help close feedback loops, as well as motor control systems — it's pretty standard stuff because we use stock powered industrial trucks,” he said. “The math associated with the dense stereo data to produce reliable navigation is the hard part.”

Driverless Forklifts vs Driverless Cars

Another question was whether there are significant differences between these industrial vehicles and self-driving automobiles. “There's not a whole lot of difference. Obviously, there are differences in environments. The types of scenarios encountered are more constrained indoors, compared to the freewheeling world that autonomous cars have to deal with. So, from a sensor suite standpoint, you tend to have more constrained distances and certainly more constrained speed.”

“Stereo cameras in the world of autonomous automobiles are getting more prevalent, specifically in the context of real-time, environmental sensing, in order to do things like semantic segmentation, object detection, classification, and tracking. The design goal for automobiles is to give the cars a kind of human sense of what's going on in the immediate environment around them, largely independent of the navigation. We've shown that cameras can be used for navigation in addition to these.” Stetson said.

The range of interactions with pedestrians, vehicles, or objects is obviously different and more constrained — these forklifts are installed into specific travel paths, so their movements are predictable — they stay in their path. Yet, the flexibility of the system allows those paths to be changed as the need arises, for example when there are changes in production flow. That's why these forklifts function so well in a typical assembly plant, where engineers are always looking for ways to improve operations and reduce overhead and wasted effort.

The required speed for processing the navigation information is a a function of speed of the vehicle. The faster it's moving, the more quickly it needs to take in information, process it, decide on a course of action, and then initiate it. Standard, human-driven industrial trucks typically run at a maximum of eight to nine miles an hour. Obviously, in the streets, cars can go significantly faster than that. “As the technology advances on all fronts, we'll see it being used in increasingly fast vehicles,” said Stetson. “One of the challenges facing the autonomous car market is getting sensors that are reliable at, say, 250 meters, because at 80 miles per hour, 250 meters is the distance needed to reliably detect something, classify it, decide on a course of action, and initiate the action in time to stop if need be. As for our vehicles, we have a whole collection of safety standards imposed on us by various organizations, one of which is ANSI B56.5. We do extensive safety testing on our system to ensure that we can stop within those target numbers over the range of distances that we deal with.”

Safety and Reliability

Figure 5. Autonomous forklifts can safely operate around factory personnel.

Christensen added: “Although we are certainly slower in these indoor environments, we are far from light-weight. We're transporting up to 10,000 pounds at a time, and the vehicle and battery add another 3,000 pounds.” That mass requires extensive testing. “The proof is in the pudding: this past February we exceeded a million miles of fully autonomous production travel with zero personnel safety-related incidents. The forklift industry is incredibly dangerous. In fact, safety is actually one of the important factors growing the demand for autonomy. Our forklifts operate significantly more safely than humans do. There's a fatality about every week in the U.S. from a forklift accident. It's really no surprise: you're moving loads around, the weight is shifting, and you're doing repetitive tasks. Humans are really bad at repetitive tasks. We get bored, and when we get bored, we take shortcuts. When we're moving really heavy things, those mistakes get people hurt,” he said.

Outlook for the Future

Finally, I asked Christensen what he thought about the outlook for his industry. He replied: “The adoption of automation is on the rise for mobile, autonomous material handling. Manufacturing is becoming much more complex. Companies are doing smaller lot sizes with mass customization, which makes supply chains far more complicated, which makes material handling challenges far more complicated. On the warehouse and distribution side demand is growing very rapidly — the ‘Amazonification’ of the whole industry. You can buy everything online, and you expect it instantaneously. The amount of pressure that puts on the logistics and supply chain is enormous and at the same time there is a labor shortage of people who are capable of doing these jobs or want to do them.”

This article was written by Ed Brown, Editor of Sensor Technology. For more information, visit here .


Sensor Technology Magazine

This article first appeared in the March, 2019 issue of Sensor Technology Magazine.

Read more articles from this issue here.

Read more articles from the archives here.