Fuel economy is a key factor in worldwide energy-consumption. One way to improve fuel economy is using driver feedback to promote efficient habits (eco-driving). Emerging research is looking into using Advanced Driver-Assistance Systems (ADAS) as a tool for promoting eco-driving.

For fixed speeds, fuel economy can also be enhanced by improving powertrain efficiency through optimizing the use of engine and battery power; termed Optimal Energy Management Strategy (Optimal EMS). Optimal EMS functions by controlling engine power. According to recent research, the largest improvements are possible when eco-driving and an Optimal EMS are combined.

The advanced driver-assistance systems (ADAS) now used for active safety use sensors such as light detection and ranging (LIDAR), radio detection and ranging (RADAR), ultrasonic sensors, and various types of imaging systems.

Computer vision to identify the surrounding environment and the classification of objects in a video is a significant area of research in ADAS. The use of a camera is one of the easier methods to determine the type of object that the vehicle is approaching. The general flow for this object detection includes image acquisition, pre-processing, segmentation, object detection and tracking, depth estimation, and system control. To more reliably accomplish the task of object detection, recent approaches are exploring deep learning algorithms such as convolutional neural networks (CNN) for greater accuracy.

Data Acquisition

Fig. 1. Primary city street with traffic light and vehicle detection. (© SAE International)

Two routes were chosen to test different conditions: a highway drive cycle, and a city drive cycle. Four test runs were driven for each cycle.

Sensors were used in a test vehicle to determine vehicle speed and acceleration, along with a video feed of the driving environment.

A ZED stereo vision camera was used to obtain video data for post-processing. The camera was placed at the top of the windshield near the rearview mirror, minimizing the effect of glare and maximizing lane line, sign, and vehicle visibility. The downward camera angle reduced the effects of lighting conditions because the lens wasn't overexposed from the sun and still had a full view of the vehicle environment.

ADAS Information for Optimal EMS

Traditional ADAS tracks and utilizes data that would allow a vehicle to drive more safely. Such data includes details of vehicle location, distance of objects to the driver, lane detection, etc. However, for Optimal EMS prediction, only the data that would directly affect vehicle speed was used. As an example, if it is known that the vehicle/driver is going to slow down for a while, then the Optimal EMS may elect to turn off the engine to reduce fuel consumption.

The data can also help identify situations where a vehicle may need to slow down. Traffic light state and stop sign state were chosen because of their roles in regulating speed. For traffic lights, red, yellow, green, and N/A (not available) are states that were deemed to be relevant prediction data. Stop signs define an upcoming event that would slow the vehicle. However, detecting a stop sign nearby when turning onto a street is different from finding a stop sign down the street, thus requiring a different calculation with each stop sign.

In most cases, the only vehicle that affects the driver's speed is the one directly ahead. In situations where a car in an adjacent lane merges nearby, it will be tracked once it's in the main lane. Only the scenarios where the vehicle's spatial relationship to the vehicles ahead of it changes, are considered. The output states for the vehicle in-front state data are defined as increasing, decreasing, and same regarding distance and N/A for no vehicles in the current lane.

An upcoming turn usually indicates a decrease in speed. Significant bends in the road or turning 90° warrant a reduction of speed, making the turning feature a necessary one for Optimal EMS prediction.

Ground Truth Development

Fig. 2. City street with stop sign and vehicle detection. (© SAE International)

The data required from an ADAS system has to provide as close to perfect predictions as possible. This is called ground truth data and is obtained by having a human instead of a computer algorithm closely analyze the environment. Each prediction feature requires human-annotated data for all eight videos at a data rate of 1-Hz. This data is collected to show how an Optimal EMS prediction would fare with completely accurate ADAS data.

ADAS Detection Development

A combination of custom and known algorithms, called ADAS1 and ADAS2, was developed to automatically capture the ADAS EMS prediction features. Only ADAS information that could be obtained from a stereo vision camera was obtained. The features of ADAS1 and 2 and the ground truth selection are shown in the accompanying table.

The eight drive-cycle videos were analyzed by the computer vision algorithms to generate the EMS prediction data. Every 30th frame (for a 30-fps video capture rate) was read for output as a simplification to reduce the need for extensive computational overhead. Various pre-processing steps were applied to the frame image before sending it to the object detection and tracking algorithms.

Read Frame Every frame was obtained from the ZED camera, which has a 110° wide-angle lens at an aspect ratio of 16:9 and a frame resolution of 1280 × 720, captured at 30 fps. It recorded the left and right frame as a single image; therefore, a separation step was required for the left and right frames to be split into separate image arrays. The left frame array was used for object detection and tracking, with the right frame being ignored, to reduce computational overheads.

Pre-processing The second step was to crop the image to reduce the number of computations required. Only the driving and adjacent lanes were included as well as enough height to see the traffic lights. Another step was converting to different color spaces, for example, grayscale. In ADAS2 a Gaussian blur filter was used to prepare several frames for pixel brightness detection in order to make the image more uniform.

Object Detection In order to detect objects such as stop signs that would predict a possible speed change, an image frame is sent to the Convolutional Neural Network (CNN) layer, which returns a list of objects detected with a given confidence score.

The CNN output layer indicated vehicle detection with the help of bounding boxes around those vehicles. Vehicles detected in the image were considered only if found in the driver's lane. The detected vehicle was found and compared to the width of the lane. If found in the range, the object's bounding box was tracked. If the bounding box object was larger in the next frame, it was assumed that the vehicle is approaching. If the bounding box was the same size or smaller, the vehicle is considered to have remained at the same distance. The bounding box area method worked adequately because an exact distance is not needed to detect relative vehicle movement across frames.

Traffic light state detection was done similarly to vehicle detection with the addition of an extra processing step. A list of bounding boxes from the CNN for traffic lights was iterated and filtered based on the confidence score, to select the one traffic light for object state classification that had the highest confidence. Object filtering was performed to eliminate the need to find the state of each traffic light. Stop sign and pedestrian detection were added if their confidence score was higher than 30.

Fig. 3. Highway with vehicle detection. (© SAE International)

Object State Classification Traffic lights and vehicle brake lights required object state classification to determine what state the traffic light indicated and if the brake lights were on. To accomplish this, the bounding box information of the objects was taken, and a sub-image was created based on the size of the bounding boxes. Algorithms then determined the color states of these sub-images. (Figures 1 – 3)

Baseline Energy Management Strategy Simulation

A 2010 Toyota Prius was selected as the vehicle model due to its commercial prevalence and because it has the best fuel economy in its class. The Autonomie modeling software used demonstrated strong correlation with real-world testing.

When the numbers from real-world measured values were compared to the simulation values, the simulation fuel economy was within 3% of all of the physically measured fuel economy numbers and the baseline EMS was considered validated.