When it comes to future automotive technology, much of the focus has been on fully autonomous driving. Not so much in the spotlight, but it’s gaining momentum, is the ongoing in-cabin technology improvements. These improvements have come in response to problems including that of infants and children accidentally left in the car, drivers falling asleep, and distracted driving. Behind the scene is artificial intelligence (AI)-based machine vision innovations.

At a recent Sensors Expo & Autonomous Vehicle Sensors Conference (AVSC) event, with in-cabin technology as part of the AVSC program, I spoke with a number of in-cabin technology pioneers, including AVSC chairperson Willard Tu, the head of the Xilinx automotive business unit, to get a glimpse of the future.

So what new functionalities are incabin innovations delivering? Currently, inside the cabin (in-cabin) automotive sensors have been installed to detect a collision and release an airbag to protect the driver and passengers. Other sensors can detect unlocked car or truck doors and sound an alarm. In the future, artificial intelligence (AI) and sensors integrated with HD cameras will help increase safety and comfort by monitoring driver alertness, for example, along with the in-cabin environment.

Xilinx FPGA

Xilinx, known for its leadership in FPGA technologies, is now applying AI, automotive technology, and programmable SoCs to the in-cabin environment. At CES 2019 Daimler announced the MBUX Interior Assist, found in the new GLE and CLA models. The MBUX Interior Assist uses the Xilinx Zynq UltraScale+ MPSoC to support gesture interpretation and response. The collaboration between Xilinx and Daimler will deliver an AI-based gesture input system that recognizes the occupants’ natural movements, so the vehicle can predict driver and passenger requests, distinguish between driver and passenger gestures, and react to body language to automate comfort functions.

Xilinx AI inference technology makes it possible to monitor vehicle occupants, identifying their emotions, gestures, and in-cabin environment preferences such as temperature and music. The MBUX Interior Assist can be programmed by the automakers to run proprietary image-processing algorithms. These algorithms make it possible for a camera to track the driver’s eyes and head position or other gestures. Designed to be thermally efficient, the unit can be positioned in a number of places, such as the inside of the advanced driver assistance system (ADAS) central processing unit, inside the car, or on the windshield.

“The AI evolution is driving the in-cabin experience as in-vehicle driver and occupant monitoring systems rely on AI inference to quickly and accurately identify occupants, body position, state of health, eye gaze, head pose, gestures, and even emotions which will enable features such as personalization, security and safety,” said Willard Tu, senior director, automotive, Xilinx. “Driver monitoring systems will be the genesis of the In-Cabin AI proliferation. Xilinx devices provide an adaptable platform for AI inference which will allow each OEM to customize the in-cabin experience.”

A unique Xilinx FPGA and MPSoC feature, Dynamic Function eXchange, enables the same device to be reconfigured to handle multiple, mutually exclusive functions to maximize silicon space.

Figure 2. Veoneer’s AI-based Vehicle Trust Solutions depends on machine vision to see with LiDAR, HD camera and radar. (Source: Veoneer)

AI-based Vehicle Trust Solutions

As drivers, we make decisions all the time.

  • Should I brake now?

  • If someone suddenly jumps in front of my car, should I stop or turn, and in which direction?

  • If two objects appear in front of my car, and I know I won’t be able to stop in time to avoid the collision, which object should I hit?

Our brains perform millions of calculations very quickly and, sometimes, our instincts take over. In the case of assisted driving or, eventually, fully autonomous driving, will “drivers” be able to trust the machine to make all the right decisions?

Based in Stockholm, Sweden, Veoneer Research, which focuses on collaborative and automated driving, offers products based on the concept that a real trust between human “drivers” and cars can be developed over time with an AI-based, human-machine interface. In other words, AI will apply learned algorithms to determine what the human beings are thinking and make decisions to interact accordingly. For example, drivers tend to get distracted by events occurring outside the vehicle. AI technology can be used to alert the driver to focus on driving.

Currently, Veoneer is developing technologies to monitor activities both within and outside the vehicle to achieve SAE Level 4 (ADAS). SAE International sets a 5-level standard for autonomous driving. Level 5 is, hands-off, fully autonomous driving while Level 4 is almost fully autonomous but still requires driver engagement.

The company’s Learning Intelligent Vehicle (LIV 3.0) platform aims to develop solutions for humans and machines to interact and learn together. These deep learning algorithms develop effective communication based on sensing the direction of the driver’s gaze, emotion, cognitive load, drowsiness, hand position, and posture, and then uses this information with data about the external environment to enhance safer driving experiences. Additionally, Veoneer is the first manufacturer to develop a thermal camera for use in SAE Level 4 driving. With such a camera, objects not seen by the naked eye can be detected and appear on the driver’s screen, giving advanced warning to the driver as well as preparing the vehicle to slow down.

“Veoneer designs and compiles state-of-the-art software, hardware, and systems that make driving safer, and provides the key technologies that drive the reliability, adoption, and success of self-driving mobility solutions. Our products are adaptative and will continue to improve. Therefore, using FGPAs will allow us to be flexible in developing future trust-based vehicle technologies,” commented Thomas Herbert, Product Director Emerging Business, Veoneer.

Cameras use the infrared spectrum just outside of human detection. This requires special illumination to produce the best results and support nighttime driving without blinding the driver. Illumination typically is separated from the camera to reduce IR “red eye” (bright pupil), which improves detection performance.

In-Vehicle Scene Understanding

Founded in 2013, the Palo Alto, California-based AI startup, Eyeris, focuses on in-vehicle scene understanding. The company’s vision AI system uses a portfolio of advanced Deep Neural Networks (DNNs) and Interior Image Segmentation™ technology. The EyerisNet portfolio of DNNs uses multiple automotive-grade 2D RGB-IR image sensors. It targets modern automotive-grade FPGA-based chips, among other AI chip compute architectures, for real-time data analytics on the edge to optimize vehicle safety and comfort. Unlike general-purpose GPUs, FPGA-based and ASIC-based AI chips can enable a high level of programmability that caters to specific application workloads in order to optimize efficient inference for most of today’s neural networks.

Figure 3. EyerisNet’s vision AI system, equipped with multiple automotive-grade 2D RGB-IR image sensors, is able to detect occupants’ human emotion. (Source: Eyeris)

Over the last five years, Eyeris has collected over 10 million in-cabin images at its San Jose R&D lab to train its portfolio of 10 DNNs and has successfully demonstrated a fully integrated multi-camera in-cabin monitoring solution inside its Tesla Model S demo vehicle. This solution won two AutoSens 2019 Awards in September for “Best Automotive Safety System” and “Most Innovative In-cabin Application.” Eyeris is also a winner of the 2017 TU-Automotive Awards Detroit for its driver and occupant monitoring AI.

The company achieves an understanding of the entire in-vehicle space due to its unique approach to simplifying the classification of any passenger vehicle interior environment into three distinct categories: people, objects and surfaces. The Eyeris human behavior understanding module includes a comprehensive suite of DNNs for upper body tracking and face analytics, along with recognition of activities such as texting, eating, drinking, sleeping, and dancing.

The object detection module caters more to the use case of “forgotten objects left behind” during ride-sharing or public transit. Additionally, the vision AI is able to discern that a bottle of water in hand means a person is drinking and not eating. Lastly, the Eyeris surface classification module includes seats, footwells, door panels, windows, instrument cluster, steering wheel, armrest, etc.; understanding these surfaces creates a point of reference for the in-vehicle scene-understanding AI. For example, knowing the exact upper body height, width, posture and orientation of the front passenger, along with the exact location of his right shoulder and elbow, can trigger dynamic deployment of airbags with the right velocity, intensity, and angle in order to optimize passenger safety and protection.

Looking Ahead

New technologies based on FPGAs and SoCs will drive innovative automotive in-cabin applications. Additionally, artificial intelligence (AI) and various sensors, including HD cameras and motion sensing, will provide the benefits of comfort and safety.

Able to be used in many different ways, the Xilinx AI inference technology platform monitors vehicle occupants and detects their emotions, gestures, and interior preferences. Veoneer applies FPGAs and SoCs and the Learning Intelligent Vehicle (LIV 3.0) platform, along with a deep learning algorithm that can be used in vehicles to sense the direction of the driver’s gaze as well as emotion, cognitive load, drowsiness, hand position, and posture. With these data, the vehicle will be able to assist the drivers with alerts and with additional technologies in order to promote safety. Using automotive-grade FPGA-based chips among other AI chip compute architectures, for real-time data analytics, EyerisNet combines an AI system and advanced DNNs to make good decisions for vehicles.

It is expected that more and more in-cabin applications will be added to vehicles. Potentially, life-saving designs such as position sensing airbag deployment will be able to “fire” in real-time with precision, based on data collected on the passengers’ position and posture.

This article was written by John W. Koon, Contributing Editor, Photonics & Imaging Technology.