As Advanced Driver Assistance System (ADAS) technology evolves, how do you test autonomous cars that ‘learn from their mistakes?’
During this month’s Tech Briefs webcast presentation, “The Evolution of ADAS,” an attendee asked Douglas Farrell from National Instruments, an Austin, TX-based manufacturer of automated test equipment and virtual software:
With deep learning, it would seem like self-driving vehicles would continuously improve the level of safety of their operations and "learn from their mistakes." How then do you set a pass/fail threshold that may be ever-changing?
Douglas Farrell: One of the things that you have to worry about with machine learning is that you’re not sure even what the things are that you're going to have to test, because the software is constantly evolving. You're going to need define scenarios, and expected outcomes for each scenario. You'll need to make sure that the vehicle is meeting whatever expected outcome you had.
As you mentioned, these outcomes could be changing and improving over time, and you’re going to have to be mindful and cognizant of that. We’ll have to be heavily involved in making sure that we've defined our test cases and scenarios, and the expected outcomes of these scenarios, and that they're always being modified according to how the latest state-of-the-art technology is saying that these vehicles should be operating.
We will always know that a vehicle should stay in its lane no matter what — unless you’re changing lanes. That's a pass/fail scenario that is easily stated: If the vehicle swerves out of its lane, that's a fail.
As we start talking about failures interoperating between systems, it's going to be a lot more complicated. We're going to have to look at how different components of the vehicle think that we passed or failed the system. For example: Did multiple sensor types detect the same issue in the road, or do we think that that was a failure in one of the subsystems? We're going to have to look at how the holistic vehicle defines the pass/fail scenarios for how the vehicle is behaving. Again, that's an argument for the need for system integration testing.
What do you think? Share your comments and questions below.
Also: Read more Automotive tech briefs.