Validation Projects in Autonomous Driving
Why are we differentiating between validation and training projects when it comes to autonomous driving functions? There are some profound differences between the two types of projects.
AI Training Project
A project with the goal to train a machine learning model on a certain task. To do so, data needs to be collected and annotated, so that the model can learn from it. In autonomous driving we are talking about perception algorithms that help self-driving functions to recognize their environment recorded by the vehicles' sensors.
AI Validation Project
A project with the goal to find out if a perception model works properly under real world conditions. Let's look into the details of validation project types and requirements.
There are 2 main routes for validating autonomous driving functions depending on what part of the autonomous driving stack is subject to validation.
Reference Sensor Set-up
- Step 1: The sensor under test and a reference sensor are used on the same car for a long distance of driving.
- Step 2: The sensor under test is doing its job as intended.
- Step 3: The reference sensor is used to produce a different view of the same driving situation.
- Step 4: Once the driving is done, the output of both sensors is compared against each other to find out if there is some serious malfunctioning going on which needs to get fixed.
Single Sensor Set-up
- Step 1: Only the sensor under test is used for a long distance of driving.
- Step 2: The sensor under test is doing its job as intended and data logging systems record the captured data.
- Step 3: Once the driving is done, the recorded data stream is processed twice. Once from the driving stack under test and once from the ground truth labelling team.
- Step 4: The sensor output is compared to the ground truth to find out if there is some serious malfunctioning going on which needs to get fixed.
01. Data Volume Requirements
A project with the goal to train an AI-model to perfection certainly requires a decent amount of data of high variance in different conditions. Yet, the amount of validation data to ensure proper functioning in an environment where low performance can lead to fatalities is orders of magnitudes higher than what will be needed for training.
We are talking billions of annotations for validation vs. millions of annotations for training.
To run an annotation project with billions of annotations successfully, the labeling tool suite needs to fulfil certain requirements:
Workflow Management
Flexible workflows which can be easily optimized for large project requirements
High Volume Data Processing
Rock-solid processing of petabytes of data
Automation
High automation rates reducing manual labour to make validation affordable
Ease of Use
Easy onboarding of new labelling experts whenever required
VIDEO: Why is automated labeling essential for autonomous driving?
Bringing sophisticated AI-based driving functions to the road safely requires a growing amount of annotated data. Watch Daniel explain what challenges are connected to the annotation process and what can be done with it.
02. Data Quality Requirements
An algorithm can only get as good as the data it was trained on. If quality requirements in terms of box tightness, consistency, completeness and coverage are not met, the model is likely to not be able to unfold its potential in real world scenarios. To achieve a potentially usable model requires high manual quality assurance efforts.
- Coverage (e.g. object detection, recall rates)
- Consistency (e.g. consistent classes)
- Correctness (e.g. clear class boundaries)
- Precision (e.g. box tightness)
Quality requirements for a validation project are shifted towards the coverage aspect (recall) compared to the precision focus in training.
A Matter of Perspective: 3D vs. 2D Sensor Data for Machine Perception
Sensors enabling machines to see have far surpassed human capabilities, yet their ability to 'connect the dots' has its limits. Does 'depth' make a difference?
Read in our blog analyzing the pros and cons of 3D sensor data use in machine learning.
TESTIMONIAL: Large Scale Annotation for the Future of Autonomous Driving
What does it take to get the highest throughput in large scale annotation? Learn how the engineering and R&D forerunner LTTS teamed up with UAI Annotator to give automotive customers an edge to lead in the autonomous driving race.
More to find in our large scale annotation tool testimonial here.
WEBINAR: Where to start with AI-powered annotation automation?
Discover in an on-demand webinar on how to automate 3D object annotation in autonomous driving. Let Daniel Rödler, Director of Product explain how understand.ai deploys AI to automate annotations and share years of experience working on autonomous driving annotation projects.
Watch the webinar on annotation automation now!
Semantic Segmentation, the Misunderstood Data Annotation Type
Why is semantic segmentation so essential for perception data annotation? Which type is the most useful for machine learning? What are caveats to it?
Discover more in our blog on semantic segmentation.
03. Data Diversity Requirements
Diversity is data is essential in both use cases, for training as well for validation you’d aim for a well-balanced and representative dataset of normal scenarios, edge cases and critical situations.
~ 94% of data is useless as it only covers “normal” driving, but > 90% of all the trouble is caused by critical events and edge cases
Making validation projects financially feasible
To run a successful validation project on huge amounts of data, watch out for a high automation rate and the right quality criteria. Then you’re all set for fast delivery at an affordable price point.
