Validation Projects in Autono­mous Driving

Why are we differentiating between validation and training projects when it comes to autonomous driving functions? There are some profound differences between the two types of projects. 

Neural Networks Training conceptual

AI Training Project

A project with the goal to train a machine learning model on a certain task. To do so, data needs to be collected and annotated, so that the model can learn from it. In autonomous driving we are talking about perception algorithms that help self-driving functions to recognize their environment recorded by the vehicles' sensors.

AI Validation Project

A project with the goal to find out if a perception model works properly under real world conditions. Let's look into the details of validation project types and requirements. 

There are 2 main routes for validating autonomous driving functions depending on what part of the autonomous driving stack is subject to validation. 

noun_sensor_UAI-1

Reference Sensor Set-up

  • Step 1: The sensor under test and a reference sensor are used on the same car for a long distance of driving. 
  • Step 2: The sensor under test is doing its job as intended. 
  • Step 3: The reference sensor is used to produce a different view of the same driving situation. 
  • Step 4: Once the driving is done, the output of both sensors is compared against each other to find out if there is some serious malfunctioning going on which needs to get fixed.
rotating_lidar

Single Sensor Set-up

  • Step 1: Only the sensor under test is used for a long distance of driving. 
  • Step 2: The sensor under test is doing its job as intended and data logging systems record the captured data. 
  • Step 3: Once the driving is done, the recorded data stream is processed twice. Once from the driving stack under test and once from the ground truth labelling team. 
  • Step 4: The sensor output is compared to the ground truth to find out if there is some serious malfunctioning going on which needs to get fixed.

01. Data Volume Requirements

A project with the goal to train an AI-model to perfection certainly requires a decent amount of data of high variance in different conditions. Yet, the amount of validation data to ensure proper functioning in an environment where low performance can lead to fatalities is orders of magnitudes higher than what will be needed for training.

shutterstock_1727114083

We are talking billions of annotations for validation vs. millions of annotations for training.

To run an annotation project with billions of annotations successfully, the labeling tool suite needs to fulfil certain requirements:

noun_Workflow_UAI

Workflow Management

Flexible workflows which can be easily optimized for large project requirements

noun_Big data_UAI

High Volume Data Processing

Rock-solid processing of petabytes of data

automation

Automation

High automation rates reducing manual labour to make validation affordable
noun_easy_UAI-2-2

Ease of Use

Easy onboarding of new labelling experts whenever required

webinar_teaser

WEBCAST: How to automate 3D object annotation in autonomous driving?

Watch the recorded webcast Where to start with AI-powered annotation automation. Let Daniel Rödler, Director of Product explain how understand.ai deploys AI to automate annotations and share years of experience working on autonomous driving annotation projects.

02. Data Quality Requirements

An algorithm can only get as good as the data it was trained on. If quality requirements in terms of box tightness, consistency, completeness and coverage are not met, the model is likely to not be able to unfold its potential in real world scenarios. To achieve a potentially usable model requires high manual quality assurance efforts.

  • Coverage (e.g. object detection, recall rates)
  • Consistency (e.g. consistent classes)
  • Correctness (e.g. clear class boundaries)
  • Precision (e.g. box tightness)
2D bounding boxes - annotated NYC street

Quality requirements for a validation project are shifted towards the coverage aspect (recall) compared to the precision focus in training.

3D bounding box of a car

A Matter of Perspective: 3D vs. 2D Sensor Data for Machine Perception

Sensors enabling machines to see have far surpassed human capabilities, yet their ability to 'connect the dots' has its limits. Does 'depth' make a difference?

Read in our blog analyzing the pros and cons of 3D sensor data use in machine learning.

noun_large-scale data_UAI

TESTIMONIAL: Large Scale Annotation for the Future of Autonomous Driving

What does it take to get the highest throughput in large scale annotation? Learn how the engineering and R&D forerunner LTTS teamed up with UAI Annotator to give automotive customers an edge to lead in the autonomous driving race.

More to find in our large scale annotation tool testimonial here. 

noun_fusion_uai

BLOG: The Case for Sensor Fusion for 3D Object Annotations

In machine perception, there’s one approach that utilizes the benefits of both 2D and 3D data while avoiding the downsides of both. It’s Sensor Fusion - a method of merging the information coming from two or more sensor sources to create a new representation of the environment - one superior to each of the individual ones.

Learn more here

sem-seg_1-2-crop-1

Semantic Segmentation, the Misunderstood Data Annotation Type

Why is semantic segmentation so essential for perception data annotation? Which type is the most useful for machine learning? What are caveats to it?

Discover more in our blog on semantic segmentation.

03. Data Diversity Requirements

Diversity is data is essential in both use cases, for training as well for validation you’d aim for a well-balanced and representative dataset of normal scenarios, edge cases and critical situations. 

~ 94% of data is useless as it only covers “normal” driving, but > 90% of all the trouble is caused by critical events and edge cases

Tell us what is important to you. What topics should we cover?

 

 

Making validation projects financially feasible

To run a successful validation project on huge amounts of data, watch out for a high automation rate and the right quality criteria. Then you’re all set for fast delivery at an affordable price point.

 
 
UAI data processing pipeline
 
UAI Annotator is the automation engine for data annotation. Tooling designed by AI-natives to cut costs and to accelerate the training and testing of your assisted & autonomous driving technology. Let us help with your next validation project. 

 

GET A DEMO AND SEE FOR YOURSELF