High-Volume, High-Quality ADAS Annotation Done Right  

 

1.2 million 3D object annotations labeled at 99% quality in one month. Learn how understand.ai helped Mercedes-Benz AG  to make a training and validation project a success story.

End of the year approaching. Other projects to follow. Required quality of annotations - 99%.

The goal was to train the perception AI for a radar sensor and validate its correct functioning. The reference sensor in this case was a lidar.  Pressured by the budget allocation running out at the end of the year, annotating more than a million objects within a month was just the first of the challenges. The contractual quality target for the understand.ai delivered annotations as part of the publicly funded project @CITY was set at 99%. 

 

The required annotations were bounding boxes for cars and polyline annotation for curbstones in a 3D sensor fusion environment.  

 

The customer was looking for an annotation service provider who can master the throughput task, deliver desired quality target and support a specific type of polyline annotation. Data security and compliance with GDPR was non-negotiable. After surveying online and sourcing recommendations from previous experience with data annotation providers in-house, understand.ai (UAI) was selected for its combination of annotation capabilities and favorable price-performance ratio.

UAI-Annotator-MB-3D-Annotation_2

 

 

 

 

 

 

 

 

anonymizer

Secure and GDPR Compliant

The project launched in November 2020 with Step 0. To be compliant with any data protection laws, algorithm powered UAI Anonymizer was used to blur all faces and license plates in an automated fashion. TLS (Transport Layer Security) protocols, authentication and authorization procedures were established to guarantee safe handling of customer data.

noun_easy_UAI-2-2

Easy Annotation Quality Control

Annotation process is a two-way road involving quality assurance and frequent data checks, adjustments, and the ability to align data processing between the ADAS development team and the annotation tooling. The browser based UAI Annotator tool offered a convenient way to review labels shortening the feedback loops and time to accepted annotation batches. 

3Dvs2D-blog_box-sizes_2

Tailored 3D Annotation and an Agile Project Management

The project’s goal was to label a large volume of objects within a short time with an added challenge of specialty annotation in a 3D sensor fusion set-up. The dimensions and the complexity required:  

  • a frictionless review loop between Mercedes-Benz and UAI,
  • agile feature customization,
  • speed in all steps of the annotation process - from project management to quality assurance.

For 3D curbstone annotation a customized “birds eye view” approach was developed. The UAI project team managing the annotation process and the labeling partner enabled smooth communication and agile change management throughout the entire project.  The Quality Assurance team took care of the ground truth quality of the output. 

stock_square_cropped

Flexible Annotation Volume Management

Thanks to the tooling's ability to separate static and dynamic objects en masse and track & control them via a dashboard feature, the project managers were able to put criteria-based volume caps to control throughput and the overall number of desired annotations. For example, non-essential static objects beyond a certain distance from the ego vehicle were filtered out, reducing the overall number of annotations. 

Still, the resulting number of objects to be annotated was too large to handle without a considerable automation rate. The annotation process would take either too long or would become unaffordable when adding more human annotators to achieve the desired throughput.

Higher Throughput Thanks to Higher Automation

The annotation automation engine inside the UAI Annotator delivered throughput rates that made the project feasible within the ambitious timelines. The performance of these algorithms was enhanced by several factors contributing to higher automation rate:

  • highly calibrated GPS & odometry data 
  • well positioned & high resolution sensors
  • reasonable labeling specifications
  • labeling of objects within LiDAR range 

The automation approach of UAI Annotator combined with a fitting sensor & project set-up resulted in an impressive throughput of 1.2 million objects annotated at 99% quality within a month.

UAI-Annotator-MB-3D-Annotation_1

ADAS Annotation Done Right

Autonomous driving functions have an undisputed place in the R&D pipelines of all automotive OEM companies. But even in the case of lower autonomy levels, the safety concerns and the complexity of the vehicle perception stacks make the data annotation projects a high effort undertaking. Annotation for ADAS / AD is more than a plug and play tooling solution.

 

understand.ai combined a highly automated tooling with an agile custom development and project management services to deliver a large quantity of high-quality 3D annotations within a short time. Thanks to the high automation level and flexible adjustments, assisted by the convenient procedure of label review, Mercedes-Benz was able to become faster while holding onto the project's 99% quality target.