1.2 million 3D object annotations labeled at 99% quality in one month. Learn how understand.ai helped Mercedes-Benz AG to make a training and validation project a success story.
The goal was to train the perception AI for a radar sensor and validate its correct functioning. The reference sensor in this case was a lidar. Pressured by the budget allocation running out at the end of the year, annotating more than a million objects within a month was just the first of the challenges. The contractual quality target for the understand.ai delivered annotations as part of the publicly funded project @CITY was set at 99%.
The required annotations were bounding boxes for cars and polyline annotation for curbstones in a 3D sensor fusion environment.
The customer was looking for an annotation service provider who can master the throughput task, deliver desired quality target and support a specific type of polyline annotation. Data security and compliance with GDPR was non-negotiable. After surveying online and sourcing recommendations from previous experience with data annotation providers in-house, understand.ai (UAI) was selected for its combination of annotation capabilities and favorable price-performance ratio.
The project launched in November 2020 with Step 0. To be compliant with any data protection laws, algorithm powered UAI Anonymizer was used to blur all faces and license plates in an automated fashion. TLS (Transport Layer Security) protocols, authentication and authorization procedures were established to guarantee safe handling of customer data.
Annotation process is a two-way road involving quality assurance and frequent data checks, adjustments, and the ability to align data processing between the ADAS development team and the annotation tooling. The browser based UAI Annotator tool offered a convenient way to review labels shortening the feedback loops and time to accepted annotation batches.
The project’s goal was to label a large volume of objects within a short time with an added challenge of specialty annotation in a 3D sensor fusion set-up. The dimensions and the complexity required:
For 3D curbstone annotation a customized “birds eye view” approach was developed. The UAI project team managing the annotation process and the labeling partner enabled smooth communication and agile change management throughout the entire project. The Quality Assurance team took care of the ground truth quality of the output.
Thanks to the tooling's ability to separate static and dynamic objects en masse and track & control them via a dashboard feature, the project managers were able to put criteria-based volume caps to control throughput and the overall number of desired annotations. For example, non-essential static objects beyond a certain distance from the ego vehicle were filtered out, reducing the overall number of annotations.
The annotation automation engine inside the UAI Annotator delivered throughput rates that made the project feasible within the ambitious timelines. The performance of these algorithms was enhanced by several factors contributing to higher automation rate:
The automation approach of UAI Annotator combined with a fitting sensor & project set-up resulted in an impressive throughput of 1.2 million objects annotated at 99% quality within a month.
Autonomous driving functions have an undisputed place in the R&D pipelines of all automotive OEM companies. But even in the case of lower autonomy levels, the safety concerns and the complexity of the vehicle perception stacks make the data annotation projects a high effort undertaking. Annotation for ADAS / AD is more than a plug and play tooling solution.
understand.ai combined a highly automated tooling with an agile custom development and project management services to deliver a large quantity of high-quality 3D annotations within a short time. Thanks to the high automation level and flexible adjustments, assisted by the convenient procedure of label review, Mercedes-Benz was able to become faster while holding onto the project's 99% quality target.