The MICCAI 2022 associated challenge called 3DTeethSeg22 (https://3dteethseg.grand-challenge.org/) has been running for quite some time now. We are thankful for participating in this challenge since it promotes usable automatic 3D teeth segmentation techniques around the world.
However, there are some considerations regarding the training dataset provided for the usability for training the AI. Teams of participants that are aware of the considerations of the dataset and have the necessary software to address these considerations might having an advantage over other teams. We want to raise awareness of these considerations for the organizers and the participants as well as to spark dialogue on how to further enhance the challenge.
If all participants and teams would have access to the training dataset without these considerations and issues a more level playing field in this challenge could be possible. We (and perhaps other teams in parallel have done the same) cleaned the training dataset and used this for training. We already noticed that this has greatly impacted the scores on the “Preliminary Test : 3D Teeth Segmentation and Labeling Leaderboard” of the challenge.
Since for internal forum posts on the grand-challenge system it is impossible to add images or tables we add this page to support the forum post.
Affected scans, considerations & issues
| Affected Jaw | n | % |
|---|---|---|
| Upper Jaw | 186 | 31.0% |
| Lower Jaw | 155 | 25.8% |
| Total | 341 | 28.4% |
| Issue / Consideration | # of affected scans | % of scans |
|---|---|---|
| Mixed (deciduous/permanent) dentition | 279 | 23.3% |
| Incorrect Annotation(s) | 114 | 9.5% |
| Missing annotation(s) | 104 | 8.7% |
| Bad segmentation(s) | 85 | 7.1% |
| Label adjustment | 45 | 3.8% |
| Braces | 6 | 0.5% |
| Ambiguity | 2 | 0.2% |
| Overall Chaos ( > 3 issues) | 2 | 0.2% |
Example images
Below there are a set of example images from the original dataset showing a sample of some of the issues/considerations. Do note that these samples were found and marked during our dataset scouting stage and may have some incorrect assumptions scribbled in them. All scans have later on been systematically checked and corrected by our team if necessary. The re-annotation of the incorrect segmentations were fixed using in-house software.
On behalf of the “Chompers” team:
- Steven Kempers
- Niels van Nistelrooij
- Shankeeth Vinayahalingam
- Guido de Jong