ODIN: Pluggable Meta-annotations and Metrics for the Diagnosis of Classification and Localization

Machine Learning (ML) tasks, especially Computer Vision (CV) ones, have greatly progressed after the introduction of Deep Neural Networks. Analyzing the performance of deep models is an open issue, addressed with techniques that inspect the response of inner network layers to given inputs. A complementary approach relies on ad-hoc metadata added to the input and used to factor the performance into indicators sensitive to specific facets of the data. We present ODIN an open source diagnosis framework for generic ML classification tasks and for CV object detection and instance segmentation tasks that lets developers add meta-annotations to their data sets, compute performance metrics split by meta-annotation values, and visualize diagnosis reports. ODIN is agnostic to the training platform and input formats and can be extended with application- and domain-specific meta-annotations and metrics with almost no coding. It integrates a rapid annotation tool for classification and object detection data sets. In this paper, we exemplify ODIN through CV tasks, but the tool can be used for generic ML classification.