Virtual simulation benchmark for the evaluation of simultaneous localization and mapping and 3D reconstruction algorithm uncertainty

Simultaneous localization and mapping (SLAM) algorithms allow us to obtain a unique 3D shape and 3D sensor trajectory by combining partial scans obtained by moving a 3D scanner. The performances of these algorithms are significantly affected by experimental conditions, characteristics of the target and values of the parameters of the reconstruction algorithm. Therefore, the uncertainty and reliability of SLAM techniques need to be assessed before their application, e.g. for robot navigation, autonomous vehicles or industrial fields. To evaluate the uncertainty of these algorithms, specific datasets containing 3D scans, with the possibility to control different conditions, e.g. sensor trajectory, depth or color noise, sensor velocity and framerate, are necessary. In this article, we present a procedure to obtain virtual datasets with complete control of the environment, 3D sensor and trajectory conditions, starting from any real 3D dataset acquisition, characterized by a sufficiently low uncertainty. These datasets can be generated to test the effect of SLAM algorithm parameters to determine the best parameters to be used to exploit the algorithm characteristics to obtain the best result in each operating context. The advantage of this procedure is the possibility to perfectly control each condition and to evaluate its effect on the final result. This procedure was applied to two reconstruction algorithms as examples; namely, the Open3D reconstruction tool and ElasticFusion. The results demonstrate that the setting of algorithm parameters, e.g. the tolerance on depth correspondence between frames or the number of fragments, or the change in number of frames acquired, can have a strong influence on the resulting 3D reconstruction and trajectory. Moreover, the effect of not closing the loop trajectory on reconstruction performance is quantified for different application scenarios.