Autonomous robot missions are typically tested through field experiments. This approach is very costly and can be risky. In order to explore more operational situations at lower cost and without incurring risk, our work is developing simulation tests: the robot carries out its missions in virtual worlds. The research was carried out in collaboration with the RIS team in the robotics department. They were supported by two case studies of autonomous outdoor navigation: the navigation of Mana, an experimental all-terrain robot developed at LAAS, and that of Oz, an agricultural weeding robot developed by Naïo Technologies.

Testing in virtual worlds requires us to revisit the issues of input generation and output analysis. In this work, test generation is based on a model of worlds that includes a UML structural view of world elements, and a formal grammar defining the parameters for generating these elements. Random processes are implemented at two levels: firstly, to produce valid parameter configurations, i.e. those that conform to the grammar, and secondly, to create test scenes in a format that can be understood by the simulator. Test traces can be analyzed either quantitatively or qualitatively. In the first case, the robot's difficulty is evaluated in terms of the percentage of mission failures, the time taken to complete the mission, or trajectory detours. In the second case, the focus is on detecting abnormal behaviors such as collisions, crossing a speed threshold, perception errors, or behaviors specific to the current mission phase. An important question concerns the effectiveness of testing, in an environment that simplifies real-world physics.We have studied this question for a sample of faults extracted from Mana's development history.Most of these faults do not require fine reproduction of physical aspects and can be found by low-fidelity simulation testing.The effectiveness of simulation testing was confirmed for Oz, in an experiment involving parallel simulation and field testing.Despite the simplifications induced by simulation, our tests provided a good match with most of the failures observed in the field, and even highlighted an unknown case confirmed by the manufacturer.

[1] Sotiropoulos T., Waeselynck H., Guiochet J., Ingrand F., “Can robot navigation bugs be found in simulation? An exploratory study”, IEEE International Conference on Software Quality, Reliability & Security (QRS 2017), Prague, République Tchèque, 2017.