Song Gesture Recognition for a Robot-Enhanced Imitation Therapy

Robot-Enhanced Therapies (RET) offer a promising alternative for Autism Spectrum Disorder (ASD) children. Within this framework, an imitation therapy is proposed where children replicate gestures demonstrated by a robot, including those derived from popular children songs. To enable effective feedback provision by the robot, gesture recognition becomes paramount. This paper introduces two approaches for achieving this goal: a multiclass classifier, intended to recognize which gesture is executed, and a set of binary classifiers discerning whether the expected gesture is performed or not. Both models rely on kinematic data acquired through the Azure Kinect camera and a Residual Network as classification model. Moreover, considering the challenges in children's data collection for model training, the work explores the impact that the collection of their data can bring to the outcomes of a gesture classification algorithm. Beside testing on adults and healthy children datasets, the study leverages a dataset comprising 69 gestures performed by ASD children during therapy sessions. Results indicate that binary models trained also on children data outperform the multiclass approach trained solely on adults data, achieving a median accuracy of 86%. This underscores the effectiveness of binary classifiers and suggests that integrating children's data can enhance algorithm performance. To strengthen findings, future research should expand dataset size, especially considering ASD children, and explore alternative action recognition algorithms like Long Short-Term Memories (LSTM).