Multimodal Interaction Experience for Users with Autism in a 3D Environment

This paper describes the multimodal 3D game-based learning environments, accessible to users with autism, which we designed, built, and tested. Our innovative interaction system proposes a combination of visual, voice, and textual modalities able to guide the user through the 3D environment and allowing her/him to access contents in a multimodal and personalised way. The design process had to consider the different modalities users could have used to access the contents, and this multimodal user interface had implications on accessibility. By taking care of the multimodality of the realised environment, and of the relational context, children and adults with autism can have the unique opportunity of bypassing some of the difficulties they encounter during their social life, such as visual contact, unwieldy movements, and face-to-face interactions. From this point of view, shared 3D virtual environments with multimodal interaction interfaces represent an important opportunity able to improve the social life of persons with autism. The entire structure of the work has been characterised by five main pillars and innovative aspects: multimodal strategies to explore the environment and to access contents; user model; user profiles; personalisation of the contents; experimentation/validation. The proposed user model (based on the ICF* specification, a customised, extended version of the WHO International Classification of Functioning, Disability and Health guidelines - ICF) allowed an extensible, detailed personalisation, as different attributes described users both from the static and the dynamic points of view. Users with different levels of autism were profiled using our model, and involved in the experiment, permitting us to validate the effectiveness of the global approach. The results of such experiment showed us that indeed users appreciated the system and were able to take advantage of all educational opportunities the system provided.