Sonification and visualization are data representation techniques that share similar modes and purposes but operate through distinct sensory modalities. While visualization has already reached a scientific formalization, in data sonification, the construct of meaning is hampered by the lack of perceptual codes derived from shared modes of perception. We use visualization as a benchmark to assess data sonification theory and define the basis of a universally shared acoustic code. Our analysis relies on the relationship between visual and auditory contexts in the sonification projects produced by the design community itself. The research considers a collection of 80 audiovisual sonifications, i.e. data representation projects that integrate consciously and equally sound and video layers. The phenomenological analysis conducted on the selected projects leads to the identification of cross-sensory correspondences that take part in the translation process of audio and visual sensory contexts. Our analysis shows that in everyday listening conditions a sensorial exchange is activated, therefore the congruence of cross-sensory stimuli improves the understanding of space-time qualities. When designing a system, it is appropriate to focus primarily on features already familiar to the user. In order to explore phenomena from different perspectives, intersections, translations, contaminations, and analogies become necessary tools. Our study supports the transition to representations closer to everyday experience, leveraging qualitative references to the senses and their interaction in a cross-modal environment. The analysis results are visually synthesized in a cross-sensory map to design intersubjective values. Such a map is the result of an experimental approach. It delineates a shared space for both sonification experts and designers to define guidelines and shared practices. It aims to facilitate a coherent relationship between sensory registers in data representation.

A cross-sensory Map. Analysis between Visual and Auditory Correspondences in Audiovisual Sonification

Caiola, Valentina;Lenzi, Sara;Ricco', Dina
2022-01-01

Abstract

Sonification and visualization are data representation techniques that share similar modes and purposes but operate through distinct sensory modalities. While visualization has already reached a scientific formalization, in data sonification, the construct of meaning is hampered by the lack of perceptual codes derived from shared modes of perception. We use visualization as a benchmark to assess data sonification theory and define the basis of a universally shared acoustic code. Our analysis relies on the relationship between visual and auditory contexts in the sonification projects produced by the design community itself. The research considers a collection of 80 audiovisual sonifications, i.e. data representation projects that integrate consciously and equally sound and video layers. The phenomenological analysis conducted on the selected projects leads to the identification of cross-sensory correspondences that take part in the translation process of audio and visual sensory contexts. Our analysis shows that in everyday listening conditions a sensorial exchange is activated, therefore the congruence of cross-sensory stimuli improves the understanding of space-time qualities. When designing a system, it is appropriate to focus primarily on features already familiar to the user. In order to explore phenomena from different perspectives, intersections, translations, contaminations, and analogies become necessary tools. Our study supports the transition to representations closer to everyday experience, leveraging qualitative references to the senses and their interaction in a cross-modal environment. The analysis results are visually synthesized in a cross-sensory map to design intersubjective values. Such a map is the result of an experimental approach. It delineates a shared space for both sonification experts and designers to define guidelines and shared practices. It aims to facilitate a coherent relationship between sensory registers in data representation.
2022
Actas del VII Congreso Internacional de Sinestesia: Ciencia y Arte. El Reto físico/digital // VII International Congress Synaesthesia: Science and Art. The digital / physical challenge
978-84-948665-6-2
Cross-sensory data representation
Visualization
Data sonification
Sensory correspondences
Perceptual codes
File in questo prodotto:
File Dimensione Formato  
2022_Riccò_Caiola_Lenzi_SYN2022_pp371-381.pdf

Accesso riservato

: Publisher’s version
Dimensione 7.31 MB
Formato Adobe PDF
7.31 MB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11311/1224410
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact