Working towards a unified framework for joint audio &visual analytics


Data analysis, such as the analysis of production processes, includes a number of activities that cannot be automated completely. Therefore, the work of human analysts is often required.  Both information visualization and sonification – the representation of data with images or sounds – are considered effective methods to involve humans in data analysis.  These two fields utilize the highly sophisticated visual and auditory information processing capabilities of humans.  While information visualization approaches explore computer-based, interactive visual representations of data, sonification illustrates information through the use of nonspeech audio.  With the increasing amount and complexity of data, both approaches have reached their limits, which is also the case for the human perceptual capacities serving as the basis for interpreting images and sounds.

Although extensive research has been carried out on auditory and visual representation of data, there have been few attempts to combine these two channels in a systematic and complementary manner. So far, not enough attention has been paid to the interrelation of these modalities. Existing research on combinations has often focused on one modality while neglecting the other. A methodical approach to combine information visualization and sonification with regard to a complementary framework is still missing. Due to the rapidly increasing amounts of complex data to be analyzed, each individual approach reaches its limits. For this reason, more capable methods are urgently needed.

Project content and aims

SoniVis aims to bridge the gap between sonification and visualization. To do this, we are working on creating a basis for a unified design theory for audio-visual analytics that combines these two modalities in a complementary way for exploratory data analysis. Within the framework of this project, we 

  • will outline the range of design options for complementary audio-visual data analysis approaches,
  • examine the practical feasibility in the form of a design study based on the analysis of production processes, and 
  •  conduct a controlled experiment to empirically compare different complementary audio-visual analytics approaches.

Method & Results

The project follows a human-centred and problem-driven research process. It is based on the paradigm of design science research and closely intertwines design and evaluation phases. Therefore, approaches can be reviewed promptly, mitigating the risk of non-informative results.
Expected research outputs are primarily design principles (core principles and concepts to guide design) and instantiations (digital artifacts, i.e. situated implementations in certain environments). Based on this, the researchers work towards a design theory for integrating interactive visualization and sonification for data analysis tasks.

Press Coverage

"Sehen und Hören in Einklang bringen"

Medium: Die Presse
Author: Veronika Schmidt


Enge, K. (2021, October). Exploratory Data Analysis using Multimodal Representations: Joint Sonification and Visualization - IEEE VIS Doctoral Colloquium [Presentation]. IEEE VIS Doctoral Colloquium, Virtual Meeting.
Enge, K. (2021, June 25). Exploratory Data Analysis using Multimodal Representations: Joint Sonification and Visualization - ICAD Student Think Tank 2021 [Presentation]. ICAD Student Think Tank 2021, Virtual Meeting.
Enge, K. (2021, June 19). Exploratory Data Analysis using Multimodal Representations: Joint Sonification and Visualization - KUG Doktorand*innen Forum [Presentation]. KUG Doktorand*innen Forum, Graz (Austria).
Enge, K. (2021, June 18). Listening experiment on the plausibility of acoustic modeling in virtual reality [Invited Talk]. All Around Audio Symposium 2021, St. Pölten, Austria.
Enge, K., Wasserfall, S., & Frauscher, E. (2021, December 5). Workshop on the Virtual Urban Soundscape Designer [Workshop]. klingt gut! symposium on sound 2021, Hamburg.
Aigner, W. (2021, March 23). Visual Analytics for Time-Oriented Data [Invited Talk]. ERFA Industrial Data & Analytics - Zeitreihenanalyse, Kremsmünster, Austria.
Enge, K., Rind, A., Iber, M., Höldrich, R., & Aigner, W. (2021). It’s about Time: Adopting Theoretical Constructs from Visualization for Sonification. Proceedings of the 16th International Audio Mostly Conference (AM"21), 64–71.
Aigner, W., Enge, K., Iber, M., Rind, A., Elmqvist, N., Höldrich, R., Rönnberg, N., & Walker, B. N. (2021). Workshop on Audio-Visual Analytics – Identifying Research Gaps for Integrating Sonification and Visualization. Audio Mostly Workshops.
Involved Institutes, Groups and Centers
Institute of Creative\Media/Technologies
Research Group Media Computing