Body and Space: Combining Modalities for Musical Expression

Copyright is held by the author/owner(s). TEI 2013, February 10-13, 2013, Barcelona, Spain ACM Abstract This paper presents work in progress on applying a Multimodal interaction (MMI) approach to studying interactive music performance. We report on a study where an existing musical work was used to provide a gesture vocabulary. The biophysical sensing already used in the work was used as input modality, and augmented with several other input sensing modalities not in the original piece. The bioacoustics-based sensor, accelerometer sensors, and full-body motion capture system generated data recorded into a multimodal database. We plotted the data from the different modalities and offer observations based on visual analysis of the collected data. Our preliminary results show that there is complementarity of different forms in the information. We noted three types of complementarity: synchronicity, coupling, and correlation.