Audiovisual performance is a fertile area for creative expression, an intersection of experimental cinema and computer music that has seen a groundswell of interest in recent years. To create works in this emerging medium, a complex network of relationships between sounds, images and sensor input must be organized. This complexity poses major technical and aesthetic challenges which a systemati...

............................................................................................... v Introduction............................................................................................. 1 Review of the literature ............................................................................... 3 Auditory speech perception ............................................................

Modern interactive means combined with new digital media processing and representation technologies can provide a robust framework for enhancing user experience in multimedia entertainment systems and audiovisual artistic installations with non-traditional interaction/feedback paths based on user affective state. In this work, the ‘Elevator’ interactive audiovisual platform prototype is present...

Many of today’s audiovisual application systems offer some kind of interactivity. Yet, quality assessments of these systems are often performed without taking into account the possible effects of divided attention caused by interaction or user task. We present a subjective assessment performed among 40 test subjects to investigate the impact of divided attention on the perception of audiovisual...

Speech is the most important form of human communication but ambient sounds and competing talkers often degrade its acoustics. Fortunately the brain can use visual information, especially its highly precise spatial information, to improve speech comprehension in noisy environments. Previous studies have demonstrated that audiovisual integration depends strongly on spatiotemporal factors. Howeve...

Many organisms sample their environment through multiple sensory systems and the integration of multisensory information enhances learning. However, the mechanisms underlying multisensory memory formation and their similarity to unisensory mechanisms remain unclear. Filial imprinting is one example in which experience is multisensory, and the mechanisms of unisensory neuronal plasticity are wel...

In a recent study in younger adults (19-29 year olds) we showed evidence that distributed audiovisual attention resulted in improved discrimination performance for audiovisual stimuli compared to focused visual attention. Here, we extend our findings to healthy older adults (60-90 year olds), showing that performance benefits of distributed audiovisual attention in this population match those o...

This paper reports results of a study investigating the visual information conveyed by the dynamics of internal articulators. Intelligibility of synthetic audiovisual speech with and without visualization of the internal articulator movements was compared. Additionally speech recognition scores were contrasted before and after a short learning lesson in which articulator trajectories were expla...

This document describes the working environment developed by ESCoM laboratory (Equipe Sémiotique Cognitive et Nouveaux Médias, Fondation Maison des Sciences de l’Homme) during the ASA-SHS project (Audiovisual Semiotic Workshop for Human and Social Sciences). ASA-SHS is a R&D project financed by the French National Research Agency (ANR, Programme Blanc 2008) and targets to develop strategies of ...

We propose residUUm, an audiovisual performance tool that uses sonification to orchestrate a particle system of shapes, as an attempt to build an audiovisual user interface in which all the actions of a performer on a laptop are intended to be explicitly interpreted by the audience. We propose two approaches to performing with residUUm and discuss the methods utilized to fulfill the promise of ...