This chapter aims at describing how speech is multimodal not only in its perception but also in its production. It first focuses on multimodal perception of speech segments and speech prosody. It describes how multimodal perception is linked to speech production and explains why we consider speech perception as a sensory-motor process. It then analyses some aspects of hand-mouth coordination in...

The number of cars provided with an in-car communication system has considerably increased during the past few years. Using a mobile phone whilst driving is a safety-critical task and can cause usability issues. Speech modality has been incorporated in order to allocate hands and eyes solely to the driving task speech. This paper discusses an investigation into in-car communication systems and ...

1 Abstract We present two user evaluations of a speech centric multimodal Directory Assistance (DA) service that has been implemented on an iPAQ. The service combines speech and pen at the input side with text and speech at the output side. The first user experiment was an expert review. Five usability experts judged the initial design of the multimodal DA. Based on their comments a new design ...

We address the problem of segmentation and recognition of sequences of multimodal human interactions in meetings. These interactions can be seen as a rough structure of a meeting, and can be used either as input for a meeting browser or as a first step towards a higher semantic analysis of the meeting. A common lexicon of multimodal group meeting actions, a shared meeting data set, and a common...

Contemporary psychiatry is looking at affective sciences to understand human behavior, cognition and the mind in health and disease. Since it has been recognized that emotions have a pivotal role for the human mind, an ever increasing number of laboratories and research centers are interested in affective sciences, affective neuroscience, affective psychology and affective psychopathology. Ther...

In the MATIS project a multimodal system has been developed for train timetable information. The aim of the project was to obtain guidelines for designing multimodal interfaces for information systems. The MATIS system accepts input both in spoken and in graphical mode (no keyboard input) and provides feedback in the same two modes. The user can choose at any time which of the input modalities ...

The superior colliculus is organized topographically as a neural map. The deep layers of the colliculus detect and localize targets in the environment by integrating input from multiple sensory systems. Some deep colliculus neurons receive input of only one sensory modality (unimodal) while others receive input of multiple modalities. Multimodal deep SC neurons exhibit multisensory enhancement,...

So far, not much is known on the relationship of quality aspects of multimodal dialog systems. This paper aims at closing this gap by analyzing the influence of input and output modalities on the systems’ usability. The underlying study has been carried out with a smart-home system offering speech, gesture and touch as well as the combination of these three for input and a speech-to-text system...

Systems that attempt to understand natural human input make mistakes, even humans. However, humans avoid misunderstandings by confirming doubtful input. Multimodal systems--those that combine simultaneous input from more than one modality, for example speech and gesture--have historically been designed so that they either request confwmation of speech, their primary modality, or not at all. Ins...

Recent approaches to providing users with more natural methods of interacting with virtual environment applications have shown that more than one mode of input can be both beneficial and intuitive as a communication medium between humans and computer applications. Although there are many different modes that could be used in these applications, hand gestures and speech appear to be two of the m...