Research on the influence of multimodal information on infants' learning is inconclusive. While one line of research finds that multimodal input has a negative effect on learning, another finds positive effects. The present study aims to shed some new light on this discussion by studying the influence of multimodal information and accompanying stimulus complexity on the learning process. We ass...

A framework of complementary behavior has been proposed which maintains that direct manipulation and speech interfaces have reciprocal strengths and weaknesses. This suggests that user interface performance and acceptance may increase by adopting a multimodal approach that combines speech and direct manipulation. This effort examined the hypothesis that the speed, accuracy, and acceptance of mu...

This paper describes out initial work in semantic interpretation of multimodal user input that consist of speech and pen gestures. We have designed and collected a multimodal corpus of over a thousand navigational inquiries around the Beijing area. We devised a processing sequence for extracting spoken references from the speech input (perfect transcripts) and interpreting each reference by gen...

Two important themes in current work on interfaces are multimodal interaction and the use of dialogue. Human multimodal dialogues are symmetric, i.e., both participants communicate multimodally. We describe a proof of concept system that supports symmetric multimodal communication for speech and sketching in the domain of simple mechanical device design. We discuss three major aspects of the co...

This usability study aims at evaluating in which way different groups of users deal with a multimodal interface for navigating in arbitrary virtual worlds. Besides classical input devices like keyboard, mouse and touchscreen the test subjects can control the system by natural speech utterances as well as dynamic hand and head gestures. Experts tend to use haptic devices, whereas normal computer...

In this paper we describe a new automated suite of tools for capturing and analyzing data on multimodal systems called STAMP. STAMP is designed to support research and development efforts for advancing next-generation multimodal systems. STAMP permits researchers to analyze multimodal system performance by: (1) recording data on users’ multimodal input and the system’s responding, (2) supportin...

Multimodal systems process combined natural input modes—such as speech, pen, touch, hand gestures, eye gaze, and head and body movements—in a coordinated manner with multimedia system output. These systems represent a new direction for computing that draws from novel input and output technologies currently becoming available. Since the appearance of Bolt's [1] " Put That There " demonstration s...

Integrating new input-output modalities, such as speech, gaze, gestures, haptics, etc., in user interfaces is currently considered as a significant potential contribution to implementing the concept of Universal Access (UA) in the Information Society; see (Oviatt, 2003), for instance. UA in this context means providing everybody, including handicapped users, with easy human-computer interaction...

Multimodal systems process combined natural input modes—such as speech, pen, touch, hand gestures, eye gaze, and head and body movements—in a coordinated manner with multimedia system output. These systems represent a new direction for computing that draws from novel input and output technologies currently becoming available. Since the appearance of Bolt's [1] " Put That There " demonstration s...

Advances in information technologies are producing a very large number of possible interface modality combinations which are potentially useful for the expression and exchange of information in humancomputer interaction. However, a principled basis for analysing arbitrary input/output modality types and combinations as to their capabilities of information representation and exchange is still la...