Multimodal systems process two or more combined user input modes— such as speech, pen, touch, manual gestures, gaze, and head and body movements— in a coordinated manner with multimedia system output. This class of systems represents a new direction for computing, and a paradigm shift away from conventional WIMP interfaces. Since the appearance of Bolt’s (1980) “Put That There” demonstration sy...

Multimodal systems process two or more combined user input modes— such as speech, pen, touch, manual gestures, gaze, and head and body movements— in a coordinated manner with multimedia system output. This class of systems represents a new direction for computing, and a paradigm shift away from conventional WIMP interfaces. Since the appearance of Bolt’s (1980) “Put That There” demonstration sy...

This article presents the development of an adaptive multimodal Digital Talking Book player using FAME. FAME is a conceptual framework for the development of adaptive multimodal applications. FAME's adaptation is based on context changes. The context is defined by three entities: User, Platform and Environment. The adaptation can impact multimodal operations, as well as interaction and presenta...

Mobile technologies have yet to be widely adopted by the Architectural, Engineering, and Construction (AEC) industry despite being one of the major growth areas in computing in recent years. This lack of uptake in the AEC industry is likely due, in large part, to the combination of small screen size and inappropriate interaction demands of current mobile technologies. This paper discusses the s...

This paper describes how the thermal user interface system can be integrated into other input devices to develop multimodal user interfaces that present multisensory feedback. Multimodal virtual environments supporting both haptic force and haptic thermal feedback will allow researchers to investigate a wide range of research issues involved in presence study. Among presence-related research qu...

In order to realize their full potential, multimodal systems need to support not just input from multiple modes, but also synchronized integration of modes. Johnston et al (1997) model this integration using a unification operation over typed feature structures. This is an effective solution for a broad class of systems, but limits multimodal utterances to combinations of a single spoken phrase...

This paper presents the results of an elaborate study on pen and speech-based multimodal interaction systems. The performance o f the “COM IC” system is assessed through human factors analyses and evaluation o f the acquired multimodal data. The latter requires tools that are able to m onitor user input, system feedback, and performance of the multimodal system components. Such tools can bridge...

In this paper, we describe our e orts in designing and building a prototype multimodal system for children users. Data collection e orts and user experience results from a WoZ study using a popular computer game are reviewed rst. Automatic speech recognition and spoken language understanding technology for children speakers are discussed next. A multimodal prototype is designed for a personal a...

Previous studies on usability of graphical design-widgets, like menus and buttons, proposed the use of speech and non-speech (earcons and auditory icons) for solving their usability problems. In this paper we investigate speech as an input metaphor to enhance learnability, or the ability to use a system with no prior knowledge, in order to design interfaces using a multimodal interface design t...