This paper discusses an evaluation of an augmented reality (AR) multimodal interface that uses combined speech and paddle gestures for interaction with virtual objects in the real world. We briefly describe our AR multimodal interface architecture and multimodal fusion strategies that are based on the combination of time-based and domain semantics. Then, we present the results from a user study...

Multimodal interface aims at overcoming the shortcomings of conventional human-computer communication -serialized input, excessive accuracy, and low input efficiency -by exploiting the complementary nature of multiple modalities in interpreting user intent, thus improving the efficiency and naturalness of humancomputer interaction. In this paper, we present motivations driving multimodal interf...

The area of multimodal interaction has expanded rapidly and since the seminal “Put that there” demonstrator of R. Bolt that combines speech, gesture and eye tracking, significant achievements have been made in terms of both modalities and real multimodal systems. Nevertheless the design and development of multimodal systems still remains a difficult task. In this article, we present a graphical...

In this work we show that there is a need of using multimodal resources during human-computer interaction (HCI) in intelligent systems. We propose that not only creating multimodal output for the user is important, but to take multimodal input resources into account for the decision when and how to interact. Especially the use of multimodal input resources for the decision when and how to provi...

While current e-learning activities mainly focus on static mode, this paper proposed a simple Active Multimodal Presentation (AMP) application, and evaluated the student achievement in static and active modes. Of the various AMP acquisition and presentation tools, this paper addresses the use of active text as input and output interaction and animation with video for presenting some learning co...

This paper describes a prototype of a multimodal railway information system that was built by extending an existing speech-only system. The purpose of the extensions is to alleviate a number of shortcomings of speech-only interfaces.

— Human-to-human conversation remains such a significant part of our working activities because of its naturalness. Multimodal interaction systems combine visual information with voice, gestures and other modalities to provide flexible and powerful dialogue approaches. The use of integrated multiple input modes enables users to benefit from the natural approach used in human communication. Howe...

Mobile multimodal dialogue systems allow the user and the system to adapt their choice of input and output modality according to various technical and cognitive resource limitations and the task at hand. We present the multimodal dialogue system SmartKom, that can be used as mobile travel companion for car drivers and pedestrians. SmartKom combines speech, gestures, and facial expressions for i...

This article provides an overview of research in multimodal language processing and associated resources. It defines multimodal processing, describes key challenges, identifies potential benefits, and outlines the major tasks, including multimodal input interpretation, multimodal output generation, and multimodal information access. The article exemplifies the state of the art in multimedia and...