Gestures and multimodal input

Gestures and multimodal input

Joint interpretation of input speech and pen gestures for multimodal human-computer interaction

This paper describes out initial work in semantic interpretation of multimodal user input that consist of speech and pen gestures. We have designed and collected a multimodal corpus of over a thousand navigational inquiries around the Beijing area. We devised a processing sequence for extracting spoken references from the speech input (perfect transcripts) and interpreting each reference by gen...

A multimodal database of gestures and speech

This paper describes a multimodal database which consists of image data of human gestures and corresponding speech data for the research on multimodal interaction systems. The purpose of this database is to provide an underlying foundation for research and development of multimodal interactive systems. Our primary concern in selecting utterances and gestures for inclusion in the database was to...

A Multimodal User Authentication System Using Faces and Gestures

As a novel approach to perform user authentication, we propose a multimodal biometric system that uses faces and gestures obtained from a single vision sensor. Unlike typical multimodal biometric systems using physical information, the proposed system utilizes gesture video signals combined with facial images. Whereas physical information such as face, fingerprints, and iris is fixed and not ch...

Investigating the communication of emotions through multimodal technologies and gestures

In human–human interactions, touch is an important means of communicating emotional content. Touch-based technologies may have a role to play in augmenting computer-mediated communications, such as Chat and Instant Messaging. The study reported here investigated whether emotions could be effectively understood via haptic and gestural interfaces. Each interface was developed to communicate a spe...