exploiting multimodal information like acceleration and heart rate is a promising method to achieve human action recognition. a semi-supervised action recognition approach aucc (action understanding with combinational classifier) using the diversity of base classifiers to create a high-quality ensemble for multimodal human action recognition is proposed in this paper. furthermore, both labeled ...

We address the problem of segmentation and recognition of sequences of multimodal human interactions in meetings. These interactions can be seen as a rough structure of a meeting, and can be used either as input for a meeting browser or as a first step towards a higher semantic analysis of the meeting. A common lexicon of multimodal group meeting actions, a shared meeting data set, and a common...

We developed a label-free nonlinear optical (NLO) microscope integrating the stimulated Raman scattering, multi-color two-photon excited fluorescence and second harmonic generation. The system produces multimodal images of protein content, mitochondria distribution and sarcomere structure of fresh muscle samples. With the advanced imaging technique, we studied the mal-development of skeletal mu...

In this paper, we present a multimodal discourse ontology that serves as a knowledge representation and annotation framework for the discourse understanding component of an artificial personal office assistant. The ontology models components of natural language, multimodal communication, multi-party dialogue structure, meeting structure, and the physical and temporal aspects of human communicat...

We introduce a novel system of interconnected SelfOrganizing Maps that can be used to build feedforward and recurrent networks of maps. Prime application of interconnected maps is in modelling systems that operate with multimodal data as for example in visual and auditory cortices and multimodal association areas in cortex. A detailed example of animal categorization in which the feedworward ne...

Modern user interfaces can take advantage of multiple input modalities such as speech, gestures, handwriting... to increase robustness and flexibility. The construction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammar structu...

Nowadays, multimodal data available to us through the Internet and other electronic media are explosively increasing both in number and in variety. Efficiently analyzing these data will save people a large amout of time. However, due to the property of multimodal data, methods used to uni-modal data usually perform ineffectively here. To handle such massive multimodal data for various purposes,...

Multimodal communication involves multiple communicative channels including speech, facial movement, and gesture. Relatively few studies on how various communicative modalities are aligned in natural, face-to-face communication exist. As part of a larger project, the current study investigates how discourse structure, speech features, eye gaze, and facial movements interrelate during a map coor...