Multimodal interaction is employed in a variety of contemporary information systems in order to enhance the flexibility and naturalness of the user interface. In this paper, we propose a comprehensive framework which allows the modeling and specification of multimodal interactions. To this end, we employ an extended notion of ‘dialogue acts’ which can be performed by linguistic or non-linguisti...

Despite significant advances in multimodal imaging techniques and analysis approaches, unimodal studies are still the predominant way to investigate brain changes or group differences, including structural magnetic resonance imaging (sMRI), functional MRI (fMRI), diffusion tensor imaging (DTI) and electroencephalography (EEG). Multimodal brain studies can be used to understand the complex inter...

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduc...

BACKGROUND Multimodal optical microscopy, a set of imaging techniques based on unique, yet complementary contrast mechanisms and spatially and temporally co-registered data acquisition, has emerged as a powerful biomedical tool. However, the analysis of the dense, high-dimensional datasets acquired by these instruments remains mostly qualitative and restricted to analysis of each modality indiv...

ZHANG, WEI. Multimodal Pedagogical Planning for 3D Learning Environments A Unified Framework. (Under the Direction of Dr. James C. Lester and Dr. R. Michael Young.) Pedagogical planning lies at the heart of knowledge-based learning environments. In recent years, multimodality and authoring have become key issues in the creating of learning environments. The purpose of this research has been to ...

Multimodal learning with deep Boltzmann machines (DBMs) is an generative approach to fuse multimodal inputs, and can learn the shared representation via Contrastive Divergence (CD) for classification and information retrieval tasks. However, it is a 2-fan DBM model, and cannot effectively handle multiple prediction tasks. Moreover, this model cannot recover the hidden representations well by sa...