Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
نویسندگان
چکیده
منابع مشابه
Multimodal sparse representation learning and applications
Unsupervised methods have proven effective for discriminative tasks in a singlemodality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared sparse representation. In particular, we propose the use of joint dictionary l...
متن کاملMultimodal Information Fusion
Humans interact with each other using different modalities of communication. These include speech, gestures, documents, etc. It is only natural that human computer interaction (HCI) should facilitate the same multimodal form of communication. In order to capture this information, one uses different types of sensors, i.e., microphones to capture the audio signal, cameras to capture life video im...
متن کاملHierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis
For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal inf...
متن کاملGated Multimodal Units for Information Fusion
This paper presents a novel model for multimodal learning based on gated neu-ral networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit u...
متن کاملAn Architecture for Multimodal Information Fusion
A. Shaikh, S. Juth, A. Medl, I. Marsic, C. Kulikowski, and J. L. Flanagan CAIP Center, Rutgers University 96 Frelinghuysen Road, Piscataway, NJ 08854-8088 [email protected] Abstract This paper presents a multimodal interface featuring fusion of multiple modalities for natural human-computer interaction. The architecture of the interface and the methods applied are described, and the results...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEEE Journal of Selected Topics in Signal Processing
سال: 2020
ISSN: 1932-4553,1941-0484
DOI: 10.1109/jstsp.2020.2987728