Recent advances in neural variational inference have facilitated efficient training of powerful directed graphical models with continuous latent variables, such as variational autoencoders. However, these models usually assume simple, unimodal priors — such as the multivariate Gaussian distribution — yet many realworld data distributions are highly complex and multi-modal. Examples of complex a...

We investigate the effects of incorporating visual signal from images into unsupervised Semantic Textual Similarity (STS) measures. STS measures exploiting visual signal alone are shown to outperform, in some settings, linguistic-only measures by a wide margin, whereas multi-modal measures yield further performance gains. We also show that selective inclusion of visual information may further b...

We describe a new kind of image representation in terms of local multi–modal Primitives. These Primitives are motivated by processing of the human visual system as well as by functional considerations. We discuss analogies of our representation to human vision and concentrate specifically on the implications of the necessity of communication of information in a complex multi-modal system.

Abstract The pandemic has forced young people to stay away from school and friends, complete online learning at home live home. Therefore, various mental illnesses such as anxiety depression occur more frequently. Chatbot is a communication method that acceptable people. This paper proposes multi-modal chatbot seq2seq framework, which divides the state of into different types through informatio...

Modern autonomous vehicles rely heavily on mechanical LiDARs for perception. Current perception methods generally require $360^\circ$ point clouds, collected sequentially as the LiDAR scans azimuth and acquires consecutive wedge-shaped ...

Short Description: We have implemented a model which is ment for extracting coherent representations from multi-modal data. The model consists of a hierarchy of interconnected units. Initial experiments which verify that the model works in principle were reported in D5.3. This deliverable reports experiments with more realistic multi-modal data and discusses the background and motivation of the...

In this paper, a multi-modal data collection tool called MICoT is described. We highlight the various design and implementation aspects that we consider to be important for MICoT. An example is given to illustrate the application of the tool to collect data for our research in multi-modal dialog system.

This paper discusses the use of information gain (IG) to reduce and improve the textual information included in multi-modal information retrieval systems. Furthermore, a number of experiments are described that combine this reduction technique with a visualand textual-information merge. These show that the textual information manages to improve conventional multi-modal systems.