This paper describes a new flexible representation for the annotation of complex structures of metadata over heterogeneous data collections containing text and other types of media such as images or audio files. We argue that existing frameworks are not suitable for this purpose, most importantly because they do not easily generalize to multi-document and multimodal corpora, and because they of...

This article provides an overview of research in multimodal language processing and associated resources. It defines multimodal processing, describes key challenges, identifies potential benefits, and outlines the major tasks, including multimodal input interpretation, multimodal output generation, and multimodal information access. The article exemplifies the state of the art in multimedia and...

In recent years because of the advances in computer vision research, free hand gestures have been explored as means of human-computer interaction (HCI). Together with improved speech processing technology it is an important step toward natural multimodal HCI. However, inclusion of non-predefined continuous gestures into a multimodal framework is a challenging problem. In this paper, we propose ...

We propose algorithm for local image difference measurement for multimodal image data based on value of mutual information of both images. Algorithm works with registered gray-level images. Similarity measure takes into account entropy of compared images. Results of proposed algorithm can be used for image comparison and better difference localization. Introduction Multimodal data sets are nowa...

In this talk, we will, show how techniques for planning text and discourse can be generalized to plan the structure and content of multimodal communications , that integrate natural language, pointing, graphics, and animations. The central claim of this talk is that the generation of multimodal discourse can be considered as an incremental planning process that aims to achieve a given communica...

The SyncPlayer is a prototypical software framework that integrates various Music Information Retrieval (MIR) techniques such as music synchronization, audio structure analysis and content-based retrieval into a powerful, multimodal system [1, 2]. The SyncPlayer system basically consists of three software components: a server component, a client component, and some tools for data administration...

Although vector strength (VS) and the Rayleigh tests are widely used to quantify neuronal firing synchrony to cyclic events, their use is valid only for singly peaked, unimodal distributions. In this report, we propose a new method to quantify synchrony, applicable to both unimodal and multimodal distributions. We also propose a statistical test to examine temporal structure under a null hypoth...

A modification of the standard Simulated Annealing (SA) algorithm is presented for finding the global minimum of a continuous multidimensional, multimodal function. We report results of computational experiments with a set of test functions and we compare to methods of similar structure. The accompanying software accepts objective functions coded both in Fortran 77 and C++.

In this article it will be shown that morpheme length in Lakota obeys a special multimodal distribution resulting from a difference equation of second order. This is due to the particular structure of Lakota syllables, which provide the building blocks for morphemes.

A dissipative particle swarm optimization is developed according to the self-organization of dissipative structure. The negative entropy is introduced to construct an opening dissipative system that is far-from-equilibrium so as to driving the irreversible evolution process with better fitness. The testing of two multimodal functions indicates it improves the performance effectively.