Deformable registration of images obtained from different modalities remains a challenging task in medical image analysis. This paper addresses this problem and proposes a new similarity metric for multi-modal registration. The metric is based on the orientation of image gradients. In contrast to earlier work, our metric is based on the structure tensor of the spatial image gradients and can ro...

This paper sketches the concept of haptic-multimodal coupling between operator, coautomation, base system and environment. Haptic-multimodal couplings use mainly the haptic interaction resource, e.g. the combination of hands and feet with active inceptors like active sidesticks or steering wheels, and complement this with e.g. visual and acoustic feedback. Haptic-multimodal couplings can serve ...

The aim of this paper is to present the design of a multimodal database suitable for research on new possibilities for automatic diagnosis of patients with severe obstructive sleep apnoea (OSA). Early detection of severe apnoea cases can be very useful to give priority to their early treatment optimizing the expensive and time-consuming tests of current diagnosis methods based on full overnight...

Deep learning has been successfully applied to multimodal representation learning problems, with a common strategy to learning joint representations that are shared across multiple modalities on top of layers of modality-specific networks. Nonetheless, there still remains a question how to learn a good association between data modalities; in particular, a good generative model of multimodal dat...

Much research has been directed towards developing mul-timodal interfaces in the past twenty years. Many current multimodal systems, however, can only handle multimodal inputs at the sentence level. The move towards multimodal dialogue signiicantly increases the complexity of the system as the representations of the input now range over time and input modality. We developed a framework to addre...

We propose an approach to analyze and synthesize a set of human facial and vocal expressions, and then use the classified expressions to decide the robot’s response in a human-robot-interaction. During a human-tohuman conversation, a person senses the interlocutor’s face and voice, perceives her/his emotional expressions, and processes this information in order to decide which response to give....

Large annotation projects, typically those addressing the question of multimodal annotation in which many different kinds of informationhave to be encoded, have to elaborate precise and high level annotation schemes. Doing this requires first to define the structure of theinformation: the different objects and their organization. This stage has to be as much independent as possible ...

The report discusses a cognitive model in which a key concept is Multimodal Record (MMR)--an organized aggregate of transcripts of signals representing all information an Agent continuously acquired for a certain period of time. The MMR consists of a video track, sound track, and a number of tracks containing transcripts of the values of temperature, pressure, etc., as well as transcripts of st...

The size of an organism reflects its metabolic rate, growth rate, mortality, and other important characteristics; therefore, the distribution of body size is a major determinant of ecosystem structure and function. Body-size distributions often are multimodal, with several peaks of abundant sizes, and previous studies suggest that this is the outcome of niche separation: species from distinct p...