We present M3I, an extensive multimodal interaction framework for mobile devices, which simplifies and accelerates the creation of multimodal applications for prototyping and research. It provides an abstraction of information representations in different communication channels, unifies access to implicit and explicit information, and wires together the logic behind context-sensitive modality s...

Multimodal nanoprobes have attracted intensive attentions since they can integrate various imaging modalities to obtain complementary merits of single modality. Meanwhile, recent interest in laser-induced photoacoustic imaging is rapidly growing due to its unique advantages in visualizing tissue structure and function with high spatial resolution and satisfactory imaging depth. In this review, ...

This paper describes ongoing work on a tool developed for annotating document images for their multimodal features and compiling this information into a corpus. The tool leverages open source computer vision and natural language processing libraries to describe the content and structure of multimodal documents and to generate multiple layers of XML annotation. The paper introduces the annotatio...

In this text, we use a Bayesian framework for active multimodal perception of 3D structure and motion — which, while not strictly neuromimetic, finds its roots in the role of the dorsal perceptual pathway of the human brain — to implement a strategy of active exploration based on entropy. The computational models described in this text support a robotic implementation of multimodal active perce...

The use of speech-accompanying hand gestures to depict objects, to structure the discourse or to give directions is ubiquitous in face-to-face interaction. In this paper, we analyse multimodal signals consisting of speech and gesture from the perspective of semantic composition in constraint-based grammars: we elevate standard methods from linguistics to description of multimodal input so as to...

We describe the reasoning mechanisms used in a fully-implemented dialogue system. This dialogue system, based on a speech acts formalism, supports a multimodal conversational character for Interactive Television. The system maintains an explicit representation of programme descriptions , which also constitutes an attentional structure. From the contents of this representation , it is possible t...

  Background: Our institution’s traditional pain management strategy after shoulder arthroplasty has involved the utilization of postoperative patient-controlled narcotic analgesia. More recently, we have implemented a protocol (TLC) that utilizes a multimodal approach. The purpose of this study was to determine whether this change has improved pain control and decreased narcotic utilization. ...

conclusions the combination of intravenous paracetamol 1 gr and ketamine 0.5 mg/kg resulted in an overall reduction in pain scores, decreased postoperative analgesic requirements, and lower agitation score compared with intravenous paracetamol 1 gr and tramadol 0.7 mg/kg for patients undergoing renal surgery. results postoperative pain scores were significantly lower in the pk group than in the...

for demonstration Abstract—In this demo, we present a multimodal conversation system, implemented using a Nao robot and Wikipedia. The system was developed at the 8th International Workshop on Multimodal Interfaces in Metz, France, 2012. The system is based on an interactive, open-domain spoken dialogue system called WikiTalk, which guides the user through conversations based on the link struct...

Multimodal interaction is employed in a variety of contemporary information systems in order to enhance the flexibility and naturalness of the user interface. In this paper, we propose a comprehensive framework which allows the modeling and specification of multimodal interactions. To this end, we employ an extended notion of ‘dialogue acts’ which can be performed by linguistic or non-linguisti...