The human brain's representation of objects has been proposed to exist as a network of coactivated neural regions present in multiple cognitive systems. However, it is not known if there is a region specific to the process of activating an integrated object representation in semantic memory from multimodal feature stimuli (e.g., picture-word). A previous study using word-word feature pairs as s...

In myoelectric prostheses, movement artifacts are known to impair control performance. This study relates to a novel sensor which measures surface electromyograms (SEMG) as well as contact force at the electrode-skin interface. Its purpose is to explore the in-socket mechanical realities of movement artifacts in order to produce control algorithms that are more robust to said artifacts. The new...

As robots enter the human environment and come into contact with inexperienced users, they need to be able to interact with users in a multimodal fashion—keyboard and mouse are no longer acceptable as the only input modalities. In this paper we introduce a novel approach for programming robots interactively through a multimodal interface. The key characteristic of this approach is that the user...

This thesis focuses on unifying multiple modal data flows that may be provided by sensors of anagent. This unification, inspired by psychological experiments like the ventriloquist effect, is based ondetecting correlations which are defined as temporally recurrent spatial patterns that appear in the inputflows. Learning of the input flow correlations space consists on sampling this ...

A basic capability driving the development of systems is the ability to execute multiple versions of a system against the same set of data. That is essential for instance to verify that coding problems were corrected, to provide guarantees that changes to the code did not introduce errors (regression testing) and for comparison of performance among different algorithmic solutions. Multimodal sy...

We introduce the notion of symmetric multimodality for dialogue systems in which all input modes (eg. speech, gesture, facial expression) are also available for output, and vice versa. A dialogue system with symmetric multimodality must not only understand and represent the user's multimodal input, but also its own multimodal output. We present the SmartKom system, that provides full symmetric ...

How do infants segment continuous streams of speech to discover words of their language? Current theories emphasize the role of acoustic evidence in discovering word boundaries (Cutler 1991; Brent 1999; de Marcken 1996; Friederici & Wessels 1993; see also Bolinger & Gertsman 1957). To test an alternate hypothesis, we recorded natural infant-directed speech from caregivers engaged in play with t...

In this paper we described our initial work on the development of an embodied conversational agent platform. In the present stage our main focus it is on the development of a multimodal input interface to the system. In this paper we will present an Input and Output Manager block that combines speech, synthetic talking face, text and graphical interfaces. The system support speech input through...

In this paper, a multi-modal data collection tool called MICoT is described. We highlight the various design and implementation aspects that we consider to be important for MICoT. An example is given to illustrate the application of the tool to collect data for our research in multi-modal dialog system.

Object manipulation constitutes a large part of our daily hand movements. Recognition of such movements by a robot in an interactive scenario is an issue that is rapidly gaining attention. In this paper we present an approach to identification of a class of high-level manual object manipulations. Experiments have shown that the naive approach based on classification of low-level sensor data yie...