In the past decade, accelerometer sensors became small enough to embed in mobile devices that people are using every day. Thus, arm movements became attractive to control mobile games and game consoles. However, there is no widely-used gesture-vocabulary for controlling devices and gesture recognition is not standardized. Furthermore, there is no widely-accepted benchmark for measuring performa...

We propose a framework for joint analysis of speech prosody and arm motion towards automatic synthesis and realistic animation of beat gestures from speech prosody and rhythm. In the analysis stage, we first segment motion capture data and speech audio into gesture phrases and prosodic units via temporal clustering, and assign a class label to each resulting gesture phrase and prosodic unit. We...

Reliable extraction of human pose features that are invariant to view angle and body shape changes is critical for advancing human movement analysis. In this dissertation, the multifactor analysis techniques, including the multilinear analysis and the multifactor Gaussian process methods, have been exploited to extract such invariant pose features from video data by decomposing various key cont...

Co-speech gesture facilitates learning to a greater degree in children than in adults, suggesting that the mechanisms underlying the processing of co-speech gesture differ as a function of development. We suggest that this may be partially due to children's lack of experience producing gesture, leading to differences in the recruitment of sensorimotor networks when comparing adults to children....

In face-to-face conversation, communicators orchestrate multimodal contributions that meaningfully combine the linguistic resources of spoken language and the visuo-spatial affordances of gesture. In this paper, we characterise this meaningful combination in terms of the COHERENCE of gesture and speech. Descriptive analyses illustrate the diverse ways gesture interpretation can supplement and e...

The movements we make with our hands both reflect our mental processes and help to shape them. Our actions and gestures can affect our mental representations of actions and objects. In this paper, we explore the relationship between action, gesture and thought in both humans and non-human primates and discuss its role in the evolution of language. Human gesture (specifically representational ge...

Gestural interfaces are now a familiar mode of user interaction and gestural input is an important part of the way that users can interact with such interfaces. However, entering gestures accurately and efficiently can be challenging. In this paper we present two styles of visual gesture autocompletion for 2D predictive gesture entry. Both styles enable users to abbreviate gestures. We experime...

Infants' understanding of a pointing gesture represents a major milestone in their communicative development. The current consensus is that infants are not capable of following a pointing gesture until 9-12 months of age. In this article, we present evidence from 4- and 6-month-old infants challenging this conclusion. Infants were tested with a spatial cueing paradigm in Experiment 1 (500-ms st...

There seem to be strong links between gestural imagery and musical imagery, and it is suggested that gestural imagery can be instrumental in triggering and sustaining mental images of musical sound. Gestural images are seen as integral to most experiences of music, and several practical and theoretical musical disciplines could profit from focusing on these gestural images. Research in support ...

In spite of many choices available for gesture recognition algorithms, the selection of a proper algorithm for a specific application remains a difficult task. The available algorithms have different strengths and weaknesses making the matching between algorithms and applications complex. Accurate evaluation of the performance of a gesture recognition algorithm is a cumbersome task. Performance...