A technique for representing and learning smooth nonlinear manifolds is presented and applied to sev­ eral lip reading tasks. Given a set of points drawn from a smooth manifold in an abstract feature space, the technique is capable of determining the structure of the surface and of finding the closest manifold point to a given query point. We use this technique to learn the "space of lips" in a...

In this paper we propose a visual-based speech learning framework to assist deaf persons by comparing the lip movements between a student and an E-tutor in an intelligent tutoring system. The framework utilizes lip reading technologies to determine if a student learns the correct pronunciation. Different from conventional speech recognition systems, which usually recognize a speaker’s utterance...

Audio-visual speech recognition is used to make speech recognition more robust in case when the acoustic speech signal is affected by increased environmental noise. Since visual speech expression is not affected by the acoustic noise, it is used as supplemental information for the recognition. Usually, the shape of lips of a speaker is described by either pixel-based or shape-based features. Ma...

Recently there is an increased interest in using the visual features for improved speech processing. Lip reading plays a vital role in visual speech processing. In this paper, a new approach for lip reading is presented. Visual speech recognition is applied in mobile phone applications, human-computer interaction and also to recognize the spoken words of hearing impaired persons. The visual spe...

In this paper we present a new method to animate the face of a speaking avatar —i.e., a synthetic 3D human face— such that it realistically pronounces any given text, based on the audio only. Especially the lip movements must be rendered carefully, and perfectly synchronised with the audio, in order to have a realistic looking result, from which it should in principle be possible to understand ...

We have previously reported that a psychrotrophic bacterium, Pseudomonas sp. strain KB700A, which displays sigmoidal growth even at -5 degrees C, produced a lipase. A genomic DNA library of strain KB700A was introduced into Escherichia coli TG1, and screening on tributyrin-containing agar plates led to the isolation of the lipase gene. Sequence analysis revealed an open reading frame (KB-lip) c...

In this paper, we present a spatio-temporal feature representation and a probabilistic matching function to recognise lip movements from pronounced digits. Our model (1) automatically selects spatio-temporal features extracted from 10 digit model templates and (2) matches them with probe video sequences. Spatio-temporal features embed lip movements from pronouncing digits and contain more discr...

In this paper, we describe audiovisual automatic speech recognition experiments carried using visual parameters extracted from “natural” images. Unlike many other experiments in the AV ASR field, these visual parameters are obtained without any hand-labeling phase and are naturally noisy, due to the extraction process. We evaluate our models with different strategies among which : use of a shap...

Speech is the most common communication method between humans and involves the perception of both auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, but it has been demonstrated that video can provide information that is complementary to the audio. Thus, the study of automatic lip-reading is important and is still an open problem. One of the ke...

We present the development of a modular system for flexible human–computer interaction via speech. The speech recognition component integrates acoustic and visual information (automatic lip-reading) improving overall recognition, especially in noisy environments. The image of the lips, constituting the visual input, is automatically extracted from the camera picture of the speaker’s face by the...