Recent research has revealed that the presence of irrelevant visual information during retrieval of long-term memories diminishes recollection of task-relevant visual details. Here, we explored the impact of irrelevant auditory information on remembering task-relevant visual details by probing recall of the same previously viewed images while participants were in complete silence, exposed to wh...

We investigate the use of visual, mouth-region information in improving automatic speech recognition (ASR) of the speech impaired. Given the video of an utterance by such a subject, we first extract appearance-based visual features from the mouth region-of-interest, and we use a feature fusion method to combine them with the subject’s audio features into bimodal observations. Subsequently, we a...

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial animation parameters (FAPs) supported by the MPEG-4 standard for the visual representation of speech. We also describe a robust and automatic algorit...

BACKGROUND Schizophrenia (SCZ) results in abnormalities affecting both working memory (WM) and emotional processing. Prior work suggests that individuals with SCZ exhibit increased effects of distraction on WM-a deficit that might be exacerbated via emotional interference. However, no study characterized effects of negative and nonemotional interference on visual WM in SCZ with functional magne...

Speech is a means of communication that is intrinsically bimodal: the audio signal originates from the dynamics of the articulators. This paper reviews recent works in the field of audiovisual speech and more specifically on techniques developed to measure the level of correspondence between audio and visual speech. It overviews the most common audio and visual speech front-end processing, tran...