We tackle a new challenge of modeling a perceptual experience in which a stimulus in one modality gives rise to an experience in a different sensory modality, termed synesthesia. To meet the challenge, we propose a probabilistic framework based on graphical models that enables to link visual modalities and auditory modalities via natural language text. An online prototype system is developed fo...

Abstract concepts are traditionally thought to differ from concrete concepts by their lack of perceptual information, which causes them to be processed more slowly and less accurately than perceptually-based concrete concepts. In two studies, we examined this assumption by comparing concreteness and imageability ratings to a set of perceptual strength norms in five separate modalities: sound, t...

Concepts are the basis of the human cognitive system, and the question of what constitutes the content of these mental representations has long occupied the cognitive sciences. Work in psychology, linguistics and cognitive neuroscience has converged on the idea that we develop our conceptual representations through our perception of and interaction with our environment. To date, such research h...

In vision, it is well established that the perceptual load of a relevant task determines the extent to which irrelevant distractors are processed. Much less research has addressed the effects of perceptual load within hearing. Here, we provide an extensive test using two different perceptual load manipulations, measuring distractor processing through response competition and awareness report. A...

This paper compares a neural network DSM relying on textual co-occurrences with a multi-modal model integrating visual information. We focus on nominal vs. verbal compounds, and zoom into lexical, empirical and perceptual target properties to explore the contribution of the visual modality. Our experiments show that (i) visual features contribute differently for verbs than for nouns, and (ii) i...

Two experiments were conducted that examined information integration and rule-based category learning, using stimuli that contained auditory and visual information. The results suggest that it is easier to perceptually integrate information within these sensory modalities than across modalities. Conversely, it is easier to perform a disjunctive rule-based task when information comes from differ...

This study examined the perceptual processing of time-gated auditory-visual (AV), auditory (A), and visual (V) spoken words. The primary goal was to assess the extent to which stimulus information versus perceptual processing limitations underlie modality-related perceptual encoding speed differences in AV, A, and V spoken word recognition. Another goal was to add to the scant literature on the...

Ambiguous stimuli can produce spontaneous perceptual alternations in the mind of the observer, even though the stimulus itself remains the same. Common features in the temporal dynamics of bistability have been observed for various types of stimuli, both visual and auditory. This raises the question of whether bistable perception results from stereotyped, local competition between stimulus-spec...

In this review, we distinguish strong and weak forms of synesthesia. Strong synesthesia is characterized by a vivid image in one sensory modality in response to stimulation in another one. Weak synesthesia is characterized by cross-sensory correspondences expressed through language, perceptual similarity, and perceptual interactions during information processing. Despite important phenomenologi...