Cross-Modal Clustering

This paper presents a self-supervised algorithm for learning perceptual structures based upon correlations in different sensory modalities. The brain and cognitive sciences have gathered an enormous body of neurological and phenomenological evidence in the past half century that demonstrates the extraordinary degree of interaction between sensory modalities during the course of ordinary perception. This paper presents a new framework for creating artificial perceptual systems inspired by these findings, where the primary architectural motif is the cross-modal transmission of perceptual information to enhance each sensory channel individually. The basic hypothesis underlying this approach is that the world has regularities – natural laws tend to correlate physical properties – and biological perceptory systems have evolved to take advantage of this. They share information continually and opportunistically across seemingly disparate perceptual channels, not epiphenomenologically, but rather as a fundamental component of normal perception. It is therefore essential that their artificial counterparts be able to share information synergistically within their perceptual channels, if they are to approach degrees of biological sophistication. This paper is a preliminary step in that direction.