Pattern inference theory: A probabilistic approach to vision∗

The function of vision is to get correct and useful answers about the state of the world. However, given that the state of the world is not uniquely specified by the visual input, the visual system must make good guesses or inferences. Thus, theories of visual system functions will be theories of inference, and we need a language in which theories of inference can be described. Analogous to calculus having a minimum expressiveness required to formulate theories in physics, we argue that the language of Bayesian inference is fundamental to quantitatively describe how reliable answers about the world can be obtained from image patterns. Bayes provides a minimal formalism that can deal with the sophistication and versatility of perception missing from some other approaches. Key missing components include the ability to model uncertainty, probabilistic modeling of pattern synthesis as a necessary prerequisite to understanding pattern inference, the means to handle the complexity of natural images, and the diversity of visual tasks. Most of the formal elements that we describe are not new and have their roots in signal detection theory and ideal observer analysis. We start from there to review and codify principles drawn from recent applications of Bayesian decision theory, Bayes nets and pattern theory to vision. To emphasize the importance of dealing with the complexity of natural image and scene patterns, we call the conjunction of principles drawn from these contributions pattern inference theory. Because of its generality, we do not see pattern inference theory as an experimentally testable theory of vision; however, it does provide a set of concepts and principles to formulate testable models. The test for a good theoretical framework is utility and completeness for deriving predictive theories. To illustrate the utility of the approach, we propose Bayesian principles of least commitment and modularity, each of which leads to testable hypotheses. Several recent examples of pattern inference theories are reviewed. 1 Perception is pattern decoding Few would dispute the view that visual perception is the brain’s process for arriving at useful information about the world from images. Divergent opinions, however, have been expressed over how to describe the computations (or lack thereof) underlying visual behavior. Visual perception has been described as unconscious inference (Helmholtz and Southall, 1924; Gregory, 1980), reconstruction (Craik, 1943), resonance (Gibson, 1966), problem solving (Rock, 1983), computation (Marr, 1982), and more recently as Bayesian inference (Knill and Richards, 1996). In part, the debate gets muddled due to lack of a well-specified explanatory goal and level of abstraction. To clarify, we see the grand challenge to be the development of testable, quantitative theories of visual performance that take into account the complexities of natural images and the richness of visual behavior. But here the level of explanation is crucial: if our theories are too abstract, we lose the specificity of quantitative predictions; if the theories are too fine-grained, the model mechanisms for natural pattern processing will be too complex to test. Our proposed strategy follows that of statistical mechanics. Few physicists doubt that the large-scale properties of physical systems rest on the lawful function of individual molecules, just as few brain scientists doubt that an organism’s behavior depends on the lawful function of neurons. Physicists would agree that the modeling level has to be appropriate to the measurements and phenomena of large-scale systems; thus statistical mechanics links molecular kinetics to thermodynamics. Although the bridge between neurons and system behavior has yet to be built, the language of Bayesian statistics