Perceptual image quality assessment using a normalized Laplacian pyramid
چکیده
We present an image quality metric based on the transformations associated with the early visual system: local luminance subtraction and local gain control. Images are decomposed using a Laplacian pyramid, which subtracts a local estimate of the mean luminance at multiple scales. Each pyramid coefficient is then divided by a local estimate of amplitude (weighted sum of absolute values of neighbors), where the weights are optimized for prediction of amplitude using (undistorted) images from a separate database. We define the quality of a distorted image, relative to its undistorted original, as the root mean squared error in this “normalized Laplacian” domain. We show that both luminance subtraction and amplitude division stages lead to significant reductions in redundancy relative to the original image pixels. We also show that the resulting quality metric provides a better account of human perceptual judgements than either MS-SSIM or a recently-published gain-control metric based on oriented filters. Introduction Many problems in image processing rely, at least implicitly, on a measure of image quality. Although mean squared error (MSE) is the standard choice, it is well known that it is not very well matched to the distortion perceived by human observers [1, 2, 3]. Objective measures of perceptual image quality attempt to correct this by incorporating known characteristics of human perception (see reviews [4, 5]). These measures typically operate by transforming the reference and distorted images and quantifying the error within that “perceptual” space. For instance, the seminal models described in [6, 7, 8] are based on psychophysical measurements of the dependence of contrast sensitivity on spatial frequency and contextual masking. Other models are designed to mimic physiological responses of neurons in the primary visual cortex. They typically include multi-scale oriented filtering followed by local gain control to normalize response amplitudes (e.g. [2, 9, 10]). Although the perceptual and physiological rationale for these models are compelling, they have complex parameterizations and are difficult to fit to data. Some models have been shown to be well-matched to the statistical properties of natural images, consistent with theories of biological coding efficiency and redundancy reduction [11, 12]. In particular, application of Independent Component Analysis (ICA) [13] (which seeks a linear transformation that best eliminates statistical dependencies in responses), or sparse coding [14] (which seeks to encode images with a small subset of basis elements), yields oriented filters resembling V1 receptive fields. Local gain control, in a form known as “divisive normalization” that has been widely used to describe sensory neurons [15], has been shown to decrease the dependencies between filter responses at adjacent spatial positions, orientations, and scales [16, 17, 18, 19, 20]. A widely-used measure of perceptual distortion is the structural similarity metric (SSIM) [21], which is designed to be invariant to “nuisance” variables such as the local mean or local standard deviation, while retaining sensitivity to the remaining “structure” of the image. It is generally used within a multi-scale representation (MS-SSIM), allowing it to handle features of all sizes [22]. While SSIM is informed by the invariances of human perception, the form of its computation (a product of the correlations between mean-subtracted, variance-normalized, and structure terms) has no obvious mapping onto physiological or perceptual representation. Nevertheless, the computations that underlie the embedding of those invariances – subtraction of the local mean, and division by the local standard deviation – are reminiscent of the response properties of neurons in the retina and thalamus. In particular, responses of these cells are often modeled as bandpass filters (“center-surround”) whose responses are rectified and subject to gain control according to local luminance and contrast (e.g., [23]). Here, we define a new quality metric, computed as the root mean squared error of an early visual representation based on center-surround filtering followed by local gain control. The filtering is performed at multiple scales, using the Laplacian pyramid [24]. While the model architecture and choice of operations are motivated by the physiology of the early visual system, we use a statistical criterion to select the local gain control parameters. Specifically, the weights used in computing the gain signal are chosen so as to minimize the conditional dependency of neighboring transformed coefficients. Despite the simplicity of this representation, we find that it provides an excellent account of human perceptual quality judgments, outperforming MS-SSIM, as well as V1-inspired models, in predicting the human data in the TID 2008 database [25]. Normalized Laplacian pyramid model Our model is comprised of two stages (figure 1): first, the local mean is removed by subtracting a blurred version of the image, and then these values are normalized by an estimate of the local amplitude. The perceptual metric is defined as the root mean squared error of a distorted image compared to the original, measured in this transformed domain. We view the local luminance subtraction and contrast normalization as a means of reducing redundancy in natural images. ©2016 Society for Imaging Science and Technology DOI: 10.2352/ISSN.2470-1173.2016.16HVEI-103 IS&T International Symposium on Electronic Imaging 2016 Human Vision and Electronic Imaging 2016 HVEI-103.1 Figure 1. Normalized Laplacian pyramid model diagram, shown for a single scale (k). The input image at scale k, x(k) (k = 1 corresponds to the original image), is modified by subtracting the local mean (eq. 2). This is accomplished using the standard Laplacian pyramid construction: convolve with lowpass filter L(ω), downsample by a factor of two in each dimension, upsample, convolve again with L(ω), and subtract from the input image x(k). This intermediate image z(k) is then normalized by an estimate of local amplitude, obtained by computing the absolute value, convolving with scale-specific filter P(k)(ω), and adding the scale-specific constant σ (k) (eq. 3)). As in the standard Laplacian Pyramid, the blurred and downsampled image x(k+1) is the input image for scale (k+1). Most of the redundant information in natural images is local, and can be captured with a Markov model. That is, the distribution of an image pixel (xi) conditioned on all others is well approximated by the conditional