Gradient algorithms for designing predictive vector quantizers
چکیده
A predictive vector quantizer (PVQ) is a vector extension of a predictive quantizer. It consists of two parts: a conventional memoryless vector quantizer (VQ) and a vector predictor. Two gradient algorithms for designing a PVQ are developed in this paper: the steepest descent (SD) algorithm and the stochastic gradient (SG) algorithm. Both have the property of improving the quantizer and the predictor in the sense of minimizing the distortion as measured by the average mean-squared error. The differences between the two design approaches are the period and the step size used in each iteration to update the codebook and predictor. The SG algorithm updates once for each input training vector and uses a small step size, while the SD updates only once for a long period, possibly one pass over the entire training sequence, and uses a relatively large step size. Code designs and tests are simulated for b th Gauss-Markov sources and for sampled speech waveforms, and the results are compared to codes designed using techniques that attempt to optimize only the quantizer for the predictor and not vice versa. I . INTRODUCTION A VECTOR quantizer is a system for mapping a sequence of continuous or high rate discrete k-dimensional vectors into a digital sequence suitable for communication over or storage in a digital channel. While Shannon theory states that memoryless vector quantization is sufficient to achieve nearly optimal performance, such performance is guaranteed only for large vector dimension. Unfortunately, however, for a fixed rate in bits per sample, the codebook size grows exponentially with vector dimension and, hence, the complexity of minimum distortion searches required by the encoder also grows exponentially. For this reason, recent research has focused on design techniques for vector quantizers that have structures which yield a slower growth of encoder complexity with rate or dimension. Two vector quantizer structures that have proved promising are feedback vector quantizers with full searches of small codebooks and memoryless vector quantizers with large codebooks and suboptimal, but efficient, search algorithms. A general survey of vector quantization, including many examples of both structures, may be found in [ 11. We here develop new design algorithms for predictive vector quantizers, a special case of the class of feedback quantizers. Manuscript received December 29, 1984; revised August 12, 1985. This work was supported in part by the Joint Services Electronics Program at Stanford University and by the National Science Foundation under Grant The authors are with the Information Systems Laboratory, Department IEEE Log Number 8608122. ECS83-17981. of Electrical Engineering, Stanford University, Stanford, CA 44305. A predictive vector quantizer (PVQ) or vector predictive quantizer is a vector extension of a predictive quantizer or DPCM system. In the encoding process, an error vector formed as the difference between the input vector and the prediction of this vector is coded by a memoryless vector quantizer. The vector quantizer chooses the minimum distortion codeword from a stored codebook, and transmits the index of this codeword to the receiver. A PVQ is a feedback VQ because the encoder output is fed back to the predictor for use in approximating the new input vector. The general structure of PVQ was introduced by Cuperman and Gersho [2], [3], who developed a PVQ design algorithm for waveform coding with two main steps. First, a set of linear vector predictive coefficients is computed from the input training sequence by generalized.LPC techniques. Second, a vector quantizer codebook is designed for the innovation sequence formed by subtracting the input vector from a linear predicted value based on the actual past inputs (“open-loop’’ design) or for the actual prediction error formed as the difference between the input vector and the linear predicted value based on the past quantized outputs (“closed-loop’’ design). The generalized Lloyd algorithm was used for the vector codebook design (see, e.g., [4]). As with traditional design techniques for scalar predictive quantization or DPCM, the predictor is designed under the assumption that the prediction is based on past input vectors rather than on their quantized values; that is, it is effectively assumed that the quantized reproduction is nearly perfect. This approximation may be quite poor if the quantizer has a low rate. This causes a potential problem in the system design: the predictor ’which is optimum given past true values will not be so for past quantized values. A second potential problem arises when the open-loop design technique is used-the vector quantizer designed to be good for the ideal innovations sequence may not be as good when applied to the actual prediction error sequence. The closed-loop design resolves this problem and was found to provide 1-2 dB improvement over the open-loop design for sampled speech at rates ranging from 1 to 2 bits/sample with vector dimensions ranging from 1 to 5 . This paper presents an approach to design predictive vector quantizers by applying standard techniques of adaptive filtering to design both vector quantizers and vector predictors in PVQ. The quantizers and predictors are iteratively optimized for each other using gradient 0096-3518/86/0800-0679$01 .OO O 1986 IEEE 680 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-34, NO. 4, AUGUST 1986 search techniques. This is accomplished by simultaneously adjusting both the linear predictive coefficients and the vector quantizer codebook with each fixed number of training vectors, where the number of training vectors used for each update can range from one vector to the entire training sequence, The adjustments attempt to minimize the cumulative sample squared error between the input vectors and the reconstructed signals. No assumption of perfect reproduction is needed in this algorithm and, hence, it may yield better codes. The algorithm is used only in the design of the system, not as an on-line adaptation mechanism as in the adaptive gradient algorithms of, e.g., Gibson et al. [5] and Dunn [ 6 ] . Some preliminary work on a scalar version of this algorithm was developed in unpublished work of Y. Linde. Preliminary results of the research described here were reported in [ 11. On the positive side, simulations on Gauss-Markov and sampled speech indicate that the SD and SG design techniques yield good codes when the parameters are chosen intelligently. On the negative side, the resulting codes yield overall performance quite close to those designed using the Cuperman-Gersho technique. These results are of interest not only because they show that popular adaptive filtering algorithms can be modified to design good predictive vector quantizers, but because they show that optimizing the quantizer for the vector predictor yields good overall performance even when the predictor is not optimized for the quantizer. In other words, predictive vector quantizers are robust against inaccuracies in the predictor provided the quantizer is matched to the predictor. As a final observation, the algorithms developed here have also been extended to the design of good predictive trellis encoding systems [7] and joint source and channel trellis encoding systems [ 81. The basic structure of a PVQ system is presented in the second section. The principles of the gradient design algorithm are discussed in the third section. Simulation results for two different sources follow. Finally, comments and suggestions for future research are mentioned. 11. PREDICTIVE VECTOR QUANTIZER A PVQ system is sketched in Fig. 1. Let be a vector-valued random process or source with alphabet B, e.g., k-dimensional Euclidean space R k . A PVQ consists of three functions. An encoder y which assigns to each error vector e, = x, x", a channel symbol y(e,) in some channel symbol set M , a decoder fl assigning to each channel symbol u, in M a value in a reproduction alphabet 8, and a prediction function or next state functionfwhich approximates the next input vector x, + as x", + = f ( i n , in ). Given a sequence of input vectors and an initial prediction fo, the channel symbol sequence u,, reproduction sequence i,, and prediction sequence x", are defined recursively for n = 0, 1 , 2, , as u, = r(e,> = r(xn x",),