We emphasize the importance of variance reduction in stochastic methods and propose a probabilistic interpretation as a way to store information about past gradients. The resulting algorithm is very similar to the momentum method, with the difference that the weight over past gradients depends on the distance moved in parameter space rather than the number of steps.

We introduce covariance reducing models for studying the sample covariance matrices of a random vector observed in different populations. The models are based on reducing the sample covariance matrices to an informational core that is sufficient to characterize the variance heterogeneity among the populations. They possess useful equivariance properties and provide a clear alternative to spectr...

We develop a variance reduction method for smoothing splines. We do this by showing that the quadratic interpolation method introduced in Cheng et al. (2006), for local linear estimators, also works for smoothing splines. For a given point of estimation, Cheng et al. (2006) define a variance-reduced local linear estimate as a linear combination of classical estimates at three nearby points. We ...

Many Monte Carlo simulation problems lend themselves readily to the application of variance reduction techniques. These techniques can result in great improvements in simulation efficiency. This document describes the basic concepts of variance reduction (Part I), and a methodology for application of variance reduction techniques is presented in Part II. Appendices include the basic analytical ...

Multi-fold cross-validation is an established practice to estimate the error rate of a learning algorithm. Quantifying the variance reduction gains due to cross-validation has been challenging due to the inherent correlations introduced by the folds. In this work we introduce a new and weak measure called loss stability and relate the cross-validation performance to this measure; we also establ...

In this paper, we propose some novel normalization and fusion techniques for biometric matching score level fusion in person verification. While conventional matching score level fusion methods use global score statistics, we consider in this work both genuine and impostor statistics separately. Performing a joint mean normalization of the separate monomodal scores, multimodal scores with less ...

Variance within the manufacturing system leads to uneven shop loads, long manufacturing lead times, and unreliable customer service. This study compares techniques that reduce system variance to techniques that manage system variance. The study is placed in a dual resource constrained job shop. Results indicate that reducing system variance improves flow time and customer service performance me...

With the objective of supporting a more precise formulation of this question, and the discussion of the relevant issues, we begin with a formal definition of “language model”. We then propose a description of what “the Language Modeling approach” can be thought to consist of, in the context of IR research; first with a strict interpretation in mind, and then with a more informal view. We conclu...

Importance sampling is a widely used variance reduction technique to compute sample quantiles such as value-at-risk. The variance of the weight sample quantile estimator is usually a difficult quantity to compute. In this paper, we present the exact convergence rate and asymptotic distributions of the bootstrap variance estimators for quantiles of weighted empirical distributions. Under regular...

Importance sampling is a widely used variance reduction technique to compute sample quantiles such as value at risk. The variance of the weighted sample quantile estimator is usually a difficult quantity to compute. In this paper we present the exact convergence rate and asymptotic distributions of the bootstrap variance estimators for quantiles ofweighted empirical distributions. Under regular...