Shrinkage estimation usually reduces variance at the cost of bias. But when we care only about some parameters of a model, I show that we can reduce variance without incurring bias if we have additional information about the distribution of covariates. In a linear regression model with homoscedastic Normal noise, I consider shrinkage estimation of the nuisance parameters associated with control...

Hierarchical models are extensively studied and widely used in statistics and many other scientific areas. They provide an effective tool for combining information from similar resources and achieving partial pooling of inference. Since the seminal work by James and Stein (1961) and Stein (1962), shrinkage estimation has become one major focus for hierarchical models. For the homoscedastic norm...

This paper considers the problem of estimating a high-dimensional vector of parameters θ ∈ R from a noisy observation. The noise vector is i.i.d. Gaussian with known variance. For a squared-error loss function, the James-Stein (JS) estimator is known to dominate the simple maximum-likelihood (ML) estimator when the dimension n exceeds two. The JS-estimator shrinks the observed vector towards th...

The estimation problem in multivariate linear calibration with elliptical errors is considered under a loss function which can be derived from the Kullback-Leibler distance. First, we discuss the problem under normal errors and we give unbiased estimate of risk of an alternative estimator by means of the Stein and Stein-Haff identities for multivariate normal distribution. From the unbiased est...

Hans C. van Houwelingen Saskia le Cessie Department of Medical Statistics, Leiden, The Netherlands P.O.Box 9604 2300 RC Leiden, The Netherlands email: [email protected] Abstract A review is given of shrinkage and penalization as tools to improve predictive accuracy of regression models. The James-Stein estimator is taken as starting point. Procedures covered are the Pre-test Estimation, ...

This paper applies the bootstrap methods proposed by Efron (1979) to the Stein variance estimator proposed by Stein (1964). It is shown by Monte Carlo experiments that the parametric bootstrap yields the considerable accurate estimates of mean, standard error and confidence limits of the Stein variance estimator.

We present a procedure for effective estimation of entropy and mutual information from smallsample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperform...

We develop an adaptive monotone shrinkage estimator for regression models with the following characteristics: i) dense coefficients with small but important effects; ii) a priori ordering that indicates the probable predictive importance of the features. We capture both properties with an empirical Bayes estimator that shrinks coefficients monotonically with respect to their anticipated importa...

An estimator in the extended class of Stein estimators has two undesirable properties. For a small value of prior guess, it ignores the data. Moreover, for some cases its risk is not uniformly smaller than that of Stein estimator. We show that there exists a lower bound on T(S) to guarantee a smaller risk, and the resulting estimator does not ignore the data.