Many multivariate Gaussian models can conveniently be split into independent, block-wise problems. Common settings where this situation arises are balanced ANOVA models, balanced longitudinal models, and certain block-wise shrinkage estimators in nonparametric regression estimation involving orthogonal bases such as Fourier or wavelet bases. It is well known that the standard, least squares est...

This paper proposes a new two stage least squares (2SLS) estimator which is consistent and asymptotically normal in the presence of many weak instruments and heteroskedasticity. The first stage of the estimator consists of two components: first, an adaptive absolute shrinkage and selection operator (LASSO) that selects the instruments; and second, an OLS regression with the selected regressors....

We present a procedure for effective estimation of entropy and mutual information from smallsample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperform...

An important application of DNAmicroarray data is cancer classification. Because of the high-dimensionality problem of microarray data, gene selection approaches are often employed to support the expert systems in diagnostic capability of cancer with high classification accuracy. Penalized logistic regression using the least absolute shrinkage and selection operator (LASSO) is one of the key st...

Starting from a practical implementation of Roth and Fisher’s algorithm to solve a Lasso-type problem, we propose and study the Active Set Iterative Shrinkage/Thresholding Algorithm (AS-ISTA). The convergence is proven by observing that the algorithm can be seen as a particular case of a coordinate gradient descent algorithm with a Gauss-Southwell-r rule. We provide experimental evidence that t...

We study the degrees of freedom in shrinkage estimation of the regression coefficients. Generalizing the idea of the Lasso, we consider the problem of estimating the coefficients by the projection of the ordinary least squares estimator onto a closed convex set. Then an unbiased estimator of the degrees of freedom is derived in terms of geometric quantities under a smoothness condition on the b...

An important question in feature selection is whether a selection strategy recovers the “true” set of features, given enough data. We study this question in the context of the popular Least Absolute Shrinkage and Selection Operator (Lasso) feature selection strategy. In particular, we consider the scenario when the model is misspecified so that the learned model is linear while the underlying r...