Greedy Variance Estimation for the LASSO

Recent results have proven the minimax optimality of LASSO and related algorithms for noisy linear regression. However, these results tend to rely on variance estimators that are inefficient or optimizations that are slower than LASSO itself. We propose an efficient estimator for the noise variance in high dimensional linear regression that is significantly faster than LASSO, only requiring p matrix-vector multiplications. We prove this estimator is consistent with a good rate of convergence, under the condition that the design matrix satisfies the Restricted Isometry Property (RIP). In practice, our estimator vastly outperforms state of the art methods in terms of speed while incurring only a modest bias.