Homework Assignment 5

In class we discussed model selection tools in the context of building a high-quality prediction model, but cautioned that statistical inference (testing, confidence intervals, etc) were unreliable following model selection. Let us review why. In week 2 we showed that our uncertainty regarding b1 as an estimator of β1 comes from the fact that if the data were to change, so would our estimate. The standard errors we derived, and those reported by the computer, capture this uncertainty. However, when doing variable selection, an additional layer of uncertainty is introduced: the fact that as the data changes, the very model we select may change, above and beyond to the coefficient estimates within the model changing. This second layer of uncertainty is not at all reflected in the standard errors.