Honest calibration assessment for binary outcome predictions

Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted occur with probability $x$, it should materialize approximately that frequency, which means the so-called calibration curve $p(\cdot)$ equal identity, i.e., $p(x) = x$ for all $x$ in unit interval. We propose honest assessment based on novel confidence bands curve, are valid subject only natural assumption of isotonicity. Besides testing classical goodness-of-fit null hypothesis perfect calibration, our facilitate inverted tests whose rejection allows sought-after conclusion a sufficiently well-specified model. show have finite-sample coverage guarantee, narrower than those existing approaches, and adapt local smoothness $p$ variance observations. In application modelling infant having low birth weight, bounds give informative insights into model calibration.