Beginning in 1998, U.S. commercial banks may determine their regulatory capital requirements for financial market risk exposure using value-at-risk (VaR) models. Currently, regulators have available three hypothesis-testing methods for evaluating the accuracy of VaR models: the binomial, interval forecast and distribution forecast methods. Given the low power often exhibited by their correspond...

Evaluation methodologies for rare events from meteorology, psychology and medical diagnosis are used to examine the value of probability forecasts of real GDP declines during the current (Q0) and each of the next four quarters (Q1-Q4) using data from the Survey of Professional Forecasters. We study the quality of these probability forecasts in terms of calibration, resolution, odds ratio, the r...

Macroeconomic forecasts are frequently produced, widely published, intensively discussed and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes th...

Economic policy agencies produce forecasts with accompanying narratives, and base policy changes on the resulting anticipated developments in the target variables. Systematic forecast failure, defined as large, persistent deviations of the outturns from the numerical forecasts, can make the associated narrative false, which would in turn question the validity of the entailed policy implementati...

We consider evaluating the UK Monetary Policy Committee’s inflation density forecasts using probability integral transform goodness-of-fit tests and measures related to a decision/cost based approach. In implementing the decision-based approach a number of difficulties arise in the absence of full information on payoffs, and when the actual and forecast inflation probabilities may depend on the...

Evaluating the predictive ability of mortality forecasts is important yet difficult. Death rates and mean lifespan are basic life table functions typically used to analyze to what extent the forecasts deviate from their realized values. Although these parameters are useful for specifying precisely how mortality has been forecasted, they cannot be used to assess whether the underlying mortality ...

We propose and evaluate several new scoring rules based on likelihood ratios, for comparing density forecasts in the context of VaR modelling and expected loss estimation. Our approach is motivated by the observation that some existing weighted scoring rules tend to favour fat-tailed predictive densities over thin-tailed predictive densities. Rather than restricting the weight functions, we imp...

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT...

Macroeconomic forecasts are frequently produced, published, discussed and used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyse some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are gene...

This article examines four problems with past evaluations of presidential election forecasting and suggests one aspect of the models that could be improved. Past criticism has had problems with establishing an overall appraisal of the forecasting equations, in assessing the accuracy of both the forecasting models and their forecasts of individual election results, in identifying the theoretical...