Kernel-based calibration diagnostics for recession and inflation probability forecasts

A probabilistic forecast is the estimated probability with which a future event will occur. One interesting feature of such forecasts is their calibration, or the match between the predicted probabilities and the actual outcome probabilities. Calibration has been evaluated in the past by grouping probability forecasts into discrete categories. We show here that we can do this without discrete groupings; the kernel estimators that we use produce efficiency gains and smooth estimated curves relating the predicted and actual probabilities. We use such estimates to evaluate the empirical evidence on the calibration error in a number of economic applications, including the prediction of recessions and inflation, using both forecasts made and stored in real time and pseudo-forecasts made using the data vintage available at the forecast date. The outcomes are evaluated using both first-release outcome measures and subsequent revised data. We find substantial evidence of incorrect calibration in professional forecasts of recessions and inflation from the SPF, as well as in real-time inflation forecasts from a variety of output gap models.     

Five dimensions of the uncertainty–disagreement linkage

We analyze the relationship between forecaster disagreement and macroeconomic uncertainty in the Euro area using data from the European Central Bank’s Survey of Professional Forecasters for the period 1999Q1–2018Q4 and find that disagreement is generally a poor proxy for uncertainty. However, the strength of this link varies with the dispersion statistic employed, the choice of either the point forecasts or the histogram means for calculating disagreement, the outcome variable considered and the forecast horizon. In contrast, distributional assumptions do not appear to be very influential. The relationship is weaker in subsamples before and after the outbreak of the Great Recession. Accounting for the forecasters’ entry to and exit from the survey has little impact on the results. We also show that survey-based uncertainty is associated with overall policy uncertainty, whereas forecaster disagreement is related more closely to the expected fluctuations on financial markets.     

On the forecasting performance of a small-scale DSGE model

Dynamic stochastic general equilibrium (DSGE) models have recently become standard tools for policy analysis. Nevertheless, their forecasting properties have still barely been explored. In this article, we address this problem by examining the quality of forecasts of the key U.S. economic variables: the three-month Treasury bill yield, the GDP growth rate and GDP price index inflation, from a small-size DSGE model, trivariate vector autoregression (VAR) models and the Philadelphia Fed Survey of Professional Forecasters (SPF). The ex post forecast errors are evaluated on the basis of the data from the period 1994–2006. We apply the Philadelphia Fed “Real-Time Data Set for Macroeconomists” to ensure that the data used in estimating the DSGE and VAR models was comparable to the information available to the SPF.Overall, the results are mixed. When comparing the root mean squared errors for some forecast horizons, it appears that the DSGE model outperforms the other methods in forecasting the GDP growth rate. However, this characteristic turned out to be statistically insignificant. Most of the SPF's forecasts of GDP price index inflation and the short-term interest rate are better than those from the DSGE and VAR models.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.     

Investigating the inefficiency of the CBO’s budgetary projections

This paper evaluates the efficiency of budgetary projections issued by the Congressional Budget Office (CBO) from 1984 to 2016 and investigates the cause of their inefficiency. The efficiency of both its budgetary and macroeconomic projections is rejected in many cases, especially for revenue projections. A comparison of forecast evaluations suggests that the inefficiency of revenue projections is likely due to the inefficiency of the underlying macroeconomic projections. By adjusting budgetary projections by the CBO in real time using macroeconomic forecasts by the Survey of Professional Forecasters, the accuracy of the former projections can be significantly improved in some cases, by up to 26% in terms of the root-mean-square prediction error.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.     

What do professional forecasters actually predict?

This paper studies what professional forecasters predict. We use spectral analysis and state space modeling to decompose economic time series into trend, business cycle, and irregular components. We examine which components are captured by professional forecasters by regressing their forecasts on the estimated components extracted from both the spectral analysis and the state space model. For both decomposition methods, we find that, in the short run, the Survey of Professional Forecasters can predict almost all of the variation in the time series due to the trend and the business cycle, but that the forecasts contain little or no significant information about the variation in the irregular component.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

Forecasting macroeconomic variables using disaggregate survey data

We construct factor models based on disaggregate survey data for forecasting national aggregate macroeconomic variables. Our methodology applies regional and sectoral factor models to Norges Bank’s regional survey and to the Swedish Business Tendency Survey. The analysis identifies which of the pieces of information extracted from the individual regions in Norges Bank’s survey and the sectors for the two surveys perform particularly well at forecasting different variables at various horizons. The results show that several factor models beat an autoregressive benchmark in forecasting inflation and the unemployment rate. However, the factor models are most successful at forecasting GDP growth. Forecast combinations using the past performances of regional and sectoral factor models yield the most accurate forecasts in the majority of the cases.     

Influence diagnostic in survey sampling: Estimating the conditional bias

The conditional bias has been proposed by Moreno Rebollo et al. (1999) as an influence diagnostic in survey sampling, when the inference is based on the randomization distribution generated by a random sampling. The conditional bias is a population parameter. So, from an applied point of view, it must be estimated. In this paper, we propose an estimator of the conditional bias and we study conditions that guarantee its unbiasedness. The results are applied in a Simple Random Sampling and in a Proportional Probability Aggregated Size Sampling, when the ratio estimator is used. Received October 2000     

Anomalies in macroeconomic prediction errors–evidence from Chilean private forecasters

Part of a prediction is the judgment applied by the forecaster. This judgmental input may be affected by the forecaster’s mood swings, which have been shown to affect, for example, stock market returns. The present paper analyzes the extent to which mood (approximated by the development in sentiment indicators) affects macroeconomic prediction errors; i.e., whether it explains part of the prediction bias. The evidence suggests that mood can explain part of the error in inflation and output growth predictions, and hence, that anomalies should be taken into account when trying to understand expectation formation and assess the uncertainty related to private forecasters’ point predictions.