Kernel-based calibration diagnostics for recession and inflation probability forecasts

A probabilistic forecast is the estimated probability with which a future event will occur. One interesting feature of such forecasts is their calibration, or the match between the predicted probabilities and the actual outcome probabilities. Calibration has been evaluated in the past by grouping probability forecasts into discrete categories. We show here that we can do this without discrete groupings; the kernel estimators that we use produce efficiency gains and smooth estimated curves relating the predicted and actual probabilities. We use such estimates to evaluate the empirical evidence on the calibration error in a number of economic applications, including the prediction of recessions and inflation, using both forecasts made and stored in real time and pseudo-forecasts made using the data vintage available at the forecast date. The outcomes are evaluated using both first-release outcome measures and subsequent revised data. We find substantial evidence of incorrect calibration in professional forecasts of recessions and inflation from the SPF, as well as in real-time inflation forecasts from a variety of output gap models.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.     

Professional forecasters and real-time forecasting with a DSGE model

This paper analyses the real-time forecasting performance of the New Keynesian DSGE model of Galí, Smets and Wouters (2012), estimated on euro area data. It investigates the extent to which the inclusion of forecasts of inflation, GDP growth and unemployment by professional forecasters improve the forecasting performance. We consider two approaches for conditioning on such information. Under the “noise” approach, the mean professional forecasts are assumed to be noisy indicators of the rational expectations forecasts implied by the DSGE model. Under the “news” approach, it is assumed that the forecasts reveal the presence of expected future structural shocks in line with those estimated in the past. The forecasts of the DSGE model are compared with those from a Bayesian VAR model, an AR(1) model, a sample mean and a random walk.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Five dimensions of the uncertainty–disagreement linkage

We analyze the relationship between forecaster disagreement and macroeconomic uncertainty in the Euro area using data from the European Central Bank’s Survey of Professional Forecasters for the period 1999Q1–2018Q4 and find that disagreement is generally a poor proxy for uncertainty. However, the strength of this link varies with the dispersion statistic employed, the choice of either the point forecasts or the histogram means for calculating disagreement, the outcome variable considered and the forecast horizon. In contrast, distributional assumptions do not appear to be very influential. The relationship is weaker in subsamples before and after the outbreak of the Great Recession. Accounting for the forecasters’ entry to and exit from the survey has little impact on the results. We also show that survey-based uncertainty is associated with overall policy uncertainty, whereas forecaster disagreement is related more closely to the expected fluctuations on financial markets.     

On the forecasting performance of a small-scale DSGE model

Dynamic stochastic general equilibrium (DSGE) models have recently become standard tools for policy analysis. Nevertheless, their forecasting properties have still barely been explored. In this article, we address this problem by examining the quality of forecasts of the key U.S. economic variables: the three-month Treasury bill yield, the GDP growth rate and GDP price index inflation, from a small-size DSGE model, trivariate vector autoregression (VAR) models and the Philadelphia Fed Survey of Professional Forecasters (SPF). The ex post forecast errors are evaluated on the basis of the data from the period 1994–2006. We apply the Philadelphia Fed “Real-Time Data Set for Macroeconomists” to ensure that the data used in estimating the DSGE and VAR models was comparable to the information available to the SPF.Overall, the results are mixed. When comparing the root mean squared errors for some forecast horizons, it appears that the DSGE model outperforms the other methods in forecasting the GDP growth rate. However, this characteristic turned out to be statistically insignificant. Most of the SPF's forecasts of GDP price index inflation and the short-term interest rate are better than those from the DSGE and VAR models.     

On economic evaluation of directional forecasts

It is commonly accepted that information is helpful if it can be exploited to improve a decision making process. In economics, decisions are often based on forecasts of the upward or downward movements of the variable of interest. We point out that directional forecasts can provide a useful framework for assessing the economic forecast value when loss functions (or success measures) are properly formulated to account for the realized signs and realized magnitudes of directional movements. We discuss a general approach to (directional) forecast evaluation which is based on the loss function proposed by Granger, Pesaran and Skouras. It is simple to implement and provides an economically interpretable loss/success functional framework. We show that, in addition, this loss function is more robust to outlying forecasts than traditional loss functions. As such, the measure of the directional forecast value is a readily available complement to the commonly used squared error loss criterion.     

Evaluating the accuracy of value-at-risk forecasts: New multilevel tests

We propose independence and conditional coverage tests which are aimed at evaluating the accuracy of Value-at-Risk (VaR) forecasts from the same model at different confidence levels. The proposed procedures are multilevel tests, i.e., joint tests of several quantiles corresponding to different confidence levels. In a comprehensive Monte Carlo exercise, we document the superiority of the proposed tests with respect to existing multilevel tests. In an empirical application, we illustrate the implementation of the tests using several VaR models and daily data for 15 MSCI world indices.     

Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7

In this paper, we use survey data to analyze the accuracy, unbiasedness and efficiency of professional macroeconomic forecasts. We analyze a large panel of individual forecasts that has not previously been analyzed in the literature. We provide evidence on the properties of forecasts for all G7-countries and for four different macroeconomic variables. Our results show a high degree of dispersion of forecast accuracy across forecasters. We also find that there are large differences in the performances of forecasters, not only across countries but also across different macroeconomic variables. In general, the forecasts tend to be biased in situations where the forecasters have to learn about large structural shocks or gradual changes in the trend of a variable. Furthermore, while a sizable fraction of forecasters seem to smooth their GDP forecasts significantly, this does not apply to forecasts made for other macroeconomic variables.     

International propagation of shocks: A dynamic factor model using survey forecasts

This paper studies the pathways for the propagation of shocks across the G7 and major Asia-Pacific countries using multi-horizon forecasts of real GDP growth from 1995 to 2017. We show that if the forecasts are efficient in the long run, the results obtained using these forecasts are comparable to those obtained from the actual outturns. We measure global business cycle connectedness and study the impacts of both country-specific shocks and common international shocks using a panel factor structural VAR model. Our results suggest that there is a strong convergence of business cycles within the group of industrialized countries, as well as within the group of developing economies during non-recessionary periods. In particular, we find an increased decoupling between the industrialized and developing economies after the 2008 recession. However, the direction of shock spillovers during recessions and other crisis periods varies, depending on the nature and origin of the episode.     

Investigating the inefficiency of the CBO’s budgetary projections

This paper evaluates the efficiency of budgetary projections issued by the Congressional Budget Office (CBO) from 1984 to 2016 and investigates the cause of their inefficiency. The efficiency of both its budgetary and macroeconomic projections is rejected in many cases, especially for revenue projections. A comparison of forecast evaluations suggests that the inefficiency of revenue projections is likely due to the inefficiency of the underlying macroeconomic projections. By adjusting budgetary projections by the CBO in real time using macroeconomic forecasts by the Survey of Professional Forecasters, the accuracy of the former projections can be significantly improved in some cases, by up to 26% in terms of the root-mean-square prediction error.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

Asymmetry in unemployment rate forecast errors

Asymmetries in unemployment dynamics have been observed in the time series of a number of countries, including the United States. This paper studies asymmetries in unemployment rate forecast errors. We consider conditions under which optimal forecasts will display asymmetrically-distributed errors and how the degree of asymmetry might vary with the forecast horizon. Using data from the U.S. Survey of Professional Forecasters and the Federal Reserve Greenbook, we find substantial evidence of forecast error asymmetry, which tends to increase with the forecast horizon; we also find noteworthy differences in forecasts from these two sources. The results give insight into the abilities of professional forecasters to adapt their forecasts to asymmetry in underlying processes.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

Testing the predictive accuracy of COVID-19 forecasts

We test the predictive accuracy of forecasts of the number of COVID-19 fatalities produced by several forecasting teams and collected by the United States Centers for Disease Control and Prevention for the epidemic in the United States. We find three main results. First, at the short horizon (1 week ahead) no forecasting team outperforms a simple time-series benchmark. Second, at longer horizons (3 and 4 week ahead) forecasters are more successful and sometimes outperform the benchmark. Third, one of the best performing forecasts is the Ensemble forecast, that combines all available predictions using uniform weights. In view of these results, collecting a wide range of forecasts and combining them in an ensemble forecast may be a superior approach for health authorities, rather than relying on a small number of forecasts.     

What do professional forecasters actually predict?

This paper studies what professional forecasters predict. We use spectral analysis and state space modeling to decompose economic time series into trend, business cycle, and irregular components. We examine which components are captured by professional forecasters by regressing their forecasts on the estimated components extracted from both the spectral analysis and the state space model. For both decomposition methods, we find that, in the short run, the Survey of Professional Forecasters can predict almost all of the variation in the time series due to the trend and the business cycle, but that the forecasts contain little or no significant information about the variation in the irregular component.     

Raiders of the lost high-frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting

We introduce a new dataset of real gross domestic product (GDP) growth and core personal consumption expenditures (PCE) inflation forecasts produced by the staff of the Board of Governors of the Federal Reserve System. In contrast to the eight Greenbook forecasts a year the staff produce for Federal Open Market Committee (FOMC) meetings, our dataset has roughly weekly forecasts. We use these data to study whether the staff forecasts efficiently. Prespecified regressions of forecast errors on forecast revisions show the staff's GDP forecasts exhibit time-varying inefficiency between FOMC meetings, and also show some evidence for inefficient inflation forecasts.     

Comparing the accuracy of multivariate density forecasts in selected regions of the copula support

This paper develops a testing framework for comparing the predictive accuracy of competing multivariate density forecasts with different predictive copulas, focusing on specific parts of the copula support. The tests are framed in the context of the Kullback–Leibler Information Criterion, using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties for realistic sample sizes. In an empirical application to daily changes of yields on government bonds of the G7 countries we obtain insights into why the Student-t and Clayton mixture copula outperforms the other copulas considered; mixing in the Clayton copula with the t-copula is of particular importance to obtain high forecast accuracy in periods of jointly falling yields.