Combining probability forecasts

We consider different methods for combining probability forecasts. In empirical exercises, the data generating process of the forecasts and the event being forecast is not known, and therefore the optimal form of combination will also be unknown. We consider the properties of various combination schemes for a number of plausible data generating processes, and indicate which types of combinations are likely to be useful. We also show that whether forecast encompassing is found to hold between two rival sets of forecasts or not may depend on the type of combination adopted. The relative performances of the different combination methods are illustrated, with an application to predicting recession probabilities using leading indicators.     

A new method to assess the degree of information rigidity using fixed-event forecasts

We propose a new method to explore the information content of fixed-event forecasts and estimate structural parameters that are keys to sticky and noisy information models. Estimation follows a regression-based framework in which estimated coefficients map one-to-one with parameters that measure the degree of information rigidity. The statistical characterization of regression errors explores the laws that govern expectation formation under sticky and noisy information, that is, they are coherent with the theory. This strategy is still unexplored in the literature and potentially enhances the reliability of inference results. The method also allows linking estimation results to the signal-to-noise ratio, an important parameter of noisy information models. This task cannot be accomplished if one adopts an “agnostic” characterization of regression errors. With regard to empirical results, they show a substantial degree of information rigidity in the countries studied. They also suggest that the theoretical characterization of regression errors yields a more conservative picture of the uncertainty surrounding parameter estimates.     

A textual analysis of Bank of England growth forecasts

The Bank of England publishes a quarterly Inflation Report that provides numerical forecasts and a text discussion of its assessment of the UK economy. Previous research has evaluated the quantitative forecasts that are included in these reports, but we focus on the qualitative discussion of output growth, by using an in-sample textual analysis procedure to convert these qualitative assessments into a score for each report over the period 2005–2014. We also construct out-of-sample scores for reports before and after this period. We then compare the scores both to real-time output growth data and to the corresponding quantitative projections published by the bank. We find that overall developments in the UK economy were represented accurately in the text of the Inflation Report. Furthermore, efficiency regressions suggest that there is information in the text that could improve the Bank of England’s quantitative nowcasts and one-quarter-ahead forecasts.     

Combining survey long-run forecasts and nowcasts with BVAR forecasts using relative entropy

This paper constructs hybrid forecasts that combine forecasts from vector autoregressive (VAR) model(s) with both short- and long-term expectations from surveys. Specifically, we use the relative entropy to tilt one-step-ahead and long-horizon VAR forecasts to match the nowcasts and long-horizon forecasts from the Survey of Professional Forecasters. We consider a variety of VAR models, ranging from simple fixed-parameter to time-varying parameters. The results across models indicate meaningful gains in multi-horizon forecast accuracy relative to model forecasts that do not incorporate long-term survey conditions. Accuracy improvements are achieved for a range of variables, including those that are not tilted directly but are affected through spillover effects from tilted variables. The accuracy gains for hybrid inflation forecasts from simple VARs are substantial, statistically significant, and competitive to time-varying VARs, univariate benchmarks, and survey forecasts. We view our proposal as an indirect approach to accommodating structural change and moving end points.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Combining forecasts for universally optimal performance

There are two potential directions of forecast combination: combining for adaptation and combining for improvement. The former direction targets the performance of the best forecaster, while the latter attempts to combine forecasts to improve on the best forecaster. It is often useful to infer which goal is more appropriate so that a suitable combination method may be used. This paper proposes an AI-AFTER approach that can not only determine the appropriate goal of forecast combination but also intelligently combine the forecasts to automatically achieve the proper goal. As a result of this approach, the combined forecasts from AI-AFTER perform well universally in both adaptation and improvement scenarios. The proposed forecasting approach is implemented in our R package AIafter, which is available at https://github.com/weiqian1/AIafter.     

Combining exponential smoothing forecasts using Akaike weights

Simple forecast combinations such as medians and trimmed or winsorized means are known to improve the accuracy of point forecasts, and Akaike’s Information Criterion (AIC) has given rise to so-called Akaike weights, which have been used successfully to combine statistical models for inference and prediction in specialist fields, e.g., ecology and medicine. We examine combining exponential smoothing point and interval forecasts using weights derived from AIC, small-sample-corrected AIC and BIC on the M1 and M3 Competition datasets. Weighted forecast combinations perform better than forecasts selected using information criteria, in terms of both point forecast accuracy and prediction interval coverage. Simple combinations and weighted combinations do not consistently outperform one another, while simple combinations sometimes perform worse than single forecasts selected by information criteria. We find a tendency for a longer history to be associated with a better prediction interval coverage.     

Medium term growth forecasts: Experts vs. simple models

We compare the medium-term GDP growth forecasts generated by experts to those generated by simple models. This study analyzes a large set of forecasts that covers 48 countries from 1997 to 2016. Out-of-sample exercises indicate that no noticeable difference in performance is observed for advanced economies. In contrast, in the case of emerging economies, model forecasts perform better than expert forecasts. In addition, similar patterns are found for a collection of forecasts from a different set of experts, which suggests that the reported regularity is prevalent. Further analyses suggest that the documented difference in performance can be explained by an optimism bias, excessive reactions to innovations in growth trajectories, and insufficient responses to the information contained in the current account balance.     

Forecasting and forecast narratives: The Bank of England Inflation Reports

We analyze the narratives that accompany the numerical forecasts in the Bank of England’s Quarterly Inflation Reports, 1997–2018. We focus on whether the narratives contain useful information about the future course of key macro variables over and above the point predictions, in terms of whether the narratives can be used to enhance the accuracy of the numerical forecasts. We also consider whether the narratives are able to predict future changes in the numerical forecasts. We find that a measure of sentiment derived from the narratives can predict the errors in the numerical forecasts of output growth, but not of inflation. We find no evidence that past changes in sentiment predict subsequent changes in the point forecasts of output growth or of inflation, but do find that the adjustments to the numerical output growth forecasts have a systematic element.     

Do survey joiners and leavers differ from regular participants? The US SPF GDP growth and inflation forecasts

If ‘learning by doing’ is important for macro-forecasting, newcomers might be different from regular, established participants. Stayers may also differ from the soon-to-leave. We test these conjectures for macro-forecasters’ point predictions of output growth and inflation, and for their histogram forecasts. Histogram forecasts of inflation by both joiners and leavers are found to be less accurate, especially if we suppose that joiners take time to learn. For GDP growth, there is no evidence of differences between the groups in terms of histogram forecast accuracy, although GDP point forecasts by leavers are less accurate. These findings are predicated on forecasters being homogeneous within groups. Allowing for individual fixed effects suggests fewer differences, including leavers’ inflation histogram forecasts being no less accurate.     

Combining predictive distributions for the statistical post-processing of ensemble forecasts

Statistical post-processing techniques are now used widely for correcting systematic biases and errors in the calibration of ensemble forecasts obtained from multiple runs of numerical weather prediction models. A standard approach is the ensemble model output statistics (EMOS) method, which results in a predictive distribution that is given by a single parametric law, with parameters that depend on the ensemble members. This article assesses the merits of combining multiple EMOS models based on different parametric families. In four case studies with wind speed and precipitation forecasts from two ensemble prediction systems, we investigate the performances of state of the art forecast combination methods and propose a computationally efficient approach for determining linear pool combination weights. We study the performance of forecast combination compared to that of the theoretically superior but cumbersome estimation of a full mixture model, and assess which degree of flexibility of the forecast combination approach yields the best practical results for post-processing applications.     

Combining vector forecasts to predict thoroughbred horse race outcomes

A large body of empirical studies has shown that a forecast developed by combining individual base forecasts performs surprisingly well. Previous work on the combination of forecasts has been confined to the area of time series forecasting. This work extends the combination of forecasts technique into the domain of forecasting one-time competitive events, specifically the scaled, relative finishing position of horses in thoroughbred sprint races. The present research develops a framework for the selection of the base forecasts and selects 12 base forecasts for analysis. The performance of the combination of the base forecasts is assessed on a sample of sprint races. Results of the analysis strongly suggest that the combination approach is both appropriate and effective. Some differences in results between this work and previous work in the time series domain suggest promising avenues for future research.     

The uncertainty in extreme risk forecasts from covariate-augmented volatility models

For a GARCH-type volatility model with covariates, we derive asymptotically valid forecast intervals for risk measures, such as the Value-at-Risk or Expected Shortfall. To forecast these, we use estimators from extreme value theory. In the volatility model, we allow for leverage effects and the inclusion of exogenous variables, e.g., volatility indices or high-frequency volatility measures. In simulations, we find coverage of the forecast intervals to be adequate for sufficiently extreme risk levels and sufficiently large samples, which is consistent with theory. Finally, we investigate if covariate information from volatility indices or high-frequency data improves risk forecasts for major US stock indices. While—in our framework—volatility indices appear to be helpful in this regard, intra-day data are not.     

On economic evaluation of directional forecasts

It is commonly accepted that information is helpful if it can be exploited to improve a decision making process. In economics, decisions are often based on forecasts of the upward or downward movements of the variable of interest. We point out that directional forecasts can provide a useful framework for assessing the economic forecast value when loss functions (or success measures) are properly formulated to account for the realized signs and realized magnitudes of directional movements. We discuss a general approach to (directional) forecast evaluation which is based on the loss function proposed by Granger, Pesaran and Skouras. It is simple to implement and provides an economically interpretable loss/success functional framework. We show that, in addition, this loss function is more robust to outlying forecasts than traditional loss functions. As such, the measure of the directional forecast value is a readily available complement to the commonly used squared error loss criterion.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Commodity prices and the CPI: Cointegration, information, and signal extraction

This paper provides theoretical underpinnings for the commodity price/aggregate price relationship, discusses the conditions under which commodity prices are useful information variables for monetary policy, and provides empirical results which suggest why commodity prices have not been very useful for forecasting.     

The effect of nonstationarity on combined forecasts

Previous research on the combination of forecasts has, for the most part, implicitly assumed a stationary underlying process so that parameters could be estimated from historical data. While some models weight recent data more heavily in the estimation process in an attempt to provide more accurate parameter estimates in a nonstationary environment, no research to date has specifically examined the effects of nonstationarity on the performance of combining methods. This paper reports the results of a simulation study of the effects of nonstationarity (a shift in the process) on a range of forecast combination methods. Special attention is given to the relative performance of the methods in the time periods around the shift.     

Information rigidity in growth forecasts: Some cross-country evidence

We document information rigidity in forecasts of real GDP growth in 46 countries over the past two decades. We also investigate: (i) whether rigidities differ across countries, particularly between advanced countries and emerging markets; (ii) whether rigidities are lower around turning points in the economy, such as in times of recessions and crises; and (iii) how quickly forecasters incorporate news about growth in other countries into their growth forecasts, with a focus on the way in which advanced countries’ growth forecasts incorporate news about emerging market growth, and vice versa.     

Asymmetric loss in the Greenbook and the Survey of Professional Forecasters

This paper examines the forecast rationality of the Greenbook and the Survey of Professional Forecasters (SPF) under asymmetric loss functions, using the method proposed by Elliott, Komunjer, and Timmermann (2005) with a rolling window strategy. Over rolling periods, the degree and direction of the asymmetry in forecast loss functions are time-varying. While rationality under symmetric loss is often rejected, forecast rationality under asymmetric loss fails to be rejected over nearly all rolling periods. Besides, real output growth is consistently under-predicted in the 1990s, and the inflation rate is consistently over-predicted in the 1980s and 1990s. In general, inflation forecasts, especially for long horizons, exhibit greater levels of loss asymmetry in magnitude and frequency. The loss asymmetry of real output growth forecasts is more pronounced when the last revised vintage data are used than when the real-time vintage is used. All of these results hold for both the Greenbook and SPF forecasts. The results are also similar with the use of different sets of instrumental variables for estimating the asymmetric loss and testing for forecast rationality.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.