Do survey joiners and leavers differ from regular participants? The US SPF GDP growth and inflation forecasts

If ‘learning by doing’ is important for macro-forecasting, newcomers might be different from regular, established participants. Stayers may also differ from the soon-to-leave. We test these conjectures for macro-forecasters’ point predictions of output growth and inflation, and for their histogram forecasts. Histogram forecasts of inflation by both joiners and leavers are found to be less accurate, especially if we suppose that joiners take time to learn. For GDP growth, there is no evidence of differences between the groups in terms of histogram forecast accuracy, although GDP point forecasts by leavers are less accurate. These findings are predicated on forecasters being homogeneous within groups. Allowing for individual fixed effects suggests fewer differences, including leavers’ inflation histogram forecasts being no less accurate.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Assessing the uncertainty in central banks’ inflation outlooks

Recent research has found that macroeconomic survey forecasts of uncertainty exhibit several deficiencies, such as horizon-dependent biases and lower levels of accuracy than simple unconditional uncertainty forecasts. We examine the inflation uncertainty forecasts from the Bank of England, the Banco Central do Brasil, the Magyar Nemzeti Bank and the Sveriges Riksbank to assess whether central banks’ uncertainty forecasts might be subject to similar problems. We find that, while most central banks’ uncertainty forecasts also tend to be underconfident at short horizons and overconfident at longer horizons, they are mostly not significantly biased. Moreover, they tend to be at least as precise as unconditional uncertainty forecasts from two different approaches.     

Forecasting and forecast narratives: The Bank of England Inflation Reports

We analyze the narratives that accompany the numerical forecasts in the Bank of England’s Quarterly Inflation Reports, 1997–2018. We focus on whether the narratives contain useful information about the future course of key macro variables over and above the point predictions, in terms of whether the narratives can be used to enhance the accuracy of the numerical forecasts. We also consider whether the narratives are able to predict future changes in the numerical forecasts. We find that a measure of sentiment derived from the narratives can predict the errors in the numerical forecasts of output growth, but not of inflation. We find no evidence that past changes in sentiment predict subsequent changes in the point forecasts of output growth or of inflation, but do find that the adjustments to the numerical output growth forecasts have a systematic element.     

Forecasting the UK economy with a medium-scale Bayesian VAR

We estimate a Bayesian VAR (BVAR) for the UK economy and assess its performance in forecasting GDP growth and CPI inflation in real time relative to forecasts from COMPASS, the Bank of England’s DSGE model, and other benchmarks. We find that the BVAR outperformed COMPASS when forecasting both GDP and its expenditure components. In contrast, their performances when forecasting CPI were similar. We also find that the BVAR density forecasts outperformed those of COMPASS, despite under-predicting inflation at most forecast horizons. Both models over-predicted GDP growth at all forecast horizons, but the issue was less pronounced in the BVAR. The BVAR’s point and density forecast performances are also comparable to those of a Bank of England in-house statistical suite for both GDP and CPI inflation, as well as to the official Inflation Report projections. Our results are broadly consistent with the findings of similar studies for other advanced economies.     

Euro area real-time density forecasting with financial or labor market frictions

We compare real-time density forecasts for the euro area using three DSGE models. The benchmark is the Smets and Wouters model, and its forecasts of real GDP growth and inflation are compared with those from two extensions. The first adds financial frictions and expands the observables to include a measure of the external finance premium. The second allows for the extensive labor-market margin and adds the unemployment rate to the observables. The main question that we address is whether these extensions improve the density forecasts of real GDP and inflation and their joint forecasts up to an eight-quarter horizon. We find that adding financial frictions leads to a deterioration in the forecasts, with the exception of longer-term inflation forecasts and the period around the Great Recession. The labor market extension improves the medium- to longer-term real GDP growth and shorter- to medium-term inflation forecasts weakly compared with the benchmark model.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.     

A textual analysis of Bank of England growth forecasts

The Bank of England publishes a quarterly Inflation Report that provides numerical forecasts and a text discussion of its assessment of the UK economy. Previous research has evaluated the quantitative forecasts that are included in these reports, but we focus on the qualitative discussion of output growth, by using an in-sample textual analysis procedure to convert these qualitative assessments into a score for each report over the period 2005–2014. We also construct out-of-sample scores for reports before and after this period. We then compare the scores both to real-time output growth data and to the corresponding quantitative projections published by the bank. We find that overall developments in the UK economy were represented accurately in the text of the Inflation Report. Furthermore, efficiency regressions suggest that there is information in the text that could improve the Bank of England’s quantitative nowcasts and one-quarter-ahead forecasts.     

How accurate are government forecasts of economic fundamentals? The case of Taiwan

A government’s ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.In this paper, we develop a methodology for evaluating non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that the undocumented knowledge of the Taiwanese government reduces forecast errors substantially.     

Are professional forecasters overconfident?

Do professional forecasters have an accurate sense of the uncertainties surrounding their own forecasts? This paper examines forecaster overconfidence by comparing ex ante, surveyed forecaster uncertainty with ex post, realised uncertainty based on the dispersion of an individual’s forecast errors. Unlike the literature that focuses on consensus forecasts, our focus is at the level of the individual forecaster. Using microdata from the three major surveys of professional forecasters (Euro Area, US and UK), we examine real GDP growth forecasts over the period 1999–2015. Our findings show that overconfidence dominates among individual forecasters, particularly for longer forecast horizons, and that individual forecasters appear to have little understanding of their own uncertainty.     

Are macroeconomic density forecasts informative?

We consider whether survey density forecasts (such as the inflation and output growth histograms of the US Survey of Professional Forecasters) are superior to unconditional density forecasts. The unconditional forecasts assume that the average level of uncertainty that has been experienced in the past will continue to prevail in the future, whereas the SPF projections ought to be adapted to the current conditions and the outlook at each forecast origin. The SPF forecasts might be expected to outperform the unconditional densities at the shortest horizons, but it transpires that such is not the case for the aggregate forecasts of either variable, or for the majority of the individual respondents for forecasting inflation.     

Raiders of the lost high-frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting

We introduce a new dataset of real gross domestic product (GDP) growth and core personal consumption expenditures (PCE) inflation forecasts produced by the staff of the Board of Governors of the Federal Reserve System. In contrast to the eight Greenbook forecasts a year the staff produce for Federal Open Market Committee (FOMC) meetings, our dataset has roughly weekly forecasts. We use these data to study whether the staff forecasts efficiently. Prespecified regressions of forecast errors on forecast revisions show the staff's GDP forecasts exhibit time-varying inefficiency between FOMC meetings, and also show some evidence for inefficient inflation forecasts.     

How reliable are forecasts of GDP growth and inflation for countries with limited coverage?

This paper analyses the performance of GDP growth and inflation forecasts for 25 transition countries between 1994 and 2007, as provided by 13 international institutions, including multilateral, private and academic forecasters. The empirical results show that there is a positive correlation between the number of forecasters covering a given country and the forecast accuracy. Simple combined forecasts are shown to be unbiased and more accurate than most of the individual forecasters, although also inefficient. However, only a few institutions provide efficient and unbiased forecasts, with just one out of 13 forecasters providing both unbiased and efficient forecasts of both GDP growth and inflation in the observed period. The directional analysis shows a correct forecast of the change in the forecast indicator in over two thirds of cases. However, the eventual outcome is within the range of available forecasts in less than half of the cases, with more than 40% of outcomes for GDP growth above the highest forecast. Encouragingly, forecasts are shown to be improving over time and becoming more accurate with the increase in the number of forecasting institutions – forecast accuracy measured by mean absolute error improves by 0.3 percentage points for growth and by 0.2 percentage points for inflation for each additional institution providing forecasts.     

Medium term growth forecasts: Experts vs. simple models

We compare the medium-term GDP growth forecasts generated by experts to those generated by simple models. This study analyzes a large set of forecasts that covers 48 countries from 1997 to 2016. Out-of-sample exercises indicate that no noticeable difference in performance is observed for advanced economies. In contrast, in the case of emerging economies, model forecasts perform better than expert forecasts. In addition, similar patterns are found for a collection of forecasts from a different set of experts, which suggests that the reported regularity is prevalent. Further analyses suggest that the documented difference in performance can be explained by an optimism bias, excessive reactions to innovations in growth trajectories, and insufficient responses to the information contained in the current account balance.     

Combining forecasts: Performance and coherence

There is general agreement in many forecasting contexts that combining individual predictions leads to better final forecasts. However, the relative error reduction in a combined forecast depends upon the extent to which the component forecasts contain unique/independent information. Unfortunately, obtaining independent predictions is difficult in many situations, as these forecasts may be based on similar statistical models and/or overlapping information. The current study addresses this problem by incorporating a measure of coherence into an analytic evaluation framework so that the degree of independence between sets of forecasts can be identified easily. The framework also decomposes the performance and coherence measures in order to illustrate the underlying aspects that are responsible for error reduction. The framework is demonstrated using UK retail prices index inflation forecasts for the period 1998–2014, and implications for forecast users are discussed.     

Forecast revisions of Mexican inflation and GDP growth

We analyze the forecasts of inflation and GDP growth contained in the Banco de México’s Survey of Professional Forecasters for the period 1995–2009. The forecasts are for the current and the following year, and comprise an unbalanced three-dimensional panel with multiple individual forecasters, target years, and forecast horizons. The fixed-event nature of the forecasts enables us to examine their efficiency by looking at the revision process. The panel structure allows us to control for aggregate shocks and to construct a measure of the news that impacted expectations in the period under study. We find that respondents anchor to their initial forecasts, updating their revisions smoothly as they receive more information. In addition, they do not seem to use publicly-known information in an efficient manner. These inefficiencies suggest clear areas of opportunity for improving the accuracy of the forecasts, for instance by taking into account the positive autocorrelation found in forecast revisions.     

Central bank’s forecasts and lack of transparency: An assessment of the effect on private expectations in a large emerging economy

We analyze whether the Central Bank of Brazil’s Inflation Reports projections influences the private’s inflation expectations. Specifically, we investigate how the central bank’s inflation forecasts affect the private sector’s inflation expectations through a qualitative and quantitative examination of the disagreement measure between them. Furthermore, we appraise if the lack of transparency resulting from the difference between the central bank’s inflation forecasts and the realized inflation affects the private’s inflation expectations. Although the findings confirm the previous studies that point out that the central bank transparency can affect the readjustment of market expectations, the results do not rule out the possibility of the central bank’s forecast and private’s inflation expectations being affected reciprocally.     

Too similar to combine? On negative weights in forecast combination

This paper provides the first thorough investigation of the negative weights that can emerge when combining forecasts. The usual practice in the literature is to consider only convex combinations and ignore or trim negative weights, i.e., set them to zero. This default strategy has its merits, but it is not optimal. We study the problem from various angles, and the main conclusion is that negative weights emerge when highly correlated forecasts with similar variances are combined. In this situation, the estimated weights have large variances, and trimming reduces the variance of the weights and improves the combined forecast. The threshold of zero is arbitrary and can be improved. We propose an optimal trimming threshold, i.e., an additional tuning parameter to improve forecasting performance. The effects of optimal trimming are demonstrated in simulations. In the empirical example using the European Central Bank Survey of Professional Forecasters, we find that the new strategy performs exceptionally well and can deliver improvements of more than 10% for inflation, up to 20% for GDP growth, and more than 20% for unemployment forecasts relative to the equal-weight benchmark.     

Asymmetric loss in the Greenbook and the Survey of Professional Forecasters

This paper examines the forecast rationality of the Greenbook and the Survey of Professional Forecasters (SPF) under asymmetric loss functions, using the method proposed by Elliott, Komunjer, and Timmermann (2005) with a rolling window strategy. Over rolling periods, the degree and direction of the asymmetry in forecast loss functions are time-varying. While rationality under symmetric loss is often rejected, forecast rationality under asymmetric loss fails to be rejected over nearly all rolling periods. Besides, real output growth is consistently under-predicted in the 1990s, and the inflation rate is consistently over-predicted in the 1980s and 1990s. In general, inflation forecasts, especially for long horizons, exhibit greater levels of loss asymmetry in magnitude and frequency. The loss asymmetry of real output growth forecasts is more pronounced when the last revised vintage data are used than when the real-time vintage is used. All of these results hold for both the Greenbook and SPF forecasts. The results are also similar with the use of different sets of instrumental variables for estimating the asymmetric loss and testing for forecast rationality.     

Combining survey long-run forecasts and nowcasts with BVAR forecasts using relative entropy

This paper constructs hybrid forecasts that combine forecasts from vector autoregressive (VAR) model(s) with both short- and long-term expectations from surveys. Specifically, we use the relative entropy to tilt one-step-ahead and long-horizon VAR forecasts to match the nowcasts and long-horizon forecasts from the Survey of Professional Forecasters. We consider a variety of VAR models, ranging from simple fixed-parameter to time-varying parameters. The results across models indicate meaningful gains in multi-horizon forecast accuracy relative to model forecasts that do not incorporate long-term survey conditions. Accuracy improvements are achieved for a range of variables, including those that are not tilted directly but are affected through spillover effects from tilted variables. The accuracy gains for hybrid inflation forecasts from simple VARs are substantial, statistically significant, and competitive to time-varying VARs, univariate benchmarks, and survey forecasts. We view our proposal as an indirect approach to accommodating structural change and moving end points.