Assessing the temporal variation of macroeconomic forecasts by a panel of changing composition

This paper calls attention to the problem of changing panel composition in surveys of forecasters and documents the problem in the Survey of Professional Forecasters. To study the temporal variation of forecasts, we recommend analysis of the time series of predictions made by individual forecasters. This makes transparent the heterogeneity of the panel and avoids improper inferences due to changing panel composition. In the absence of knowledge of the process determining panel composition, we warn against the traditional practice of aggregate time series analysis, which conflates changes in the expectations of individual forecasters with changes in the composition of the panel. Should analysis of aggregated predictions be thought desirable as a simplifying device, we recommend analyses of sub‐panels of fixed composition. Copyright © 2010 John Wiley & Sons, Ltd.     

Do macroeconomic forecasters use macroeconomics to forecast?

Faced with an uncertain future, forecasters often rely on textbook relationships to build a coherent narrative for their macroeconomic forecasts. We focus on two cornerstones of modern macroeconomics – Okun’s law and the Phillips curve – and examine whether or not professionals forecast in a way that is consistent with these. Using microdata from the US, Euro Area, and UK surveys of professional forecasters, we examine forecasts over the period 1981-2017 at the level of the individual and across different time horizons. Our findings show that the majority of forecasters produce their forecasts in a manner that is consistent with macroeconomic theory.     

Differing behaviours of forecasters of UK GDP growth

The literature suggests that the dispersion of agents’ forecasts of an event flows from heterogeneity of beliefs and models. Using a data set of fixed event point forecasts of UK GDP growth by a panel of independent forecasters published by HM Treasury, we investigate three questions concerning this dispersion: (a) Are agent’s beliefs randomly distributed or do agents fall into groups with similar beliefs? (b) as agents revise their forecasts, what roles are played by their previous and consensus forecasts? and (c) is an agent’s private information of persistent value? We find that agents fall into four clusters, a large majority, a few pessimists, and two idiosyncratic agents. Our proposed model of forecast revisions shows agents are influenced positively by a change in the consensus forecast and negatively influenced by the previous distance of their forecast from the consensus. We show that the forecasts of a minority of agents significantly lead the consensus.     

Are professional forecasters overconfident?

Do professional forecasters have an accurate sense of the uncertainties surrounding their own forecasts? This paper examines forecaster overconfidence by comparing ex ante, surveyed forecaster uncertainty with ex post, realised uncertainty based on the dispersion of an individual’s forecast errors. Unlike the literature that focuses on consensus forecasts, our focus is at the level of the individual forecaster. Using microdata from the three major surveys of professional forecasters (Euro Area, US and UK), we examine real GDP growth forecasts over the period 1999–2015. Our findings show that overconfidence dominates among individual forecasters, particularly for longer forecast horizons, and that individual forecasters appear to have little understanding of their own uncertainty.     

Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7

In this paper, we use survey data to analyze the accuracy, unbiasedness and efficiency of professional macroeconomic forecasts. We analyze a large panel of individual forecasts that has not previously been analyzed in the literature. We provide evidence on the properties of forecasts for all G7-countries and for four different macroeconomic variables. Our results show a high degree of dispersion of forecast accuracy across forecasters. We also find that there are large differences in the performances of forecasters, not only across countries but also across different macroeconomic variables. In general, the forecasts tend to be biased in situations where the forecasters have to learn about large structural shocks or gradual changes in the trend of a variable. Furthermore, while a sizable fraction of forecasters seem to smooth their GDP forecasts significantly, this does not apply to forecasts made for other macroeconomic variables.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Rounding behaviour of professional macro-forecasters

The rounding of point forecasts of CPI inflation and the unemployment rate by U.S. Professional Forecasters is modest. There is little evidence that forecasts are rounded to a greater extent in response to higher perceived uncertainty surrounding future outcomes. There is clear evidence that the probability of decline forecasts are rounded: over half of the forecast probabilities of decline in the current quarter are multiples of ten. It is found here that the rounding of these probabilities correlates with worse accuracy, although it is also of note here that worse (less accurate) forecasters might round more as opposed to the degree of rounding per se worsening accuracy. By simulating the loss from rounding for a set of efficient forecasters, it is demonstrated that the explanation that respondents round otherwise efficient forecasts is implausible, and that the contribution of rounding is of minor importance.     

High-frequency credit spread information and macroeconomic forecast revision

We examine whether professional forecasters incorporate high-frequency information about credit conditions when revising their economic forecasts. Using a mixed data sampling regression approach, we find that daily credit spreads have significant predictive ability for monthly forecast revisions of output growth, at both the aggregate and individual forecast levels. The relationships are shown to be notably strong during ‘bad’ economic conditions, which suggests that forecasters anticipate more pronounced effects of credit tightening during economic downturns, indicating an amplification effect of financial developments on macroeconomic aggregates. The forecasts do not incorporate all financial information received in equal measures, implying the presence of information rigidities in the incorporation of credit spread information.     

How reliable are forecasts of GDP growth and inflation for countries with limited coverage?

This paper analyses the performance of GDP growth and inflation forecasts for 25 transition countries between 1994 and 2007, as provided by 13 international institutions, including multilateral, private and academic forecasters. The empirical results show that there is a positive correlation between the number of forecasters covering a given country and the forecast accuracy. Simple combined forecasts are shown to be unbiased and more accurate than most of the individual forecasters, although also inefficient. However, only a few institutions provide efficient and unbiased forecasts, with just one out of 13 forecasters providing both unbiased and efficient forecasts of both GDP growth and inflation in the observed period. The directional analysis shows a correct forecast of the change in the forecast indicator in over two thirds of cases. However, the eventual outcome is within the range of available forecasts in less than half of the cases, with more than 40% of outcomes for GDP growth above the highest forecast. Encouragingly, forecasts are shown to be improving over time and becoming more accurate with the increase in the number of forecasting institutions – forecast accuracy measured by mean absolute error improves by 0.3 percentage points for growth and by 0.2 percentage points for inflation for each additional institution providing forecasts.     

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policymakers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision-makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.     

Central banks’ interest rate projections and forecast coordination

We apply a simple test to study the effect of the publication of central banks’ interest-rate projections on the coordination of private-sector interest-rate forecasts. Our results indicate that the publication of interest-rate projections does not lead private-sector forecasters to coordinate their forecasts. In fact, private-sector forecasters rather seem to anti-coordinate, that is, they scatter their forecasts around a consensus forecast or around a central bank's interest-rate projections.     

What do professional forecasters actually predict?

This paper studies what professional forecasters predict. We use spectral analysis and state space modeling to decompose economic time series into trend, business cycle, and irregular components. We examine which components are captured by professional forecasters by regressing their forecasts on the estimated components extracted from both the spectral analysis and the state space model. For both decomposition methods, we find that, in the short run, the Survey of Professional Forecasters can predict almost all of the variation in the time series due to the trend and the business cycle, but that the forecasts contain little or no significant information about the variation in the irregular component.     

Does judgment improve macroeconomic density forecasts?

This paper presents empirical evidence on how judgmental adjustments affect the accuracy of macroeconomic density forecasts. Judgment is defined as the difference between professional forecasters’ densities and the forecast densities from statistical models. Using entropic tilting, we evaluate whether judgments about the mean, variance and skew improve the accuracy of density forecasts for UK output growth and inflation. We find that not all judgmental adjustments help. Judgments about point forecasts tend to improve density forecast accuracy at short horizons and at times of heightened macroeconomic uncertainty. Judgments about the variance hinder at short horizons, but can improve tail risk forecasts at longer horizons. Judgments about skew in general take value away, with gains seen only for longer horizon output growth forecasts when statistical models took longer to learn that downside risks had reduced with the end of the Great Recession. Overall, density forecasts from statistical models prove hard to beat.     

Asymmetry in unemployment rate forecast errors

Asymmetries in unemployment dynamics have been observed in the time series of a number of countries, including the United States. This paper studies asymmetries in unemployment rate forecast errors. We consider conditions under which optimal forecasts will display asymmetrically-distributed errors and how the degree of asymmetry might vary with the forecast horizon. Using data from the U.S. Survey of Professional Forecasters and the Federal Reserve Greenbook, we find substantial evidence of forecast error asymmetry, which tends to increase with the forecast horizon; we also find noteworthy differences in forecasts from these two sources. The results give insight into the abilities of professional forecasters to adapt their forecasts to asymmetry in underlying processes.     

Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)

While combining forecasts is well-known to reduce error, the question of how to best combine forecasts remains. Prior research suggests that combining is most beneficial when relying on diverse forecasts that incorporate different information. Here, I provide evidence in support of this hypothesis by analyzing data from the PollyVote project, which has published combined forecasts of the popular vote in U.S. presidential elections since 2004. Prior to the 2020 election, the PollyVote revised its original method of combining forecasts by, first, restructuring individual forecasts based on their underlying information and, second, adding naïve forecasts as a new component method. On average across the last 100 days prior to the five elections from 2004 to 2020, the revised PollyVote reduced the error of the original specification by eight percent and, with a mean absolute error (MAE) of 0.8 percentage points, was more accurate than any of its component forecasts. The results suggest that, when deciding about which forecasts to include in the combination, forecasters should be more concerned about the component forecasts’ diversity than their historical accuracy.     

Use and misuse of information in supply chain forecasting of promotion effects

Demand forecasting is critical to sales and operations planning (S&OP), but the effects of sales promotions can be difficult to forecast. Typically, a baseline statistical forecast is judgmentally adjusted on receipt of information from different departments. However, much of this information either has no predictive value or its value is unknown. Research into base rate discounting has suggested that such information may distract forecasters from the average uplift and reduce accuracy. This has been investigated in situations in which forecasters were able to adjust the statistical forecasts for promotions via a forecasting support system (FSS). In two ecologically valid experiments, forecasters were provided with the mean level of promotion uplift, a baseline statistical forecast, and quantitative and qualitative information. However, the forecasters were distracted from the base rate and misinterpreted the information available to them. These findings have important implications for the design of organizational S&OP processes, and for the implementation of FSSs.     

Forecasting from time series subject to sporadic perturbations: Effectiveness of different types of forecasting support

How effective are different approaches for the provision of forecasting support? Forecasts may be either unaided or made with the help of statistical forecasts. In practice, the latter are often crude forecasts that do not take sporadic perturbations into account. Most research considers forecasts based on series that have been cleansed of perturbation effects. This paper considers an experiment in which people made forecasts from time series that were disturbed by promotions. In all conditions, under-forecasting occurred during promotional periods and over-forecasting during normal ones. The relative sizes of these effects depended on the proportions of periods in the data series that contained promotions. The statistical forecasts improved the forecasting accuracy, not because they reduced these biases, but because they decreased the random error (scatter). The performance improvement did not depend on whether the forecasts were based on cleansed series. Thus, the effort invested in producing cleansed time series from which to forecast may not be warranted: companies may benefit from giving their forecasters even crude statistical forecasts. In a second experiment, forecasters received optimal statistical forecasts that took the effects of promotions into account fully. This increased the accuracy because the biases were almost eliminated and the random error was reduced by 20%. Thus, the additional effort required to produce forecasts that take promotional effects into account is worthwhile.     

Social networks and citizen election forecasting: The more friends the better

Most citizens correctly forecast which party will win a given election, and such forecasts usually have a higher level of accuracy than voter intention polls. How do citizens do it? We argue that social networks are a big part of the answer: much of what we know as citizens comes from our interactions with others. Previous research has considered only indirect characteristics of social networks when analyzing why citizens are good forecasters. We use a unique German survey and consider direct measures of social networks in order to explore their role in election forecasting. We find that three network characteristics –  size, political composition, and frequency of political discussion – are among the most important variables when predicting the accuracy of citizens’ election forecasts.     

Are GDP forecasts optimal? Evidence on European countries

We assess the accuracy of real GDP growth forecasts released by governments and international organizations for European countries in the years 1999–2017. We implement three testing procedures characterized by different assumptions on the forecasters’ loss functions. First, we test forecast rationality within the traditional approach based on a quadratic loss function (Mincer and Zarnowitz, 1969). Second, following Elliott, Timmermann and Komunjer (2005), we test rationality by allowing for a flexible loss function where the shape parameter driving the extent of asymmetry is unknown and estimated from the empirical distribution of forecast errors. Lastly, we implement the tests proposed by Patton and Timmermann (2007a) that hold regardless of the functional form of the loss function. We conclude that governmental forecasts are biased and not rational under a symmetric and quadratic loss function, but they are optimal under more general assumptions on the loss function. We also find that the preferences of forecasters change with the forecasting horizon: when moving from one- to two-year-ahead forecasts, the optimistic bias increases and the parameter of asymmetry in the loss function significantly increases.     

Forward guidance and expectation formation: A narrative approach

How forward guidance influences expectations is not fully understood. To study this, I construct central bank data that includes forward guidance and its attributes, central bank projections, and quantitative easing, which I combine with survey data. I describe how, when, and where forward guidance has worked. I estimate that forecasters revised their interest rate forecasts in the intended direction by five basis points on average following a forward guidance change. I provide estimates for The Federal Reserve, European Central Bank, Bank of England, Bank of Canada, Reserve Bank of Australia, Reserve Bank of New Zealand, Sveriges Riksbank, and Norges Bank.