Evaluating quantile forecasts in the M5 uncertainty competition

Probabilistic forecasts are necessary for robust decisions in the face of uncertainty. The M5 Uncertainty competition required participating teams to forecast nine quantiles for unit sales of various products at various aggregation levels and for different time horizons. This paper evaluates the forecasting performance of the quantile forecasts at different aggregation levels and at different quantile levels. We contrast this with some theoretical predictions, and discuss potential implications and promising future research directions for the practice of probabilistic forecasting.     

Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)

While combining forecasts is well-known to reduce error, the question of how to best combine forecasts remains. Prior research suggests that combining is most beneficial when relying on diverse forecasts that incorporate different information. Here, I provide evidence in support of this hypothesis by analyzing data from the PollyVote project, which has published combined forecasts of the popular vote in U.S. presidential elections since 2004. Prior to the 2020 election, the PollyVote revised its original method of combining forecasts by, first, restructuring individual forecasts based on their underlying information and, second, adding naïve forecasts as a new component method. On average across the last 100 days prior to the five elections from 2004 to 2020, the revised PollyVote reduced the error of the original specification by eight percent and, with a mean absolute error (MAE) of 0.8 percentage points, was more accurate than any of its component forecasts. The results suggest that, when deciding about which forecasts to include in the combination, forecasters should be more concerned about the component forecasts’ diversity than their historical accuracy.     

Judgmental adjustment of forecasts: A comparison of methods

Attention has recently been given to combinations of subjective and objective forecasts to improve forecast accuracy. This research offers an extension on this theme by comparing two methods that can be used to adjust an objective forecast. Wolfe and Flores (1990) show that forecasts can be judgmentally adjusted by analysts using a structured approach based on Saaty's analytic hierarchy process (AHP). In this study, the centroid method is introduced as a vehicle for forecast adjustment and is compared to the AHP. While the AHP allows for finer tuning in reflecting decision maker judgement, the centroid method produces very similar results and is much simpler to use in the forecast adjustment process.     

Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7

In this paper, we use survey data to analyze the accuracy, unbiasedness and efficiency of professional macroeconomic forecasts. We analyze a large panel of individual forecasts that has not previously been analyzed in the literature. We provide evidence on the properties of forecasts for all G7-countries and for four different macroeconomic variables. Our results show a high degree of dispersion of forecast accuracy across forecasters. We also find that there are large differences in the performances of forecasters, not only across countries but also across different macroeconomic variables. In general, the forecasts tend to be biased in situations where the forecasters have to learn about large structural shocks or gradual changes in the trend of a variable. Furthermore, while a sizable fraction of forecasters seem to smooth their GDP forecasts significantly, this does not apply to forecasts made for other macroeconomic variables.     

GDP forecasts: Informational asymmetry of the SPF and FOMC minutes

This paper emphasizes asymmetric information about the U.S. economy between the FOMC and SPF. Following Stekler and Symington (2016), it extends their text-based FOMC minutes index (FMI) of economic outlook to 1986-2016. Following Ericsson (2016), it employs truncation adjustment indicators and reinterprets the FMI calibrations as the policy-makers forecasts of the GDP growth, which carry information about the staff Greenbook forecasts prepared prior to the bi-quarterly FOMC meetings. Tests confirm unbiasedness and rationality of these forecasts. The encompassing tests indicate that both the FMI and SPF forecasts contain unique information beyond their alternative’s information set and can be weighted equally. The orthogonality tests suggest that the SPF efficiently use all their information set but could gain if the FOMC minutes were published without a lag, while the policy-makers rely mostly on their projections made earlier in the meetings, and could benefit from incorporating the already published SPF forecasts.     

Combining predictive distributions for the statistical post-processing of ensemble forecasts

Statistical post-processing techniques are now used widely for correcting systematic biases and errors in the calibration of ensemble forecasts obtained from multiple runs of numerical weather prediction models. A standard approach is the ensemble model output statistics (EMOS) method, which results in a predictive distribution that is given by a single parametric law, with parameters that depend on the ensemble members. This article assesses the merits of combining multiple EMOS models based on different parametric families. In four case studies with wind speed and precipitation forecasts from two ensemble prediction systems, we investigate the performances of state of the art forecast combination methods and propose a computationally efficient approach for determining linear pool combination weights. We study the performance of forecast combination compared to that of the theoretically superior but cumbersome estimation of a full mixture model, and assess which degree of flexibility of the forecast combination approach yields the best practical results for post-processing applications.     

The effect of nonstationarity on combined forecasts

Previous research on the combination of forecasts has, for the most part, implicitly assumed a stationary underlying process so that parameters could be estimated from historical data. While some models weight recent data more heavily in the estimation process in an attempt to provide more accurate parameter estimates in a nonstationary environment, no research to date has specifically examined the effects of nonstationarity on the performance of combining methods. This paper reports the results of a simulation study of the effects of nonstationarity (a shift in the process) on a range of forecast combination methods. Special attention is given to the relative performance of the methods in the time periods around the shift.     

A textual analysis of Bank of England growth forecasts

The Bank of England publishes a quarterly Inflation Report that provides numerical forecasts and a text discussion of its assessment of the UK economy. Previous research has evaluated the quantitative forecasts that are included in these reports, but we focus on the qualitative discussion of output growth, by using an in-sample textual analysis procedure to convert these qualitative assessments into a score for each report over the period 2005–2014. We also construct out-of-sample scores for reports before and after this period. We then compare the scores both to real-time output growth data and to the corresponding quantitative projections published by the bank. We find that overall developments in the UK economy were represented accurately in the text of the Inflation Report. Furthermore, efficiency regressions suggest that there is information in the text that could improve the Bank of England’s quantitative nowcasts and one-quarter-ahead forecasts.     

Linking words in economic discourse: Implications for macroeconomic forecasts

This paper develops indicators of unstructured press information by exploiting word vector representations. A model is trained using a corpus covering 90 years of Wall Street Journal content. The information content of the indicators is assessed through business cycle forecast exercises. The vector representations can learn meaningful word associations that are exploited to construct indicators of uncertainty. In-sample and out-of-sample forecast exercises show that the indicators contain valuable information regarding future economic activity. The combination of indices associated with different subjective states (e.g., uncertainty, fear, pessimism) results in further gains in information content. The documented performance is unmatched by previous dictionary-based word counting techniques proposed in the literature.     

Assessing the accuracy of electricity production forecasts in developing countries

This study assesses the accuracy of time series econometric methods for forecasting electricity production in developing countries. An analysis of the historical time series for 106 developing countries over the period 1960–2012 demonstrates that econometric forecasts are highly accurate for the majority of these countries. These forecasts have much smaller errors than the predictions of simple heuristic models, which assume that electricity production grows at an exogenous rate or is proportional to the real GDP growth. However, the quality of the forecasts diminishes for the countries and regions, where rapid economic and structural transformation makes it difficult to establish stable historical production trends.     

The effects of families and ability on men's education and earnings in Britain

The paper estimates the returns to education for a cohort of men born in Britain in March 1958 who have been followed since birth until the age of 33. The data used has a wealth of information on family background including parental education, social class and interest shown in the child's education as well as measures of ability. These variables are typically missing in studies looking at the returns to schooling. In the paper, we find that the average return to an additional year of full-time education for the UK men is somewhere around 5 1/2% to 6% even after correcting for the effects of measurement error. The paper also presents evidence that the returns to an additional year of schooling in the UK are heterogeneous. In particular, the paper finds some evidence that men with lower tastes for education, have significantly higher marginal returns to education. The results of the paper suggest that recent IV estimates of the returns to schooling in the UK, which exceed typical OLS estimates, may overestimate the average marginal return for the population of men as a whole.     

Accuracy of German federal election forecasts, 2013 & 2017

The present study reviews the accuracy of four methods (polls, prediction markets, expert judgment, and quantitative models) for forecasting the two German federal elections in 2013 and 2017. On average across both elections, polls and prediction markets were most accurate, while experts and quantitative models were least accurate. However, the accuracy of individual forecasts did not correlate across elections. That is, the methods that were most accurate in 2013 did not perform particularly well in 2017. A combined forecast, calculated by averaging forecasts within and across methods, was more accurate than three of the four component forecasts. The results conform to prior research on US presidential elections in showing that combining is effective in generating accurate forecasts and avoiding large errors.     

The dynamics of utility in the neoclassical OLG model

This paper develops a method to study how life-cycle utility of a sequence of cohorts converges to its steady state level in the neoclassical 2-period overlapping generations model. This method allows to characterize utility changes associated with marginal variations in exogenous policy parameters along the entire transition path between two steady states. At the same time, it is not more complicated than a pure steady state analysis. Moreover, it can be applied to economies for which an explicit solution of the transition path is not available.     

Statistical efficiency in the linear combination of forecasts

In seeking an efficient combination of forecasts which minimises the forecast error variance, many methods have been suggested. Through analysis, simulation and case studies, this paper seeks to develop insights into the statistical circumstances which influence the relative accuracy of six of these methods. The six methods chosen have all been advocated in various publications and consist of ‘equal weighting’ (i.e., pooled average), ‘optimal’ (i.e., error variance minimising), ‘optimal with independence assumption’ (i.e., error variance minimising assuming zero correlation between individual forecast errors) and three variations on the formulation of a Bayesian combination based upon posterior probabilities. The statistical circumstances reflected varying conditions of relative forecast errors, error correlations and outliers.     

The revision frequency of earnings forecasts and firm characteristics

This study investigates how the revision frequency of earnings forecasts affects firm characteristics. Previous studies generally focus on the number of analysts following a firm to measure a firm's information environment. The frequency with which news is updated is often defined as an analyst's effort. Analysts provide more information to investors if they update news more frequently. This study examines whether the frequency of information updating for a particular firm affects the firm's performance. We apply three proxies for firm performance: stock liquidity, the cost of equity capital, and firm value. Our findings indicate that the analysts’ effort as measured by the frequency of news updating is effective in providing additional power beyond the number of analysts to represent the information environment of a firm. Therefore, this study suggests that combining both the number of analysts following a firm and the frequency of news updating can be a better proxy for assessing a firm's information environment.     

DSGE forecasts of the lost recovery

The years following the Great Recession were challenging for forecasters. Unlike other deep downturns, this recession was not followed by a swift recovery, but instead generated a sizable and persistent output gap that was not accompanied by deflation as a traditional Phillips curve relationship would have predicted. Moreover, the zero lower bound and unconventional monetary policy generated an unprecedented policy environment. We document the actual real-time forecasting performance of the New York Fed dynamic stochastic general equilibrium (DSGE) model during this period and explain the results using the pseudo real-time forecasting performance results from a battery of DSGE models. We find the New York Fed DSGE model’s forecasting accuracy to be comparable to that of private forecasters, and notably better for output growth than the median forecasts from the FOMC’s Summary of Economic Projections. The model’s financial frictions were key in obtaining these results, as they implied a slow recovery following the financial crisis.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

A new method to assess the degree of information rigidity using fixed-event forecasts

We propose a new method to explore the information content of fixed-event forecasts and estimate structural parameters that are keys to sticky and noisy information models. Estimation follows a regression-based framework in which estimated coefficients map one-to-one with parameters that measure the degree of information rigidity. The statistical characterization of regression errors explores the laws that govern expectation formation under sticky and noisy information, that is, they are coherent with the theory. This strategy is still unexplored in the literature and potentially enhances the reliability of inference results. The method also allows linking estimation results to the signal-to-noise ratio, an important parameter of noisy information models. This task cannot be accomplished if one adopts an “agnostic” characterization of regression errors. With regard to empirical results, they show a substantial degree of information rigidity in the countries studied. They also suggest that the theoretical characterization of regression errors yields a more conservative picture of the uncertainty surrounding parameter estimates.     

The impact of incentives on the accuracy of subjects in judgmental forecasting experiments

There is an ongoing debate in the social sciences about whether or not financial incentives are needed in order to obtain good performance from experimental subjects. This debate often extends into the research on judgmental forecasting. Thus, an experiment was conducted to assess the effects of financial incentives on time series forecasting accuracy. There was no evidence that financial incentives impacted forecasting accuracy in stable time series. Financial incentives also had no impact immediately after instabilities occurred and no impact once the trend in the data had fully emerged.