Accuracy of German federal election forecasts, 2013 & 2017

The present study reviews the accuracy of four methods (polls, prediction markets, expert judgment, and quantitative models) for forecasting the two German federal elections in 2013 and 2017. On average across both elections, polls and prediction markets were most accurate, while experts and quantitative models were least accurate. However, the accuracy of individual forecasts did not correlate across elections. That is, the methods that were most accurate in 2013 did not perform particularly well in 2017. A combined forecast, calculated by averaging forecasts within and across methods, was more accurate than three of the four component forecasts. The results conform to prior research on US presidential elections in showing that combining is effective in generating accurate forecasts and avoiding large errors.     

How accurate are government forecasts of economic fundamentals? The case of Taiwan

A government’s ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.In this paper, we develop a methodology for evaluating non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that the undocumented knowledge of the Taiwanese government reduces forecast errors substantially.     

Evaluating quantile forecasts in the M5 uncertainty competition

Probabilistic forecasts are necessary for robust decisions in the face of uncertainty. The M5 Uncertainty competition required participating teams to forecast nine quantiles for unit sales of various products at various aggregation levels and for different time horizons. This paper evaluates the forecasting performance of the quantile forecasts at different aggregation levels and at different quantile levels. We contrast this with some theoretical predictions, and discuss potential implications and promising future research directions for the practice of probabilistic forecasting.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

Forecasting and forecast narratives: The Bank of England Inflation Reports

We analyze the narratives that accompany the numerical forecasts in the Bank of England’s Quarterly Inflation Reports, 1997–2018. We focus on whether the narratives contain useful information about the future course of key macro variables over and above the point predictions, in terms of whether the narratives can be used to enhance the accuracy of the numerical forecasts. We also consider whether the narratives are able to predict future changes in the numerical forecasts. We find that a measure of sentiment derived from the narratives can predict the errors in the numerical forecasts of output growth, but not of inflation. We find no evidence that past changes in sentiment predict subsequent changes in the point forecasts of output growth or of inflation, but do find that the adjustments to the numerical output growth forecasts have a systematic element.     

Combining exponential smoothing forecasts using Akaike weights

Simple forecast combinations such as medians and trimmed or winsorized means are known to improve the accuracy of point forecasts, and Akaike’s Information Criterion (AIC) has given rise to so-called Akaike weights, which have been used successfully to combine statistical models for inference and prediction in specialist fields, e.g., ecology and medicine. We examine combining exponential smoothing point and interval forecasts using weights derived from AIC, small-sample-corrected AIC and BIC on the M1 and M3 Competition datasets. Weighted forecast combinations perform better than forecasts selected using information criteria, in terms of both point forecast accuracy and prediction interval coverage. Simple combinations and weighted combinations do not consistently outperform one another, while simple combinations sometimes perform worse than single forecasts selected by information criteria. We find a tendency for a longer history to be associated with a better prediction interval coverage.     

Do survey joiners and leavers differ from regular participants? The US SPF GDP growth and inflation forecasts

If ‘learning by doing’ is important for macro-forecasting, newcomers might be different from regular, established participants. Stayers may also differ from the soon-to-leave. We test these conjectures for macro-forecasters’ point predictions of output growth and inflation, and for their histogram forecasts. Histogram forecasts of inflation by both joiners and leavers are found to be less accurate, especially if we suppose that joiners take time to learn. For GDP growth, there is no evidence of differences between the groups in terms of histogram forecast accuracy, although GDP point forecasts by leavers are less accurate. These findings are predicated on forecasters being homogeneous within groups. Allowing for individual fixed effects suggests fewer differences, including leavers’ inflation histogram forecasts being no less accurate.     

Combining multiple probability predictions using a simple logit model

This paper begins by presenting a simple model of the way in which experts estimate probabilities. The model is then used to construct a likelihood-based aggregation formula for combining multiple probability forecasts. The resulting aggregator has a simple analytical form that depends on a single, easily-interpretable parameter. This makes it computationally simple, attractive for further development, and robust against overfitting. Based on a large-scale dataset in which over 1300 experts tried to predict 69 geopolitical events, our aggregator is found to be superior to several widely-used aggregation algorithms.     

Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)

While combining forecasts is well-known to reduce error, the question of how to best combine forecasts remains. Prior research suggests that combining is most beneficial when relying on diverse forecasts that incorporate different information. Here, I provide evidence in support of this hypothesis by analyzing data from the PollyVote project, which has published combined forecasts of the popular vote in U.S. presidential elections since 2004. Prior to the 2020 election, the PollyVote revised its original method of combining forecasts by, first, restructuring individual forecasts based on their underlying information and, second, adding naïve forecasts as a new component method. On average across the last 100 days prior to the five elections from 2004 to 2020, the revised PollyVote reduced the error of the original specification by eight percent and, with a mean absolute error (MAE) of 0.8 percentage points, was more accurate than any of its component forecasts. The results suggest that, when deciding about which forecasts to include in the combination, forecasters should be more concerned about the component forecasts’ diversity than their historical accuracy.     

Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7

In this paper, we use survey data to analyze the accuracy, unbiasedness and efficiency of professional macroeconomic forecasts. We analyze a large panel of individual forecasts that has not previously been analyzed in the literature. We provide evidence on the properties of forecasts for all G7-countries and for four different macroeconomic variables. Our results show a high degree of dispersion of forecast accuracy across forecasters. We also find that there are large differences in the performances of forecasters, not only across countries but also across different macroeconomic variables. In general, the forecasts tend to be biased in situations where the forecasters have to learn about large structural shocks or gradual changes in the trend of a variable. Furthermore, while a sizable fraction of forecasters seem to smooth their GDP forecasts significantly, this does not apply to forecasts made for other macroeconomic variables.     

Forecasting small area populations with long short-term memory networks

Local and state governments depend on small area population forecasts to make important decisions concerning the development of local infrastructure and services. Despite their importance, current methods often produce highly inaccurate forecasts. Recent years have witnessed promising developments in time series forecasting using Machine Learning across a wide range of social and economic variables. However, limited work has been undertaken to investigate the potential application of Machine Learning methods in demography, particularly for small area population forecasting. In this paper we describe the development of two Long-Short Term Memory network architectures for small area populations. We employ the Keras Tuner to select layer unit numbers, vary the window width of input data, and apply a double training and validation regime which supports work with short time series and prioritises later sequence values for forecasts. These methods are transferable and can be applied to other data sets. Retrospective small area population forecasts for Australia were created for the periods 2006–16 and 2011–16. Model performance was evaluated against actual data and two benchmark methods (LIN/EXP and CSP-VSG). We also evaluated the impact of constraining small area population forecasts to an independent national forecast. Forecast accuracy was influenced by jump-off year, constraining, area size, and remoteness. The LIN/EXP model was the best performing method for the 2011-based forecasts whilst deep learning methods performed best for the 2006-based forecasts, including significant improvements in the accuracy of 10 year forecasts. However, benchmark methods were consistently more accurate for more remote areas and for those with populations <5000.     

Combining survey long-run forecasts and nowcasts with BVAR forecasts using relative entropy

This paper constructs hybrid forecasts that combine forecasts from vector autoregressive (VAR) model(s) with both short- and long-term expectations from surveys. Specifically, we use the relative entropy to tilt one-step-ahead and long-horizon VAR forecasts to match the nowcasts and long-horizon forecasts from the Survey of Professional Forecasters. We consider a variety of VAR models, ranging from simple fixed-parameter to time-varying parameters. The results across models indicate meaningful gains in multi-horizon forecast accuracy relative to model forecasts that do not incorporate long-term survey conditions. Accuracy improvements are achieved for a range of variables, including those that are not tilted directly but are affected through spillover effects from tilted variables. The accuracy gains for hybrid inflation forecasts from simple VARs are substantial, statistically significant, and competitive to time-varying VARs, univariate benchmarks, and survey forecasts. We view our proposal as an indirect approach to accommodating structural change and moving end points.     

Forecast revisions of Mexican inflation and GDP growth

We analyze the forecasts of inflation and GDP growth contained in the Banco de México’s Survey of Professional Forecasters for the period 1995–2009. The forecasts are for the current and the following year, and comprise an unbalanced three-dimensional panel with multiple individual forecasters, target years, and forecast horizons. The fixed-event nature of the forecasts enables us to examine their efficiency by looking at the revision process. The panel structure allows us to control for aggregate shocks and to construct a measure of the news that impacted expectations in the period under study. We find that respondents anchor to their initial forecasts, updating their revisions smoothly as they receive more information. In addition, they do not seem to use publicly-known information in an efficient manner. These inefficiencies suggest clear areas of opportunity for improving the accuracy of the forecasts, for instance by taking into account the positive autocorrelation found in forecast revisions.     

Influence of differentiated roles on group forecasting accuracy

While behavioral research on forecasting has mostly examined the individual forecaster, organizationally-based forecasting processes typically tend to rely on groups with members from different functional areas for arriving at ‘consensus’ forecasts. The forecasting performance could also vary depending on the particular group structuring utilized in reaching a final prediction. The current study compares the forecasting performance of modified consensus groups with that of staticized groups using formal role-playing. It is found that, when undistorted model forecasts are given, group forecasts (whether they are arrived at through averaging or by a detailed discussion of the forecasts) contribute positively to the forecasting accuracy. However, providing distorted initial forecasts affects the final accuracy with varying degrees of improvement over the initial forecasts. The results show a strong tendency to favor optimistic forecasts for both the staticized and modified consensus group forecasts. Overall, the role modifications are found to be successful in eliciting a differential adjustment behavior, effectively mimicking the disparities between different organizational roles. Current research suggests that group discussions may be an efficient method of displaying and resolving differential motivational contingencies, potentially leading to group forecasts that perform quite well.     

Medium term growth forecasts: Experts vs. simple models

We compare the medium-term GDP growth forecasts generated by experts to those generated by simple models. This study analyzes a large set of forecasts that covers 48 countries from 1997 to 2016. Out-of-sample exercises indicate that no noticeable difference in performance is observed for advanced economies. In contrast, in the case of emerging economies, model forecasts perform better than expert forecasts. In addition, similar patterns are found for a collection of forecasts from a different set of experts, which suggests that the reported regularity is prevalent. Further analyses suggest that the documented difference in performance can be explained by an optimism bias, excessive reactions to innovations in growth trajectories, and insufficient responses to the information contained in the current account balance.     

Modeling college major choices using elicited measures of expectations and counterfactuals

The choice of a college major plays a critical role in determining the future earnings of college graduates. Students make their college major decisions in part due to the future earnings streams associated with the different majors. We survey students about what their expected earnings would be both in the major they have chosen and in counterfactual majors. We also elicit students’ subjective assessments of their abilities in chosen and counterfactual majors. We estimate a model of college major choice that incorporates these subjective expectations and assessments. We show that both expected earnings and students’ abilities in the different majors are important determinants of a student’s choice of a college major. We also consider how differences in students’ forecasts about what the average Duke student would earn in different majors versus what they expect they would earn both influence one’s choice of a college major. In particular, our estimates suggest that 7.8% of students would switch majors if they had the same expectations about the average returns to different majors and differed only in their perceived comparative advantages across these majors.     

Forecasting monthly and quarterly time series using STL decomposition

This paper is a re-examination of the benefits and limitations of decomposition and combination techniques in the area of forecasting, and also a contribution to the field, offering a new forecasting method. The new method is based on the disaggregation of time series components through the STL decomposition procedure, the extrapolation of linear combinations of the disaggregated sub-series, and the reaggregation of the extrapolations to obtain estimates for the global series. Applying the forecasting method to data from the NN3 and M1 Competition series, the results suggest that it can perform well relative to four other standard statistical techniques from the literature, namely the ARIMA, Theta, Holt-Winters’ and Holt’s Damped Trend methods. The relative advantages of the new method are then investigated further relative to a simple combination of the four statistical methods and a Classical Decomposition forecasting method. The strength of the method lies in its ability to predict long lead times with relatively high levels of accuracy, and to perform consistently well for a wide range of time series, irrespective of the characteristics, underlying structure and level of noise of the data.     

How Will the Court Decide? – Tax Experts’ versus Laymen's Predictions

We conduct a survey of German tax professionals (tax advisors and revenue agents) and laymen to examine whether tax experts more accurately forecast the outcomes of five real cases from the German Federal Fiscal Court. With an average of 2.39 correct predictions among experts and an average of 2.49 correct predictions among laymen, our results reveal no significant difference in forecasting accuracy between the two groups. Additionally, neither general nor task-specific tax expertise increases the experts’ forecasting accuracy. This unpredictability of tax court decisions indicates that accounting rules and taxpayer penalties that rely on accurate predictions of tax court decisions may need to be re-evaluated. Moreover, our results indicate the existence of two types of ‘advisor bias’. First, tax advisors exhibit a significantly higher level of overconfidence in comparison to other experts (i.e. revenue agents) and laymen. In particular, they believe that they correctly predict, on average, 1.52 more cases than they actually do. Second, we find some evidence indicating that tax advisors acting as client advocates form stronger appeal recommendations than revenue agents.     

Inflation forecasts and core inflation measures: Where is the information on future inflation?

This paper uses the forecast from a random walk model of inflation as a benchmark to test and compare the forecast performance of several alternatives of future inflation, including the Greenbook forecast by the Fed staff, the Survey of Professional Forecasters median forecast, CPI inflation minus food and energy, CPI weighted median inflation, and CPI trimmed mean inflation. The Greenbook forecast was found in previous literature to be a better forecast than other private sector forecasts. Our results indicate that both the Greenbook and the Survey of Professional Forecasters median forecasts of inflation and core inflation measures may contain better information than forecasts from a random walk model. The Greenbook's superiority appears to have declined against other forecasts and core inflation measures.     

Validity of climate change forecasting for public policy decision making

Policymakers need to know whether prediction is possible and, if so, whether any proposed forecasting method will provide forecasts that are substantially more accurate than those from the relevant benchmark method. An inspection of global temperature data suggests that temperature is subject to irregular variations on all relevant time scales, and that variations during the late 1900s were not unusual. In such a situation, a “no change” extrapolation is an appropriate benchmark forecasting method. We used the UK Met Office Hadley Centre’s annual average thermometer data from 1850 through 2007 to examine the performance of the benchmark method. The accuracy of forecasts from the benchmark is such that even perfect forecasts would be unlikely to help policymakers. For example, mean absolute errors for the 20- and 50-year horizons were 0.18  C and 0.24  C respectively. We nevertheless demonstrate the use of benchmarking with the example of the Intergovernmental Panel on Climate Change’s 1992 linear projection of long-term warming at a rate of 0.03  C per year. The small sample of errors from ex ante projections at 0.03  C per year for 1992 through 2008 was practically indistinguishable from the benchmark errors. Validation for long-term forecasting, however, requires a much longer horizon. Again using the IPCC warming rate for our demonstration, we projected the rate successively over a period analogous to that envisaged in their scenario of exponential CO₂ growth—the years 1851 to 1975. The errors from the projections were more than seven times greater than the errors from the benchmark method. Relative errors were larger for longer forecast horizons. Our validation exercise illustrates the importance of determining whether it is possible to obtain forecasts that are more useful than those from a simple benchmark before making expensive policy decisions.