Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives

Despite the clear success of forecast combination in many economic environments, several important issues remain incompletely resolved. The issues relate to the selection of the set of forecasts to combine, and whether some form of additional regularization (e.g., shrinkage) is desirable. Against this background, and also considering the frequently-found good performance of simple-average combinations, we propose a LASSO-based procedure that sets some combining weights to zero and shrinks the survivors toward equality (“partially-egalitarian LASSO”). Ex post analysis reveals that the optimal solution has a very simple form: the vast majority of forecasters should be discarded, and the remainder should be averaged. We therefore propose and explore direct subset-averaging procedures that are motivated by the structure of partially-egalitarian LASSO and the lessons learned, which, unlike LASSO, do not require the choice of a tuning parameter. Intriguingly, in an application to the European Central Bank Survey of Professional Forecasters, our procedures outperform simple average and median forecasts; indeed, they perform approximately as well as the ex post best forecaster.     

Takeover prediction using forecast combinations

The ability to identify likely takeover targets at an early stage should provide investors with valuable information, enabling them to profit by investing in potential target firms. In this paper we contribute to the takeover forecasting literature by suggesting the combination of probability forecasts as an alternative method of improving the forecast accuracy in takeover prediction and realizing improved economic returns from portfolios made up of predicted targets. Forecasts from several non-linear forecasting models, such as logistic and neural network models and a combination of them, are used to determine the methodology that best reduces the out-of-sample misclassification error. We draw two general conclusions from our results. First, the forecast combination method outperforms the single models, and should therefore be used to improve the accuracy of takeover target predictions. Second, we demonstrate that an investment in a portfolio of the combined predicted targets results in significant abnormal returns being made by an investor, in the order of up to double the market benchmark return when using a portfolio of manageable size.     

Hard to arbitrage,hard for analysts to forecast

This paper examines whether limits to arbitrage (LA) affect analysts' earnings forecast accuracy. Using the LA index, which is constructed from unique trading constraints in the Chinese stock market and other commonly used measures, we find that forecast accuracy is much lower for stocks with high LA. Moreover, our results are more suited to explanations of cognitive bias that turn to investor sentiment or limited attention and cannot be fully explained by more objective factors, including analyst ability, broker size, broker experience, and commission pressure. We also find that LA amplifies analyst forecast dispersion. Such results indicate that LA distorts analysts’ earnings expectations and provides new insight into how LA affects anomaly returns.     

Bayesian forecast combination using time-varying features

In this work, we propose a novel framework for density forecast combination by constructing time-varying weights based on time-varying features. Our framework estimates weights in the forecast combination via Bayesian log predictive scores, in which the optimal forecast combination is determined by time series features from historical information. In particular, we use an automatic Bayesian variable selection method to identify the importance of different features. To this end, our approach has better interpretability compared to other black-box forecasting combination schemes. We apply our framework to stock market data and M3 competition data. Based on our structure, a simple maximum-a-posteriori scheme outperforms benchmark methods, and Bayesian variable selection can further enhance the accuracy for both point forecasts and density forecasts.     

The comparative forecast performance of univariate and multivariate models: an application to real interest rate forecasting

Does the use of information on the past history of the nominal interest rates and inflation entail improvement in forecasts of the ex ante real interest rate over its forecasts obtained from using just the past history of the realized real interest rates? To answer this question we set up a univariate unobserved components model for the realized real interest rates and a bivariate model for the nominal rate and inflation which imposes cointegration restrictions between them. The two models are estimated under normality with the Kalman filter. It is found that the error-correction model provides more accurate one-period ahead forecasts of the real rate within the estimation sample whereas the unobserved components model yields forecasts with smaller forecast variances. In the post-sample period, the forecasts from the bivariate model are not only more accurate but also have tighter confidence bounds than the forecasts from the unobserved components model.     

The impact of institutional change on forecast accuracy: A case study of budget forecasting in Washington State

This paper explores the relationship between institutional change and forecast accuracy via an analysis of the entitlement caseload forecasting process in Washington State. This research extends the politics of forecasting literature beyond the current area of government revenue forecasting to include expenditure forecasting and introduces an in-depth longitudinal study to the existing set of cross-sectional studies. Employing a fixed-effects model and ordinary least squares regression analysis, this paper concludes that the establishment of an independent forecasting agency and subsequent formation of technical workgroups improve forecast accuracy. Additionally, this study finds that more frequent forecast revisions and structured domain knowledge improve forecast accuracy.     

Efficient estimation of forecast uncertainty based on recent forecast errors

Multi-step-ahead forecasts of the forecast uncertainty of an individual forecaster are often based on the horizon-specific sample means of his recent squared forecast errors, where the number of past forecast errors available decreases one-to-one with the forecast horizon. In this paper, the efficiency gains from the joint estimation of forecast uncertainty for all horizons in such samples are investigated. If the forecast uncertainty is estimated by seemingly unrelated regressions, it turns out that the covariance matrix of the squared forecast errors does not have to be estimated, but simply needs to have a certain structure, which is a very useful property in small samples. Considering optimal and non-optimal forecasts, it is found that the efficiency gains can be substantial for longer horizons in small samples. The superior performance of the seemingly-unrelated-regressions approach is confirmed in several empirical applications.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

Does age structure forecast economic growth?

Increases in the proportion of the working age population can yield a “demographic dividend” that enhances the rate of economic growth. We estimate the parameters of an economic growth model using a cross section of countries over the period 1960 to 1980, and investigate whether the inclusion of age structure improves the model's forecasts for the period 1980 to 2000. We find that including the age structure improves the forecast, although there is evidence of parameter instability between periods with an unexplained growth slowdown in the second period. We use the model to generate growth forecasts for the period 2000 to 2020.     

Some theoretical results on forecast combinations

This paper proposes a framework for the analysis of the theoretical properties of forecast combination, with the forecast performance being measured in terms of mean squared forecast errors (MSFE). Such a framework is useful for deriving all existing results with ease. In addition, it also provides insights into two forecast combination puzzles. Specifically, it investigates why a simple average of forecasts often outperforms forecasts from single models in terms of MSFEs, and why a more complicated weighting scheme does not always perform better than a simple average. In addition, this paper presents two new findings that are particularly relevant in practice. First, the MSFE of a forecast combination decreases as the number of models increases. Second, the conventional approach to the selection of optimal models, based on a simple comparison of MSFEs without further statistical testing, leads to a biased selection.     

A forecast evaluation of capital investment in agriculture

A stochastic coefficients model developed by Swamy and Tinsley is used to forecast agricultural investment. In two sets of out-of-sample forecasts, one for 5 years, the other for 10 years, the Swamy-Tinsley stochastic coefficients model outperforms competing fixed and stochastic coefficients empirical models of agricultural investment for a wide array of risk functions. The Swamy-Tinsley stochastic coefficients investment model forecasts continued declines in net investment for farm machinery, with greater declines toward the end of the forecast period. The Swamy-Tinsley method produced better predictions than both stochastic and fixed-coefficients competitors.     

Macroeconomic forecasting using structural factor analysis

The use of a small number of underlying factors to summarize the information from a much larger set of information variables is one of the new frontiers in forecasting. In prior work, the estimated factors have not usually had a structural interpretation and the factors have not been chosen on a theoretical basis. In this paper we propose several variants of a general structural factor forecasting model, and use these to forecast certain key macroeconomic variables. We make the choice of factors more structurally meaningful by estimating factors from subsets of information variables, where these variables can be assigned to subsets on the basis of economic theory. We compare the forecasting performance of the structural factor forecasting model with that of a univariate AR model, a standard VAR model, and some non-structural factor forecasting models. The results suggest that our structural factor forecasting model performs significantly better in forecasting real activity variables, especially at short horizons.     

Forecast-error-based estimation of forecast uncertainty when the horizon is increased

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC.     

FFORMA: Feature-based forecast model averaging

We propose an automated method for obtaining weighted forecast combinations using time series features. The proposed approach involves two phases. First, we use a collection of time series to train a meta-model for assigning weights to various possible forecasting methods with the goal of minimizing the average forecasting loss obtained from a weighted forecast combination. The inputs to the meta-model are features that are extracted from each series. Then, in the second phase, we forecast new series using a weighted forecast combination, where the weights are obtained from our previously trained meta-model. Our method outperforms a simple forecast combination, as well as all of the most popular individual methods in the time series forecasting literature. The approach achieved second position in the M4 competition.     

Asymmetry in unemployment rate forecast errors

Asymmetries in unemployment dynamics have been observed in the time series of a number of countries, including the United States. This paper studies asymmetries in unemployment rate forecast errors. We consider conditions under which optimal forecasts will display asymmetrically-distributed errors and how the degree of asymmetry might vary with the forecast horizon. Using data from the U.S. Survey of Professional Forecasters and the Federal Reserve Greenbook, we find substantial evidence of forecast error asymmetry, which tends to increase with the forecast horizon; we also find noteworthy differences in forecasts from these two sources. The results give insight into the abilities of professional forecasters to adapt their forecasts to asymmetry in underlying processes.     

Assessing the uncertainty in central banks’ inflation outlooks

Recent research has found that macroeconomic survey forecasts of uncertainty exhibit several deficiencies, such as horizon-dependent biases and lower levels of accuracy than simple unconditional uncertainty forecasts. We examine the inflation uncertainty forecasts from the Bank of England, the Banco Central do Brasil, the Magyar Nemzeti Bank and the Sveriges Riksbank to assess whether central banks’ uncertainty forecasts might be subject to similar problems. We find that, while most central banks’ uncertainty forecasts also tend to be underconfident at short horizons and overconfident at longer horizons, they are mostly not significantly biased. Moreover, they tend to be at least as precise as unconditional uncertainty forecasts from two different approaches.     

Robust optimization of forecast combinations

We develop a methodology for constructing robust combinations of time series forecast models which improve upon a given benchmark specification for all symmetric and convex loss functions. Under standard regularity conditions, the optimal forecast combination asymptotically almost surely dominates the benchmark, and also optimizes the chosen goal function. The optimum in a given sample can be found by solving a convex optimization problem. An application to the forecasting of changes in the S&P 500 volatility index shows that robust optimized combinations improve significantly upon the out-of-sample forecasting accuracy of both simple averaging and unrestricted optimization.     

Kaggle forecasting competitions: An overlooked learning opportunity

We review the results of six forecasting competitions based on the online data science platform Kaggle, which have been largely overlooked by the forecasting community. In contrast to the M competitions, the competitions reviewed in this study feature daily and weekly time series with exogenous variables, business hierarchy information, or both. Furthermore, the Kaggle data sets all exhibit higher entropy than the M3 and M4 competitions, and they are intermittent.In this review, we confirm the conclusion of the M4 competition that ensemble models using cross-learning tend to outperform local time series models and that gradient boosted decision trees and neural networks are strong forecast methods. Moreover, we present insights regarding the use of external information and validation strategies, and discuss the impacts of data characteristics on the choice of statistics or machine learning methods. Based on these insights, we construct nine ex-ante hypotheses for the outcome of the M5 competition to allow empirical validation of our findings.     

Simple averaging of direct and recursive forecasts via partial pooling using machine learning

This article introduces the winning method at the M5 Accuracy competition. The presented method takes a simple manner of averaging the results of multiple base forecasting models that have been constructed via partial pooling of multi-level data. All base forecasting models of adopting direct or recursive multi-step forecasting methods are trained by the machine learning technique, LightGBM, from three different levels of data pools. At the competition, the simple averaging of the multiple direct and recursive forecasting models, called DRFAM, obtained the complementary effects between direct and recursive multi-step forecasting of the multi-level product sales to improve the accuracy and the robustness.     

Assessing the accuracy of electricity production forecasts in developing countries

This study assesses the accuracy of time series econometric methods for forecasting electricity production in developing countries. An analysis of the historical time series for 106 developing countries over the period 1960–2012 demonstrates that econometric forecasts are highly accurate for the majority of these countries. These forecasts have much smaller errors than the predictions of simple heuristic models, which assume that electricity production grows at an exogenous rate or is proportional to the real GDP growth. However, the quality of the forecasts diminishes for the countries and regions, where rapid economic and structural transformation makes it difficult to establish stable historical production trends.