The effect of nonstationarity on combined forecasts

Previous research on the combination of forecasts has, for the most part, implicitly assumed a stationary underlying process so that parameters could be estimated from historical data. While some models weight recent data more heavily in the estimation process in an attempt to provide more accurate parameter estimates in a nonstationary environment, no research to date has specifically examined the effects of nonstationarity on the performance of combining methods. This paper reports the results of a simulation study of the effects of nonstationarity (a shift in the process) on a range of forecast combination methods. Special attention is given to the relative performance of the methods in the time periods around the shift.     

Cross-temporal forecast reconciliation: Optimal combination method and heuristic alternatives

Forecast reconciliation is a post-forecasting process aimed to improve the quality of the base forecasts for a system of hierarchical/grouped time series. Cross-sectional and temporal hierarchies have been considered in the literature, but generally, these two features have not been fully considered together. The paper presents two new results by adopting a notation that simultaneously deals with both forecast reconciliation dimensions. (i) The closed-form expression of the optimal (in the least squares sense) point forecasts fulfilling both contemporaneous and temporal constraints. (ii) An iterative procedure that produces cross-temporally reconciled forecasts by alternating forecast reconciliation along one single dimension (either cross-sectional or temporal) at each iteration step. The feasibility of the proposed procedures, along with first evaluations of their performance as compared to the most performing ‘single dimension’ (either cross-sectional or temporal) forecast reconciliation procedures, is studied through a forecasting experiment on the 95 quarterly time series of the Australian Gross Domestic Product from Income and Expenditure sides. For this dataset, the new procedures, in addition to providing fully coherent forecasts in both cross-sectional and temporal dimensions, improve the forecast accuracy of the state-of-the-art point forecast reconciliation techniques.     

Forecast-error-based estimation of forecast uncertainty when the horizon is increased

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC.     

Combining exponential smoothing forecasts using Akaike weights

Simple forecast combinations such as medians and trimmed or winsorized means are known to improve the accuracy of point forecasts, and Akaike’s Information Criterion (AIC) has given rise to so-called Akaike weights, which have been used successfully to combine statistical models for inference and prediction in specialist fields, e.g., ecology and medicine. We examine combining exponential smoothing point and interval forecasts using weights derived from AIC, small-sample-corrected AIC and BIC on the M1 and M3 Competition datasets. Weighted forecast combinations perform better than forecasts selected using information criteria, in terms of both point forecast accuracy and prediction interval coverage. Simple combinations and weighted combinations do not consistently outperform one another, while simple combinations sometimes perform worse than single forecasts selected by information criteria. We find a tendency for a longer history to be associated with a better prediction interval coverage.     

Combining predictive distributions for the statistical post-processing of ensemble forecasts

Statistical post-processing techniques are now used widely for correcting systematic biases and errors in the calibration of ensemble forecasts obtained from multiple runs of numerical weather prediction models. A standard approach is the ensemble model output statistics (EMOS) method, which results in a predictive distribution that is given by a single parametric law, with parameters that depend on the ensemble members. This article assesses the merits of combining multiple EMOS models based on different parametric families. In four case studies with wind speed and precipitation forecasts from two ensemble prediction systems, we investigate the performances of state of the art forecast combination methods and propose a computationally efficient approach for determining linear pool combination weights. We study the performance of forecast combination compared to that of the theoretically superior but cumbersome estimation of a full mixture model, and assess which degree of flexibility of the forecast combination approach yields the best practical results for post-processing applications.     

Sectoral Shocks and Metropolitan Employment Growth

In this paper, we study the sources of industry employment growth in each of five metropolitan statistical areas (MSAs). The objective is to understand the relative importance of aggregate disturbances versus local sectoral shocks in generating observed employment fluctuations at the MSA level. The empirical evidence presented in this paper derives from structural vector autoregressions (SVARs), estimated for each of the five MSAs. Estimations use monthly employment data covering nine one-digit industrial categories for the period 1951:1–1999:8, as well as two variables that capture the influences of aggregate (i.e., national) shocks on MSAs. We find that within-MSA industry shocks explain considerably more of the forecast error variance in industry employment growth than do aggregate shocks. Sectoral shocks account for between 87 and 94% of the 36-month-ahead forecast error variance. Among individual local sectors, shocks to MSA-specific government, manufacturing, and service sector employment growth are the predominate sources of variability.     

Fast and accurate yearly time series forecasting with forecast combinations

It has long been known that combination forecasting strategies produce superior out-of-sample forecasting performances. In the M4 forecasting competition, a very simple forecast combination strategy achieved third place on yearly time series. An analysis of the ensemble model and its component models suggests that the competitive accuracy comes from avoiding poor forecasts, rather than from beating the best individual models. Moreover, the simple ensemble model can be fitted very quickly, can easily scale horizontally with additional CPU cores or a cluster of computers, and can be implemented by users very quickly and easily. This approach might be of particular interest to users who need accurate yearly forecasts without being able to spend significant time, resources, or expertise on tuning models. Users of the R statistical programming language can access this modeling approach using the “forecastHybrid” package.     

Forecasting macroeconomic time series: LASSO-based approaches and their forecast combinations with dynamic factor models

In a data-rich environment, forecasting economic variables amounts to extracting and organizing useful information from a large number of predictors. So far, the dynamic factor model and its variants have been the most successful models for such exercises. In this paper, we investigate a category of LASSO-based approaches and evaluate their predictive abilities for forecasting twenty important macroeconomic variables. These alternative models can handle hundreds of data series simultaneously, and extract useful information for forecasting. We also show, both analytically and empirically, that combing forecasts from LASSO-based models with those from dynamic factor models can reduce the mean square forecast error (MSFE) further. Our three main findings can be summarized as follows. First, for most of the variables under investigation, all of the LASSO-based models outperform dynamic factor models in the out-of-sample forecast evaluations. Second, by extracting information and formulating predictors at economically meaningful block levels, the new methods greatly enhance the interpretability of the models. Third, once forecasts from a LASSO-based approach are combined with those from a dynamic factor model by forecast combination techniques, the combined forecasts are significantly better than either dynamic factor model forecasts or the naïve random walk benchmark.     

Forecasting monthly and quarterly time series using STL decomposition

This paper is a re-examination of the benefits and limitations of decomposition and combination techniques in the area of forecasting, and also a contribution to the field, offering a new forecasting method. The new method is based on the disaggregation of time series components through the STL decomposition procedure, the extrapolation of linear combinations of the disaggregated sub-series, and the reaggregation of the extrapolations to obtain estimates for the global series. Applying the forecasting method to data from the NN3 and M1 Competition series, the results suggest that it can perform well relative to four other standard statistical techniques from the literature, namely the ARIMA, Theta, Holt-Winters’ and Holt’s Damped Trend methods. The relative advantages of the new method are then investigated further relative to a simple combination of the four statistical methods and a Classical Decomposition forecasting method. The strength of the method lies in its ability to predict long lead times with relatively high levels of accuracy, and to perform consistently well for a wide range of time series, irrespective of the characteristics, underlying structure and level of noise of the data.     

Modeling the accuracy of certain regional predictions

A model is hypothesized specifying forecast error as a function of specific ‘year effects’, particular dates of forecast, and ‘time span effects’, length of projection horizon. Model estimation methodology is presented and empirical application is made to inaccuracies arising in the RAS procedure for projecting bivariate region/industry employment arrays for two sets of regional data. In both applications, the model is found to explain a large proportion of variation in forecast errors, and estimation results permit isolation of pure time effects on deterioration of model accuracy and quantification of relative ‘year effects’.     

Linking words in economic discourse: Implications for macroeconomic forecasts

This paper develops indicators of unstructured press information by exploiting word vector representations. A model is trained using a corpus covering 90 years of Wall Street Journal content. The information content of the indicators is assessed through business cycle forecast exercises. The vector representations can learn meaningful word associations that are exploited to construct indicators of uncertainty. In-sample and out-of-sample forecast exercises show that the indicators contain valuable information regarding future economic activity. The combination of indices associated with different subjective states (e.g., uncertainty, fear, pessimism) results in further gains in information content. The documented performance is unmatched by previous dictionary-based word counting techniques proposed in the literature.     

Forecast encompassing tests for the expected shortfall

We introduce new forecast encompassing tests for the risk measure Expected Shortfall (ES). The ES has received much attention since its introduction into the Basel III Accords, which stipulate its use as the primary market risk measure for international banking regulation. We utilize joint loss functions for the pair ES and Value at Risk to set up three ES encompassing test variants. The tests are built on an asymptotic theory that is robust to misspecifications. We investigate the finite sample properties of the tests in an extensive simulation study. Finally, we use the encompassing tests to illustrate the potential of forecast combination methods for different financial assets.     

Efficient estimation of forecast uncertainty based on recent forecast errors

Multi-step-ahead forecasts of the forecast uncertainty of an individual forecaster are often based on the horizon-specific sample means of his recent squared forecast errors, where the number of past forecast errors available decreases one-to-one with the forecast horizon. In this paper, the efficiency gains from the joint estimation of forecast uncertainty for all horizons in such samples are investigated. If the forecast uncertainty is estimated by seemingly unrelated regressions, it turns out that the covariance matrix of the squared forecast errors does not have to be estimated, but simply needs to have a certain structure, which is a very useful property in small samples. Considering optimal and non-optimal forecasts, it is found that the efficiency gains can be substantial for longer horizons in small samples. The superior performance of the seemingly-unrelated-regressions approach is confirmed in several empirical applications.     

Combining probability forecasts

We consider different methods for combining probability forecasts. In empirical exercises, the data generating process of the forecasts and the event being forecast is not known, and therefore the optimal form of combination will also be unknown. We consider the properties of various combination schemes for a number of plausible data generating processes, and indicate which types of combinations are likely to be useful. We also show that whether forecast encompassing is found to hold between two rival sets of forecasts or not may depend on the type of combination adopted. The relative performances of the different combination methods are illustrated, with an application to predicting recession probabilities using leading indicators.     

Explaining variance in the accuracy of prediction markets

Thus far, the focus in prediction market research has been on establishing its forecast accuracy relative to those of other prediction methods, or on the investigation of a few single sources of forecast error. This article is the first attempt to overcome the narrow focus of the literature by combining observational and experimental analyses of prediction market errors. It investigates the prediction error of a real money prediction market uusing a logarithmic market scoring rule for 65 direct democratic votes in Switzerland. The article distinguishes between prediction market error due to the setup of the market, features of the event to be predicted, and the participants involved, and finds that the prediction market accuracy varies primarily according to the setup of the market, with the features of the event and especially the composition of the participant sample hardly mattering.     

Embrace the differences: Revisiting the PollyVote method of combining forecasts for U.S. presidential elections (2004 to 2020)

While combining forecasts is well-known to reduce error, the question of how to best combine forecasts remains. Prior research suggests that combining is most beneficial when relying on diverse forecasts that incorporate different information. Here, I provide evidence in support of this hypothesis by analyzing data from the PollyVote project, which has published combined forecasts of the popular vote in U.S. presidential elections since 2004. Prior to the 2020 election, the PollyVote revised its original method of combining forecasts by, first, restructuring individual forecasts based on their underlying information and, second, adding naïve forecasts as a new component method. On average across the last 100 days prior to the five elections from 2004 to 2020, the revised PollyVote reduced the error of the original specification by eight percent and, with a mean absolute error (MAE) of 0.8 percentage points, was more accurate than any of its component forecasts. The results suggest that, when deciding about which forecasts to include in the combination, forecasters should be more concerned about the component forecasts’ diversity than their historical accuracy.     

Testing the independence of forecast errors in the forward foreign exchange market using Markov chains: A cross-country comparison

In this paper, we apply the theory of finite state Markov chains to test the cross-country and temporal independence of forecast errors in the forward foreign exchange market. Specifically, we consider the month-end thirty day foreign exchange data for Canada, France, Italy, Japan, United Kingdom and West Germany for the period 1974–1981. Using pairwise comparisons for the various countries, we find that except for Canada, the hypothesis that the probability distribution of the forecast error of one country is independent of the forecast error of another is rejected. Further tests indicate that the temporal independence for most countries is also rejected. Based upon these results, we conclude that for these six countries, there is current information available which is ‘useful’ in predicting the future forward exchange forecast errors.     

Properties of optimal forecasts under asymmetric loss and nonlinearity

Evaluation of forecast optimality in economics and finance has almost exclusively been conducted under the assumption of mean squared error loss. Under this loss function optimal forecasts should be unbiased and forecast errors serially uncorrelated at the single period horizon with increasing variance as the forecast horizon grows. Using analytical results we show that standard properties of optimal forecasts can be invalid under asymmetric loss and nonlinear data generating processes and thus may be very misleading as a benchmark for an optimal forecast. We establish instead that a suitable transformation of the forecast error—known as the generalized forecast error—possesses an equivalent set of properties. The paper also provides empirical examples to illustrate the significance in practice of asymmetric loss and nonlinearities and discusses the effect of parameter estimation error on optimal forecasts.     

Forecast combination through dimension reduction techniques

This paper considers several methods of producing a single forecast from several individual ones. We compare “standard” but hard to beat combination schemes (such as the average of forecasts at each period, or consensus forecast and OLS-based combination schemes) with more sophisticated alternatives that involve dimension reduction techniques. Specifically, we consider principal components, dynamic factor models, partial least squares and sliced inverse regression.Our source of forecasts is the Survey of Professional Forecasters, which provides forecasts for the main US macroeconomic aggregates. The forecasting results show that partial least squares, principal component regression and factor analysis have similar performances (better than the usual benchmark models), but sliced inverse regression shows an extreme behavior (performs either very well or very poorly).     

Time Series Based Errors and Empirical Errors in Fertility Forecasts in the Nordic Countries*

We use ARCH time series models to derive model based prediction intervals for the Total Fertility Rate (TFR) in Norway, Sweden, Finland, and Denmark up to 2050. For the short term (5–10 yrs), expected TFR‐errors are compared with empirical forecast errors observed in historical population forecasts prepared by the statistical agencies in these countries since 1969. Medium‐term and long‐term (up to 50 years) errors are compared with error patterns based on so‐called naïve forecasts, i.e. forecasts that assume that recently observed TFR‐levels also apply for the future.