Bayesian forecast combination using time-varying features

In this work, we propose a novel framework for density forecast combination by constructing time-varying weights based on time-varying features. Our framework estimates weights in the forecast combination via Bayesian log predictive scores, in which the optimal forecast combination is determined by time series features from historical information. In particular, we use an automatic Bayesian variable selection method to identify the importance of different features. To this end, our approach has better interpretability compared to other black-box forecasting combination schemes. We apply our framework to stock market data and M3 competition data. Based on our structure, a simple maximum-a-posteriori scheme outperforms benchmark methods, and Bayesian variable selection can further enhance the accuracy for both point forecasts and density forecasts.     

Hierarchical forecasting with a top-down alignment of independent-level forecasts

Hierarchical forecasting with intermittent time series is a challenge in both research and empirical studies. Extensive research focuses on improving the accuracy of each hierarchy, especially the intermittent time series at bottom levels. Then, hierarchical reconciliation can be used to improve the overall performance further. In this paper, we present a hierarchical-forecasting-with-alignment approach that treats the bottom-level forecasts as mutable to ensure higher forecasting accuracy on the upper levels of the hierarchy. We employ a pure deep learning forecasting approach, N-BEATS, for continuous time series at the top levels, and a widely used tree-based algorithm, LightGBM, for intermittent time series at the bottom level. The hierarchical-forecasting-with-alignment approach is a simple yet effective variant of the bottom-up method, accounting for biases that are difficult to observe at the bottom level. It allows suboptimal forecasts at the lower level to retain a higher overall performance. The approach in this empirical study was developed by the first author during the M5 Accuracy competition, ranking second place. The method is also business orientated and can be used to facilitate strategic business planning.     

Forecasting performance of multivariate time series models with full and reduced rank: an empirical examination

Since Quenouille's influential work on multiple time series, much progress has been made towards the goal of parameter reduction and model fit. Relatively less attention has been paid to the systematic evaluation of out-of-sample forecast performance of multivariate time series models. In this paper, we update the hog data set studied by Quenouille (and other researchers who followed him). We re-estimate his model with extended observations (1867–1966), and generate recursive one- to four-steps-ahead forecasts for the period of 1967 through 2000. These forecasts are compared to forecasts from an unrestricted vector autoregression, a reduced rank regression model, an index model and a cointegration-based error correction model. The error correction model that takes into account both nonstationarity of the data and rank reduction performs best at all four forecasting horizons. However, differences among competing models are statistically insignificant in most cases. No model consistently encompasses the others at all four horizons.     

Evaluation of the best M4 competition methods for small area population forecasting

The ‘M4’ forecasting competition results were featured recently in a special issue of the International Journal of Forecasting and included projections for demographic time series. We sought to investigate whether the best M4 methods could improve the accuracy of small area population forecasts, which generally suffer from much higher forecast errors than regions with larger populations. The aim of this study was to apply the top ten M4 forecasting methods to produce 5- and 10-year forecasts of small area total populations using historical datasets from Australia and New Zealand. Forecasts were compared against the actual population numbers and forecasts from two simple benchmark models. The M4 methods were found to perform relatively well compared to our benchmarks. In the light of these findings, we discuss possible future directions for small area population forecasting research.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.     

Treating and Pruning: New approaches to forecasting model selection and combination using prediction intervals

We propose a new way of selecting among model forms in automated exponential smoothing routines, consequently enhancing their predictive power. The procedure, here addressed as treating, operates by selectively subsetting the ensemble of competing models based on information from their prediction intervals. By the same token, we set forth a pruning strategy to improve the accuracy of both point forecasts and prediction intervals in forecast combination methods. The proposed approaches are respectively applied to automated exponential smoothing routines and Bagging algorithms, to demonstrate their potential. An empirical experiment is conducted on a wide range of series from the M-Competitions. The results attest that the proposed approaches are simple, without requiring much additional computational cost, but capable of substantially improving forecasting accuracy for both point forecasts and prediction intervals, outperforming important benchmarks and recently developed forecast combination methods.     

Combination of long term and short term forecasts, with application to tourism demand forecasting

Forecast combination is a well-established and well-tested approach for improving the forecasting accuracy. One beneficial strategy is to use constituent forecasts that have diverse information. In this paper we consider the idea of diversity being accomplished by using different time aggregations. For example, we could create a yearly time series from a monthly time series and produce forecasts for both, then combine the forecasts. These forecasts would each be tracking the dynamics of different time scales, and would therefore add diverse types of information. A comparison of several forecast combination methods, performed in the context of this setup, shows that this is indeed a beneficial strategy and generally provides a forecasting performance that is better than the performances of the individual forecasts that are combined.As a case study, we consider the problem of forecasting monthly tourism numbers for inbound tourism to Egypt. Specifically, we consider 33 individual source countries, as well as the aggregate. The novel combination strategy also produces a generally improved forecasting accuracy.     

GROEC: Combination method via Generalized Rolling Origin Evaluation

Combination methods have performed well in time series forecast competitions. This study proposes a simple but general methodology for combining time series forecast methods. Weights are calculated using a cross-validation scheme that assigns greater weights to methods with more accurate in-sample predictions. The methodology was used to combine forecasts from the Theta, exponential smoothing, and ARIMA models, and placed fifth in the M4 Competition for both point and interval forecasting.     

Do IMF forecasts respect Okun’s law? Evidence for advanced and developing economies

This paper provides an assessment of the IMF’s unemployment forecasts, which have not received much scrutiny to date. The focus is on the internal consistency of the IMF’s growth and unemployment forecasts, and specifically on seeing whether the relationship between the two is consistent with the relationship in the data, i.e., with Okun’s Law. We find that the average performance is good, in the sense that the relationship between growth and unemployment forecasts is fairly comparable to that which prevails in the data: on average, the Okun coefficient in the forecasts mirrors the Okun coefficient in the data. Nevertheless, there is room for improvement, particularly in the year-ahead forecasts and for the group of middle-income countries. We show that a linear combination of Okun-based unemployment forecasts and WEO unemployment forecasts can deliver significant gains in forecast accuracy for developing economies.     

Combining exponential smoothing forecasts using Akaike weights

Simple forecast combinations such as medians and trimmed or winsorized means are known to improve the accuracy of point forecasts, and Akaike’s Information Criterion (AIC) has given rise to so-called Akaike weights, which have been used successfully to combine statistical models for inference and prediction in specialist fields, e.g., ecology and medicine. We examine combining exponential smoothing point and interval forecasts using weights derived from AIC, small-sample-corrected AIC and BIC on the M1 and M3 Competition datasets. Weighted forecast combinations perform better than forecasts selected using information criteria, in terms of both point forecast accuracy and prediction interval coverage. Simple combinations and weighted combinations do not consistently outperform one another, while simple combinations sometimes perform worse than single forecasts selected by information criteria. We find a tendency for a longer history to be associated with a better prediction interval coverage.     

Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination

In this paper, we examine the forecast accuracy of linear autoregressive, smooth transition autoregressive (STAR), and neural network (NN) time series models for 47 monthly macroeconomic variables of the G7 economies. Unlike previous studies that typically consider multiple but fixed model specifications, we use a single but dynamic specification for each model class. The point forecast results indicate that the STAR model generally outperforms linear autoregressive models. It also improves upon several fixed STAR models, demonstrating that careful specification of nonlinear time series models is of crucial importance. The results for neural network models are mixed in the sense that at long forecast horizons, an NN model obtained using Bayesian regularization produces more accurate forecasts than a corresponding model specified using the specific-to-general approach. Reasons for this outcome are discussed.     

Forecasting the daily federal funds rate

This paper compares two alternative one-day-ahead forecasts of tomorrow's federal funds rate. The first forecast is a simple random walk forecast in which the forecast of tomorrow's federal funds rate is taken to be today's federal funds rate. The second forecast is an ARIMA model forecast that was allowed to vary with changes in the Federal Reserve System's operating procedures. These two forecasts are compared in terms of their general forecast accuracy and the decision support they provide to a financial institution hypothesized to be borrowing $7 million a week in the federal funds market. Even in cases felt to be most favorable to the ARIMA forecasts, the degree of forecast accuracy and decision support superiority of the ARIMA forecasts is found to be quite small.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Do survey joiners and leavers differ from regular participants? The US SPF GDP growth and inflation forecasts

If ‘learning by doing’ is important for macro-forecasting, newcomers might be different from regular, established participants. Stayers may also differ from the soon-to-leave. We test these conjectures for macro-forecasters’ point predictions of output growth and inflation, and for their histogram forecasts. Histogram forecasts of inflation by both joiners and leavers are found to be less accurate, especially if we suppose that joiners take time to learn. For GDP growth, there is no evidence of differences between the groups in terms of histogram forecast accuracy, although GDP point forecasts by leavers are less accurate. These findings are predicated on forecasters being homogeneous within groups. Allowing for individual fixed effects suggests fewer differences, including leavers’ inflation histogram forecasts being no less accurate.     

Stability in the inefficient use of forecasting systems: A case study in a supply chain company

Computer-based demand forecasting systems have been widely adopted in supply chain companies, but little research has studied how these systems are actually used in the forecasting process. We report the findings of a case study of demand forecasting in a pharmaceutical company over a 15-year period. At the start of the study, managers believed that they were making extensive use of their forecasting system that was marketed based on the accuracy of its advanced statistical methods. Yet most forecasts were obtained using the system’s facility for judgmentally overriding the automatic statistical forecasts. Carrying out the judgmental interventions involved considerable management effort as part of a sales & operations planning (S&OP) process, yet these often only served to reduce forecast accuracy. This study uses observations of the forecasting process, interviews with participants and data on the accuracy of forecasts to investigate why the managers continued to use non-normative forecasting practices for many years despite the potential economic benefits that could be achieved through change. The reasons for the longevity of these practices are examined both from the perspective of the individual forecaster and the organization as a whole.     

Real-time macroeconomic forecasting with leading indicators: An empirical comparison

This paper demonstrates that the Conference Board’s Composite Leading Index (CLI) has significant real-time predictive ability for Industrial Production (IP) growth rates at horizons from one to six months ahead over the period 1989-2009. A popular but unrealistic analysis, which combines real-time data for CLI and final vintage data for IP as predictor variables, obscures the actual predictive content of the CLI, in the sense that in that case, the improvements in forecast accuracy relative to a univariate AR model are not significant. The CLI appears to be less useful for forecasting growth rates of the Conference Board’s Composite Coincident Index (CCI) in real time, as a univariate AR model performs better. This result is mostly due to its disappointing performance during the first five years of the forecast period. The CLI may not be the best way of exploiting the information contained in the underlying individual leading indicator variables. The use of principal components instead of CLI leads to more accurate real-time forecasts for both IP and CCI growth rates.     

Anticipating special events in Emergency Department forecasting

Accurate daily forecast of Emergency Department (ED) attendance helps roster planners in allocating available resources more effectively and potentially influences staffing. Since special events affect human behaviours, they may increase or decrease the demand for ED services. Therefore, it is crucial to model their impact and use them to forecast future attendance to improve roster planning and avoid reactive strategies. In this paper, we propose, for the first time, a forecasting model to generate both point and probabilistic daily forecast of ED attendance. We model the impact of special events on ED attendance by considering real-life ED data. We benchmark the accuracy of our model against three time-series techniques and a regression model that does not consider special events. We show that the proposed model outperforms its benchmarks across all horizons for both point and probabilistic forecasts. Results also show that our model is more robust with an increasing forecasting horizon. Moreover, we provide evidence on how different types of special events may increase or decrease ED attendance. Our model can easily be adapted for use not only by EDs but also by other health services. It could also be generalised to include more types of special events.     

Combining forecasts: Performance and coherence

There is general agreement in many forecasting contexts that combining individual predictions leads to better final forecasts. However, the relative error reduction in a combined forecast depends upon the extent to which the component forecasts contain unique/independent information. Unfortunately, obtaining independent predictions is difficult in many situations, as these forecasts may be based on similar statistical models and/or overlapping information. The current study addresses this problem by incorporating a measure of coherence into an analytic evaluation framework so that the degree of independence between sets of forecasts can be identified easily. The framework also decomposes the performance and coherence measures in order to illustrate the underlying aspects that are responsible for error reduction. The framework is demonstrated using UK retail prices index inflation forecasts for the period 1998–2014, and implications for forecast users are discussed.     

Combining vector forecasts to predict thoroughbred horse race outcomes

A large body of empirical studies has shown that a forecast developed by combining individual base forecasts performs surprisingly well. Previous work on the combination of forecasts has been confined to the area of time series forecasting. This work extends the combination of forecasts technique into the domain of forecasting one-time competitive events, specifically the scaled, relative finishing position of horses in thoroughbred sprint races. The present research develops a framework for the selection of the base forecasts and selects 12 base forecasts for analysis. The performance of the combination of the base forecasts is assessed on a sample of sprint races. Results of the analysis strongly suggest that the combination approach is both appropriate and effective. Some differences in results between this work and previous work in the time series domain suggest promising avenues for future research.