FFORMPP: Feature-based forecast model performance prediction

This paper introduces a novel meta-learning algorithm for time series forecast model performance prediction. We model the forecast error as a function of time series features calculated from historical time series with an efficient Bayesian multivariate surface regression approach. The minimum predicted forecast error is then used to identify an individual model or a combination of models to produce the final forecasts. It is well known that the performance of most meta-learning models depends on the representativeness of the reference dataset used for training. In such circumstances, we augment the reference dataset with a feature-based time series simulation approach, namely GRATIS, to generate a rich and representative time series collection. The proposed framework is tested using the M4 competition data and is compared against commonly used forecasting approaches. Our approach provides comparable performance to other model selection and combination approaches but at a lower computational cost and a higher degree of interpretability, which is important for supporting decisions. We also provide useful insights regarding which forecasting models are expected to work better for particular types of time series, the intrinsic mechanisms of the meta-learners, and how the forecasting performance is affected by various factors.     

Predictability of bull and bear markets: A new look at forecasting stock market regimes (and returns) in the US

The empirical literature of stock market predictability mainly suffers from model uncertainty and parameter instability. To meet this challenge, we propose a novel approach that combines dimensionality reduction, regime-switching models, and forecast combination to predict excess returns on the S&P 500. First, we aggregate the weekly information of 146 popular macroeconomic and financial variables using different principal component analysis techniques. Second, we estimate Markov-switching models with time-varying transition probabilities using the principal components as predictors. Third, we pool the models in forecast clusters to hedge against model risk and to evaluate the usefulness of different specifications. Our weekly forecasts respond to regime changes in a timely manner to participate in recoveries or to prevent losses. This is also reflected in an improvement of risk-adjusted performance measures as compared to several benchmarks. However, when considering stock market returns, our forecasts do not outperform common benchmarks. Nevertheless, they do add statistical and, in particular, economic value during recessions or in declining markets.     

LoMEF: A framework to produce local explanations for global model time series forecasts

Global forecasting models (GFMs) that are trained across a set of multiple time series have shown superior results in many forecasting competitions and real-world applications compared with univariate forecasting approaches. One aspect of the popularity of statistical forecasting models such as ETS and ARIMA is their relative simplicity and interpretability (in terms of relevant lags, trend, seasonality, and other attributes), while GFMs typically lack interpretability, especially relating to particular time series. This reduces the trust and confidence of stakeholders when making decisions based on the forecasts without being able to understand the predictions. To mitigate this problem, we propose a novel local model-agnostic interpretability approach to explain the forecasts from GFMs. We train simpler univariate surrogate models that are considered interpretable (e.g., ETS) on the predictions of the GFM on samples within a neighbourhood that we obtain through bootstrapping, or straightforwardly as the one-step-ahead global black-box model forecasts of the time series which needs to be explained. After, we evaluate the explanations for the forecasts of the global models in both qualitative and quantitative aspects such as accuracy, fidelity, stability, and comprehensibility, and are able to show the benefits of our approach.     

Mixed random forest,cointegration, and forecasting gasoline prices

One of the most successful forecasting machine learning (ML) procedures is random forest (RF). In this paper, we propose a new mixed RF approach for modeling departures from linearity that helps identify (i) explanatory variables with nonlinear impacts, (ii) threshold values, and (iii) the closest parametric approximation. The methodology is applied to weekly forecasts of gasoline prices, cointegrated with international oil prices and exchange rates. Recent specifications for nonlinear error correction (NEC) models include threshold autoregressive models (TAR) and double-threshold smooth transition autoregressive (STAR) models. We propose a new mixed RF model specification strategy and apply it to the determinants of weekly prices of the Spanish gasoline market from 2010 to 2019. In particular, the mixed RF is able to identify nonlinearities in both the error correction term and the rate of change of oil prices. It provides the best weekly gasoline price forecasting performance and supports the logistic error correction model (ECM) approximation.     

Automatic Interpretable Retail forecasting with promotional scenarios

Budgeting and planning processes require medium-term sales forecasts with marketing scenarios. The complexity in modern retailing necessitates consistent, automatic forecasting and insight generation. Remedies to the high dimensionality problem have drawbacks; black box machine learning methods require voluminous data and lack insights, while regularization may bias causal estimates in interpretable models.The proposed FAIR (Fully Automatic Interpretable Retail forecasting) method supports the retail planning process with multi-step-ahead category-store level forecasts, scenario evaluations, and insights. It considers category-store specific seasonality, focal- and cross-category marketing, and adaptive base sales while dealing with regularization-induced confounding.We show, with three chains from the IRI dataset involving 30 categories, that regularization-induced confounding decreases forecast accuracy. By including focal- and cross-category marketing, as well as random disturbances, forecast accuracy is increased. FAIR is more accurate than the black box machine learning method Boosted Trees and other benchmarks while also providing insights that are in line with the marketing literature.     

Ordinal-response GARCH models for transaction data: A forecasting exercise

We use numerous high-frequency transaction data sets to evaluate the forecasting performances of several dynamic ordinal-response time series models with generalized autoregressive conditional heteroscedasticity (GARCH). The specifications account for three components: leverage effects, in-mean effects and moving average error terms. We estimate the model parameters by developing Markov chain Monte Carlo algorithms. Our empirical analysis shows that the proposed ordinal-response GARCH models achieve better point and density forecasts than standard benchmarks.     

Forecasting third-party mobile payments with implications for customer flow prediction

Forecasting customer flow is key for retailers in making daily operational decisions, but small retailers often lack the resources to obtain such forecasts. Rather than forecasting stores’ total customer flows, this research utilizes emerging third-party mobile payment data to provide participating stores with a value-added service by forecasting their share of daily customer flows. These customer transactions using mobile payments can then be utilized further to derive retailers’ total customer flows indirectly, thereby overcoming the constraints that small retailers face. We propose a third-party mobile-payment-platform centered daily mobile payments forecasting solution based on an extension of the newly-developed Gradient Boosting Regression Tree (GBRT) method which can generate multi-step forecasts for many stores concurrently. Using empirical forecasting experiments with thousands of time series, we show that GBRT, together with a strategy for multi-period-ahead forecasting, provides more accurate forecasts than established benchmarks. Pooling data from the platform across stores leads to benefits relative to analyzing the data individually, thus demonstrating the value of this machine learning application.     

Forecasting the UK economy with a medium-scale Bayesian VAR

We estimate a Bayesian VAR (BVAR) for the UK economy and assess its performance in forecasting GDP growth and CPI inflation in real time relative to forecasts from COMPASS, the Bank of England’s DSGE model, and other benchmarks. We find that the BVAR outperformed COMPASS when forecasting both GDP and its expenditure components. In contrast, their performances when forecasting CPI were similar. We also find that the BVAR density forecasts outperformed those of COMPASS, despite under-predicting inflation at most forecast horizons. Both models over-predicted GDP growth at all forecast horizons, but the issue was less pronounced in the BVAR. The BVAR’s point and density forecast performances are also comparable to those of a Bank of England in-house statistical suite for both GDP and CPI inflation, as well as to the official Inflation Report projections. Our results are broadly consistent with the findings of similar studies for other advanced economies.     

Forecasting the UK economy: Alternative forecasting methodologies and the role of off-model information

How did DSGE model forecasts perform before, during and after the financial crisis, and what type of off-model information can improve the forecast accuracy? We tackle these questions by assessing the real-time forecast performance of a large DSGE model relative to statistical and judgmental benchmarks over the period from 2000 to 2013. The forecasting performances of all methods deteriorate substantially following the financial crisis. That is particularly evident for the DSGE model’s GDP forecasts, but augmenting the model with a measure of survey expectations made its GDP forecasts more accurate, which supports the idea that timely off-model information is particularly useful in times of financial distress.     

Forecasting the M4 competition weekly data: Forecast Pro’s winning approach

Forecast Pro forecasted the weekly series in the M4 competition more accurately than all other entrants. Our approach was to follow the same forecasting process that we recommend to our users. This approach involves determining the Key Performance Metric (KPI), establishing baseline forecasts using our automated expert selection algorithm, reviewing those baseline forecasts and customizing forecasts where needed. This article explores why this approach worked well for weekly data, discusses the applicability of the M4 competition to business forecasting and proposes some potential improvements for future competitions to make them more relevant to business forecasting.     

Forecasting week-to-week television ratings using reduced-form and structural dynamic models

Rather than being sold several months before a program is aired, more than 20% of TV advertising slots are retained for sale weekly near the program’s broadcast time. Distinct from the literature that mainly focuses on the forecasting of program ratings for advanced sales of advertising slots, we explore approaches that can provide more accurate forecasts for near real-time ratings. We propose two dynamic models that mainly employ individual viewing records for past episodes to forecast viewers’ decisions on episodes in the coming week, and therefore the ratings for these episodes. One is a reduced-form dynamic model that measures the influence of past watching experience by the weighted average of the viewers’ choices of past episodes. The other is a structural dynamic model that goes deeper in its use of previous viewing information by modeling the underlying process of this influence based on the Bayesian updating theory. Using data from the Hong Kong TV industry, we test and compare the two models. Results show that the reduced-form model generally performs better when the variance of ratings across episodes is small, while the structural model generates more accurate forecasts in other cases.     

Regional heterogeneity and U.S. presidential elections: Real-time 2020 forecasts and evaluation

This paper exploits cross-sectional variation at the level of U.S. counties to generate real-time forecasts for the 2020 U.S. presidential election. The forecasting models are trained on data covering the period 2000–2016, using high-dimensional variable selection techniques. Our county-based approach contrasts the literature that focuses on national and state level data but uses longer time periods to train their models. The paper reports forecasts of popular and electoral college vote outcomes and provides a detailed ex-post evaluation of the forecasts released in real time before the election. It is shown that all of these forecasts outperform autoregressive benchmarks. A pooled national model using One-Covariate-at-a-time-Multiple-Testing (OCMT) variable selection significantly outperformed all models in forecasting the U.S. mainland national vote share and electoral college outcomes (forecasting 236 electoral votes for the Republican party compared to 232 realized). This paper also shows that key determinants of voting outcomes at the county level include incumbency effects, unemployment, poverty, educational attainment, house price changes, and international competitiveness. The results are also supportive of myopic voting: economic fluctuations realized a few months before the election tend to be more powerful predictors of voting outcomes than their long-horizon analogs.     

Seasonal exponential smoothing with damped trends : An application for production planning

This paper reviews a spreadsheet-based forecasting approach which a process industry manufacturer developed and implemented to link annual corporate forecasts with its manufacturing/distribution operations. First, we consider how this forecasting system supports overall production planning and why it must be compatible with corporate forecasts. We then review the results of substantial testing of variations on the Winters three-parameter exponential smoothing model on 28 actual product family time series. In particular, we evaluate whether the use of damping parameters improves forecast accuracy. The paper concludes that a Winters four-parameter model (i.e. the standard Winters three-parameter model augmented by a fourth parameter to damp the trend) provides the most accurate forecasts of the models evaluated. Our application confirms the fact that there are situations where the use of damped trend parameters in short-run exponential smoothing based forecasting models is beneficial.     

Forecasting football match results in national league competitions using score-driven time series models

We develop a new dynamic multivariate model for the analysis and forecasting of football match results in national league competitions. The proposed dynamic model is based on the score of the predictive observation mass function for a high-dimensional panel of weekly match results. Our main interest is in forecasting whether the match result is a win, a loss or a draw for each team. The dynamic model for delivering such forecasts can be based on three different dependent variables: the pairwise count of the number of goals, the difference between the numbers of goals, or the category of the match result (win, loss, draw). The different dependent variables require different distributional assumptions. Furthermore, different dynamic model specifications can be considered for generating the forecasts. We investigate empirically which dependent variable and which dynamic model specification yield the best forecasting results. We validate the precision of the resulting forecasts and the success of the forecasts in a betting simulation in an extensive forecasting study for match results from six large European football competitions. Finally, we conclude that the dynamic model for pairwise counts delivers the most precise forecasts while the dynamic model for the difference between counts is most successful for betting, but that both outperform benchmark and other competing models.     

Model combinations through revised base rates

Standard selection criteria for forecasting models focus on information that is calculated for each series independently, disregarding the general tendencies and performance of the candidate models. In this paper, we propose a new way to perform statistical model selection and model combination that incorporates the base rates of the candidate forecasting models, which are then revised so that the per-series information is taken into account. We examine two schemes that are based on the precision and sensitivity information from the contingency table of the base rates. We apply our approach on pools of either exponential smoothing or ARMA models, considering both simulated and real time series, and show that our schemes work better than standard statistical benchmarks. We test the significance and sensitivity of our results, discuss the connection of our approach to other cross-learning approaches, and offer insights regarding implications for theory and practice.     

Forecasting economic and financial variables with global VARs

This paper considers the problem of forecasting economic and financial variables across a large number of countries in the global economy. To this end a global vector autoregressive (GVAR) model, previously estimated by Dees, di Mauro, Pesaran, and Smith (2007) and Dees, Holly, Pesaran, and Smith (2007) over the period 1979Q1–2003Q4, is used to generate out-of-sample forecasts one and four quarters ahead for real output, inflation, real equity prices, exchange rates and interest rates over the period 2004Q1–2005Q4. Forecasts are obtained for 134 variables from 26 regions, which are made up of 33 countries and cover about 90% of the world output. The forecasts are compared to typical benchmarks: univariate autoregressive and random walk models. Building on the forecast combination literature, the effects of model and estimation uncertainty on forecast outcomes are examined by pooling forecasts obtained from different GVAR models estimated over alternative sample periods. Given the size of the modelling problem, and the heterogeneity of the economies considered–industrialised, emerging, and less developed countries–as well as the very real likelihood of possibly multiple structural breaks, averaging forecasts across both models and windows makes a significant difference. Indeed, the double-averaged GVAR forecasts perform better than the benchmark competitors, especially for output, inflation and real equity prices.     

Benchmarking robustness of load forecasting models under data integrity attacks

As the internet’s footprint continues to expand, cybersecurity is becoming a major concern for both governments and the private sector. One such cybersecurity issue relates to data integrity attacks. This paper focuses on the power industry, where the forecasting processes rely heavily on the quality of the data. Data integrity attacks are expected to harm the performances of forecasting systems, which will have a major impact on both the financial bottom line of power companies and the resilience of power grids. This paper reveals the effect of data integrity attacks on the accuracy of four representative load forecasting models (multiple linear regression, support vector regression, artificial neural networks, and fuzzy interaction regression). We begin by simulating some data integrity attacks through the random injection of some multipliers that follow a normal or uniform distribution into the load series. Then, the four aforementioned load forecasting models are used to generate one-year-ahead ex post point forecasts in order to provide a comparison of their forecast errors. The results show that the support vector regression model is most robust, followed closely by the multiple linear regression model, while the fuzzy interaction regression model is the least robust of the four. Nevertheless, all four models fail to provide satisfying forecasts when the scale of the data integrity attacks becomes large. This presents a serious challenge to both load forecasters and the broader forecasting community: the generation of accurate forecasts under data integrity attacks. We construct our case study using the publicly-available data from Global Energy Forecasting Competition 2012. At the end, we also offer an overview of potential research topics for future studies.     

Anticipating special events in Emergency Department forecasting

Accurate daily forecast of Emergency Department (ED) attendance helps roster planners in allocating available resources more effectively and potentially influences staffing. Since special events affect human behaviours, they may increase or decrease the demand for ED services. Therefore, it is crucial to model their impact and use them to forecast future attendance to improve roster planning and avoid reactive strategies. In this paper, we propose, for the first time, a forecasting model to generate both point and probabilistic daily forecast of ED attendance. We model the impact of special events on ED attendance by considering real-life ED data. We benchmark the accuracy of our model against three time-series techniques and a regression model that does not consider special events. We show that the proposed model outperforms its benchmarks across all horizons for both point and probabilistic forecasts. Results also show that our model is more robust with an increasing forecasting horizon. Moreover, we provide evidence on how different types of special events may increase or decrease ED attendance. Our model can easily be adapted for use not only by EDs but also by other health services. It could also be generalised to include more types of special events.     

Forecasting from others’ experience: Bayesian estimation of the generalized Bass model

We propose a Bayesian estimation procedure for the generalized Bass model that is used in product diffusion models. Our method forecasts product sales early based on previous similar markets; that is, we obtain pre-launch forecasts by analogy. We compare our forecasting proposal to traditional estimation approaches, and alternative new product diffusion specifications. We perform several simulation exercises, and use our method to forecast the sales of room air conditioners, BlackBerry handheld devices, and compressed natural gas. The results show that our Bayesian proposal provides better predictive performances than competing alternatives when little or no historical data are available, which is when sales projections are the most useful.     

fETSmcs: Feature-based ETS model component selection

The well-developed ETS (ExponenTial Smoothing, or Error, Trend, Seasonality) method incorporates a family of exponential smoothing models in state space representation and is widely used for automatic forecasting. The existing ETS method uses information criteria for model selection by choosing an optimal model with the smallest information criterion among all models fitted to a given time series. The ETS method under such a model selection scheme suffers from computational complexity when applied to large-scale time series data. To tackle this issue, we propose an efficient approach to ETS model selection by training classifiers on simulated data to predict appropriate model component forms for a given time series. We provide a simulation study to show the model selection ability of the proposed approach on simulated data. We evaluate our approach on the widely used M4 forecasting competition dataset in terms of both point forecasts and prediction intervals. To demonstrate the practical value of our method, we showcase the performance improvements from our approach on a monthly hospital dataset.