Combining exponential smoothing forecasts using Akaike weights

Simple forecast combinations such as medians and trimmed or winsorized means are known to improve the accuracy of point forecasts, and Akaike’s Information Criterion (AIC) has given rise to so-called Akaike weights, which have been used successfully to combine statistical models for inference and prediction in specialist fields, e.g., ecology and medicine. We examine combining exponential smoothing point and interval forecasts using weights derived from AIC, small-sample-corrected AIC and BIC on the M1 and M3 Competition datasets. Weighted forecast combinations perform better than forecasts selected using information criteria, in terms of both point forecast accuracy and prediction interval coverage. Simple combinations and weighted combinations do not consistently outperform one another, while simple combinations sometimes perform worse than single forecasts selected by information criteria. We find a tendency for a longer history to be associated with a better prediction interval coverage.     

The asymmetry of judgemental confidence intervals in time series forecasting

In forecasting a time series, one may be asked to communicate the likely distribution of the future actual value, often expressed as a confidence interval. Whilst the accuracy (calibration) of these intervals has dominated most studies to date, this paper is concerned with other possible characteristics of the intervals. It reports a study in which the prevalence and determinants of the symmetry of judgemental confidence intervals in time series forecasting was examined. Most prior work has assumed that this interval is symmetrically placed around the forecast. However, this study shows that people generally estimate asymmetric confidence intervals where the forecast is not the midpoint of the estimated interval. Many of these intervals are grossly asymmetric. Results indicate that the placement of the forecast in relation to the last actual value of a time series is a major determinant of the direction and size of the asymmetry.     

The M4 Competition: 100,000 time series and 61 forecasting methods

The M4 Competition follows on from the three previous M competitions, the purpose of which was to learn from empirical evidence both how to improve the forecasting accuracy and how such learning could be used to advance the theory and practice of forecasting. The aim of M4 was to replicate and extend the three previous competitions by: (a) significantly increasing the number of series, (b) expanding the number of forecasting methods, and (c) including prediction intervals in the evaluation process as well as point forecasts. This paper covers all aspects of M4 in detail, including its organization and running, the presentation of its results, the top-performing methods overall and by categories, its major findings and their implications, and the computational requirements of the various methods. Finally, it summarizes its main conclusions and states the expectation that its series will become a testing ground for the evaluation of new methods and the improvement of the practice of forecasting, while also suggesting some ways forward for the field.     

Predicting/hypothesizing the findings of the M4 Competition

Science is caught up in a replication crisis which has negative implications for published findings that cannot be reproduced by other researchers. However, such is not the case with the M4 Competition, which not only provided the means of effectively reproducing its submissions, but also preregistered ten predictions/hypotheses about its expected results two-and-a-half months before its completion. From a scientific point of view, attempting to predict the results of a study is far more powerful than merely justifying them in hindsight after they have become available. The present paper presents these ten predictions/hypotheses that the organizers of the M4 Competition made and evaluates them based on the actual results. It is shown that at least six of the ten predictions/hypotheses were entirely correct, while two were partially correct, one required additional information to be confirmed, and the remaining one was not predicted correctly.     

The uncertainty track: Machine learning,statistical modeling,synthesis

This note provides an evaluation of the contributions of the M5 Competition to the construction of prediction intervals. We consider the choice of criteria used in the evaluations, the relative performance of designed and benchmark methods and the take-home lessons both for statistical forecasters and for those interested in forecasting retail sales.     

Evaluating quantile-bounded and expectile-bounded interval forecasts

In many different contexts, decision-making is improved by the availability of probabilistic predictions. The accuracy of probabilistic forecasting methods can be compared using scoring functions and insight provided by calibration tests. These tests evaluate the consistency of predictions with the observations. Our main agenda in this paper is interval forecasts and their evaluation. Such forecasts are usually bounded by two quantile forecasts. However, a limitation of quantiles is that they convey no information regarding the size of potential exceedances. By contrast, the location of an expectile is dictated by the whole distribution. This prompts us to propose expectile-bounded intervals. We provide interpretation, a consistent scoring function and a calibration test. Before doing this, we reflect on the evaluation of forecasts of quantile-bounded intervals and expectiles, and suggest extensions of previously proposed calibration tests in order to guard against strategic forecasting. We illustrate ideas using day-ahead electricity price forecasting.     

GROEC: Combination method via Generalized Rolling Origin Evaluation

Combination methods have performed well in time series forecast competitions. This study proposes a simple but general methodology for combining time series forecast methods. Weights are calculated using a cross-validation scheme that assigns greater weights to methods with more accurate in-sample predictions. The methodology was used to combine forecasts from the Theta, exponential smoothing, and ARIMA models, and placed fifth in the M4 Competition for both point and interval forecasting.     

A note on the Robust Trend and ARARMA methodologies used in the M3 Competition

This brief note describes two of the forecasting methods used in the M3 Competition, Robust Trend and ARARMA. The origins of these methods are very different. Robust Trend was introduced to model the special features of some telecommunications time series. It was subsequently found to be competitive with Holt’s linear model for the more varied set of time series used in the M1 Competition. The ARARMA methodology was proposed by Parzen as a general time series modelling procedure, and can be thought of as an alternative to the ARIMA methodology of Box and Jenkins. This method was used in the M1 Competition and achieved the lowest mean absolute percentage error for longer forecasting horizons. These methods will be described in more detail and some comments on their use in the M3 Competition conclude this note.     

Automatic ARIMA modeling including interventions, using time series expert software

This article has three objectives: (a) to describe the method of automatic ARIMA modeling (AAM), with and without intervention analysis, that has been used in the analysis; (b) to comment on the results; and (c) to comment on the M3 Competition in general. Starting with a computer program for fitting an ARIMA model and a methodology for building univariate ARIMA models, an expert system has been built, while trying to avoid the pitfalls of most existing software packages. A software package called Time Series Expert TSE-AX is used to build a univariate ARIMA model with or without an intervention analysis. The characteristics of TSE-AX are summarized and, more especially, its automatic ARIMA modeling method. The motivation to take part in the M3-Competition is also outlined. The methodology is described mainly in three technical appendices: (Appendix A) choice of differences and of a transformation, use of intervention analysis; ( Appendix B) available specification procedures; ( Appendix C) adequacy, model checking and new specification. The problems raised by outliers are discussed, in particular how close they are from the forecast origin. Several series that are difficult to deal with from that point of view are mentioned and one of them is shown. In the last section, we comment on contextual information, the idea of an e−M Competition, prediction intervals and the possible use of other forecasting methods within Time Series Expert.     

A longitudinal analysis of the relationship between multiple commitments and withdrawal cognitions

This paper examines the relationship between multiple commitments (affective organizational commitment, continuance organizational commitment, occupational commitment, and job involvement) and withdrawal cognitions, with three different time intervals between the two. One hundred and twenty-two community center employees (a 37% response rate) in Israel participated in the study. The findings showed that commitment forms were related to withdrawal cognitions, even when withdrawal cognitions measured earlier than or at the same time as commitment forms were controlled for. The results also showed that the timing of the measurement of the research variables had a strong effect on the findings. More specifically, the prediction of withdrawal is better, the shorter the interval between its measurement and the measurement of multiple commitments. The findings also showed that commitment and withdrawal are both dynamic concepts. That is, the effect of timing on the accuracy of the prediction can be a result of more immediate changes in commitment forms across time, or more immediate changes in withdrawal cognitions over time. Other implications of the findings for future research on commitment and withdrawal are also discussed.     

区间时间序列预测准确度探讨

区间时间序列在决策过程中提供重要的信息,特别是在经济发展、人口政策、规划管理或金融监管等方面,因此如何计算出预测区间的精确度成为一个重要议题。本文提出两种区间预测准确度分析的方法,通过估计预测结果的平均区间误差平方和及平均相对区间误差和,比较不同预测方法的优劣。并由预测区间与实际区间的重叠位置,充分说明预测方法所具有的有效性。这些分析预测区间准确度的方法,将为管理者提供更客观的决策空间。     

Forecasting in social settings: The state of the art

This paper provides a non-systematic review of the progress of forecasting in social settings. It is aimed at someone outside the field of forecasting who wants to understand and appreciate the results of the M4 Competition, and forms a survey paper regarding the state of the art of this discipline. It discusses the recorded improvements in forecast accuracy over time, the need to capture forecast uncertainty, and things that can go wrong with predictions. Subsequently, the review classifies the knowledge achieved over recent years into (i) what we know, (ii) what we are not sure about, and (iii) what we don’t knowIn the first two areas, we explore the difference between explanation and prediction, the existence of an optimal model, the performance of machine learning methods on time series forecasting tasks, the difficulties of predicting non-stable environments, the performance of judgment, and the value added by exogenous variables. The article concludes with the importance of (thin and) fat tails, the challenges and advances in causal inference, and the role of luck.     

A white-boxed ISSM approach to estimate uncertainty distributions of Walmart sales

We present our solution for the M5 Uncertainty competition. Our solution ranked sixth out of 909 submissions across all hierarchical levels and ranked first for prediction at the finest level of granularity (product-store sales, i.e. SKUs). The model combines a multi-stage state-space model and Monte Carlo simulations to generate the forecasting scenarios (trajectories). Observed sales are modelled with negative binomial distributions to represent discrete over-dispersed sales. Seasonal factors are handcrafted and modelled with linear coefficients that are calculated at the store-department level.     

Advanced workforce management for effective customer services

Because almost 60?C80% of the total costs for operating a contact centre involve wage and benefit expenses for personnel, determining the optimal number of agents available is of great importance in call centre management. In modern call centres, working hours are divided into planning intervals with identical lengths. Each planning interval is typically assumed to be a homogeneous Poisson process in a steady state, and simple queuing models, such as Erlang-C (M/M/c), are often applied to determine the optimal staffing levels of the planning intervals. However, since the actual length of the planning interval in practice is relatively short, the basic assumption of staffing analysis could be violated. In this paper, we numerically analyze an M/M/c+M call centre??s behavior in a transient state. As a result, we can determine appropriate staffing levels of a call centre with short planning intervals which do not assume to be in a steady state.     

A strategic predictive distribution for tests of probabilistic calibration

Forecasts of probability distributions are needed to support decision making in many applications. The accuracy of predictive distributions should be evaluated by maximising sharpness subject to calibration. Sharpness relates to the concentration of the predictive distributions, while calibration concerns their statistical consistency with the data. This paper focuses on calibration testing. It is important that a calibration test cannot be gamed by forecasts that have been strategically designed to pass the test. The widely used tests of probabilistic calibration for predictive distributions are based on the probability integral transform. Drawing on previous results for quantile prediction, we show that strategic distributional forecasting is a concern for these tests. To address this, we provide a simple extension of one of the tests. We illustrate ideas using simulated data.     

The M4 Competition: Results,findings, conclusion and way forward

The M4 competition is the continuation of three previous competitions started more than 45 years ago whose purpose was to learn how to improve forecasting accuracy, and how such learning can be applied to advance the theory and practice of forecasting. The purpose of M4 was to replicate the results of the previous ones and extend them into three directions: First significantly increase the number of series, second include Machine Learning (ML) forecasting methods, and third evaluate both point forecasts and prediction intervals. The five major findings of the M4 Competitions are: 1. Out Of the 17 most accurate methods, 12 were “combinations” of mostly statistical approaches. 2. The biggest surprise was a “hybrid” approach that utilized both statistical and ML features. This method’s average sMAPE was close to 10% more accurate than the combination benchmark used to compare the submitted methods. 3. The second most accurate method was a combination of seven statistical methods and one ML one, with the weights for the averaging being calculated by a ML algorithm that was trained to minimize the forecasting. 4. The two most accurate methods also achieved an amazing success in specifying the 95% prediction intervals correctly. 5. The six pure ML methods performed poorly, with none of them being more accurate than the combination benchmark and only one being more accurate than Naïve2. This paper presents some initial results of M4, its major findings and a logical conclusion. Finally, it outlines what the authors consider to be the way forward for the field of forecasting.     

Treating and Pruning: New approaches to forecasting model selection and combination using prediction intervals

We propose a new way of selecting among model forms in automated exponential smoothing routines, consequently enhancing their predictive power. The procedure, here addressed as treating, operates by selectively subsetting the ensemble of competing models based on information from their prediction intervals. By the same token, we set forth a pruning strategy to improve the accuracy of both point forecasts and prediction intervals in forecast combination methods. The proposed approaches are respectively applied to automated exponential smoothing routines and Bagging algorithms, to demonstrate their potential. An empirical experiment is conducted on a wide range of series from the M-Competitions. The results attest that the proposed approaches are simple, without requiring much additional computational cost, but capable of substantially improving forecasting accuracy for both point forecasts and prediction intervals, outperforming important benchmarks and recently developed forecast combination methods.     

The use of an expert system in the M3 competition

The Expert System that one of the authors had developed during his dissertation was tried on the data set of the M3 Competition. The expert system was originally designed to forecast monthly demand for industrial products in a distribution environment and was modified to run the data. The results of the application of the system were mixed as in some of the time series the results were statistically undistinguishable with the exception of the monthly series. In general, the intervention did not improve the accuracy and the effort required to do it was substantial.     

Prediction intervals in conditionally heteroscedastic time series with stochastic components

Differencing is a very popular stationary transformation for series with stochastic trends. Moreover, when the differenced series is heteroscedastic, authors commonly model it using an ARMA-GARCH model. The corresponding ARIMA-GARCH model is then used to forecast future values of the original series. However, the heteroscedasticity observed in the stationary transformation should be generated by the transitory and/or the long-run component of the original data. In the former case, the shocks to the variance are transitory and the prediction intervals should converge to homoscedastic intervals with the prediction horizon. We show that, in this case, the prediction intervals constructed from the ARIMA-GARCH models could be inadequate because they never converge to homoscedastic intervals. All of the results are illustrated using simulated and real time series with stochastic levels.     

Confidence Intervals for the Area Under the Receiver Operating Characteristic Curve in the Presence of Ignorable Missing Data

Receiver operating characteristic curves are widely used as a measure of accuracy of diagnostic tests and can be summarised using the area under the receiver operating characteristic curve (AUC). Often, it is useful to construct a confidence interval for the AUC; however, because there are a number of different proposed methods to measure variance of the AUC, there are thus many different resulting methods for constructing these intervals. In this article, we compare different methods of constructing Wald‐type confidence interval in the presence of missing data where the missingness mechanism is ignorable. We find that constructing confidence intervals using multiple imputation based on logistic regression gives the most robust coverage probability and the choice of confidence interval method is less important. However, when missingness rate is less severe (e.g. less than 70%), we recommend using Newcombe's Wald method for constructing confidence intervals along with multiple imputation using predictive mean matching.