Evaluating the forecasting performance of econometric models of air passenger traffic flows using multiple error measures

Airline traffic forecasting is important to airlines and regulatory authorities. This paper examines a number of approaches to forecasting short- to medium-term air traffic flows. It contributes as a rare replication, testing a variety of alternative modelling approaches. The econometric models employed include autoregressive distributed lag (ADL) models, time-varying parameter (TVP) models and an automatic method for econometric model specification. A vector autoregressive (VAR) model and various univariate alternatives are also included to deliver unconditional forecast comparisons. Various approaches for taking into account interactions between contemporaneous air traffic flows are examined, including pooled ADL models and the enhanced models with the addition of a “world trade” variable. Based on the analysis of a number of forecasting error measures, it is concluded that pooled ADL models that include the “world trade” variable outperform the alternatives, and in particular univariate methods; and, second, that automatic modelling procedures are enhanced through judgmental intervention. In contrast to earlier results, the TVP models do not improve accuracy. Depending on the preferred error measure, the difference in accuracy may be substantial.     

The performance of alternative forecasting methods for SETAR models

We compare a number of methods that have been proposed in the literature for obtaining h-step ahead minimum mean square error forecasts for self-exciting threshold autoregressive (SETAR) models. These forecasts are compared to those from an AR model. The comparison of forecasting methods is made using Monte Carlo simulation. The Monte-Carlo method of calculating SETAR forecasts is generally at least as good as that of the other methods we consider. An exception is when the disturbances in the SETAR model come from a highly asymmetric distribution, when a Bootstrap method is to be preferred.An empirical application calculates multi-period forecasts from a SETAR model of US gross national product using a number of the forecasting methods. We find that whether there are improvements in forecast performance relative to a linear AR model depends on the historical epoch we select, and whether forecasts are evaluated conditional on the regime the process was in at the time the forecast was made.     

Modeling and forecasting of Brazilian reservoir inflows via dynamic linear models

This work focuses on developing a forecasting model for the water inflow at an hydroelectric plant’s reservoir for operations planning. The planning horizon is 5 years in monthly steps. Due to the complex behavior of the monthly inflow time series we use a Bayesian dynamic linear model that incorporates seasonal and autoregressive components. We also use climate variables like monthly precipitation, El Niño and other indices as predictor variables when relevant. The Brazilian power system has 140 hydroelectric plants. Based on geographical considerations, these plants are collated by basin and classified into 15 groups that correspond to the major river basins, in order to reduce the dimension of the problem. The model is then tested for these 15 groups. Each group will have a different forecasting model that can best describe its unique seasonality and characteristics. The results show that the forecasting approach taken in this paper produces substantially better predictions than the current model adopted in Brazil (see Maceira & Damazio, 2006), leading to superior operations planning.     

Comparison of intraday probabilistic forecasting of solar irradiance using only endogenous data

Accurate solar forecasts are necessary to improve the integration of solar renewables into the energy grid. In recent years, numerous methods have been developed for predicting the solar irradiance or the output of solar renewables. By definition, a forecast is uncertain. Thus, the models developed predict the mean and the associated uncertainty. Comparisons are therefore necessary and useful for assessing the skill and accuracy of these new methods in the field of solar energy.The aim of this paper is to present a comparison of various models that provide probabilistic forecasts of the solar irradiance within a very strict framework. Indeed, we consider focusing on intraday forecasts, with lead times ranging from 1 to 6 h. The models selected use only endogenous inputs for generating the forecasts. In other words, the only inputs of the models are the past solar irradiance data. In this context, the most common way of generating the forecasts is to combine point forecasting methods with probabilistic approaches in order to provide prediction intervals for the solar irradiance forecasts. For this task, we selected from the literature three point forecasting models (recursive autoregressive and moving average (ARMA), coupled autoregressive and dynamical system (CARDS), and neural network (NN)), and seven methods for assessing the distribution of their error (linear model in quantile regression (LMQR), weighted quantile regression (WQR), quantile regression neural network (QRNN), recursive generalized autoregressive conditional heteroskedasticity (GARCHrls), sieve bootstrap (SB), quantile regression forest (QRF), and gradient boosting decision trees (GBDT)), leading to a comparison of 20 combinations of models.None of the model combinations clearly outperform the others; nevertheless, some trends emerge from the comparison. First, the use of the clear sky index ensures the accuracy of the forecasts. This derived parameter permits time series to be deseasonalized with missing data, and is also a good explanatory variable of the distribution of the forecasting errors. Second, regardless of the point forecasting method used, linear models in quantile regression, weighted quantile regression and gradient boosting decision trees are able to forecast the prediction intervals accurately.     

Forecast combinations of computational intelligence and linear models for the NN5 time series forecasting competition

In this work we introduce the forecasting model with which we participated in the NN5 forecasting competition (the forecasting of 111 time series representing daily cash withdrawal amounts at ATM machines). The main idea of this model is to utilize the concept of forecast combination, which has proven to be an effective methodology in the forecasting literature. In the proposed system we attempted to follow a principled approach, and make use of some of the guidelines and concepts that are known in the forecasting literature to lead to superior performance. For example, we considered various previous comparison studies and time series competitions as guidance in determining which individual forecasting models to test (for possible inclusion in the forecast combination system). The final model ended up consisting of neural networks, Gaussian process regression, and linear models, combined by simple average. We also paid extra attention to the seasonality aspect, decomposing the seasonality into weekly (which is the strongest one), day of the month, and month of the year seasonality.     

Parameter-efficient deep probabilistic forecasting

Probabilistic time series forecasting is crucial in many application domains, such as retail, ecommerce, finance, and biology. With the increasing availability of large volumes of data, a number of neural architectures have been proposed for this problem. In particular, Transformer-based methods achieve state-of-the-art performance on real-world benchmarks. However, these methods require a large number of parameters to be learned, which imposes high memory requirements on the computational resources for training such models. To address this problem, we introduce a novel bidirectional temporal convolutional network that requires an order of magnitude fewer parameters than a common Transformer-based approach. Our model combines two temporal convolutional networks: the first network encodes future covariates of the time series, whereas the second network encodes past observations and covariates. We jointly estimate the parameters of an output distribution via these two networks. Experiments on four real-world datasets show that our method performs on par with four state-of-the-art probabilistic forecasting methods, including a Transformer-based approach and WaveNet, on two point metrics (sMAPE and NRMSE) as well as on a set of range metrics (quantile loss percentiles) in the majority of cases. We also demonstrate that our method requires significantly fewer parameters than Transformer-based methods, which means that the model can be trained faster with significantly lower memory requirements, which as a consequence reduces the infrastructure cost for deploying these models.     

System economics: Overcoming the pitfalls of forecasting models via a multidisciplinary approach

In areas from medicine to climate change to economics, we are faced with huge challenges and a need for accurate forecasts, yet our ability to predict the future has been found wanting. The basic problem is that complex systems such as the atmosphere or the economy can not be reduced to simple mathematical laws and modeled accordingly. The equations in numerical models are therefore only approximations to reality, and are often highly sensitive to external influences and small changes in parameterisation — they can be made to fit past data, but are less good at prediction. Since decisions are usually based on our best models of the future, how can we proceed? This paper draws a comparison between two apparently different fields: biology and economics. In biology, drug development is a highly inefficient and expensive process, which in the past has relied heavily on trial and error. Institutions such as pharmaceutical companies and universities are now radically changing their approach and adopting techniques from the new field of systems biology to integrate information from disparate sources and improve the development process. A similar revolution is required in economics if models are to reflect the nature of human economic activity and provide useful tools for policy makers. We outline the main foundations for a theory of systems economics.     

Validation and forecasting accuracy in models of climate change

Forecasting researchers, with few exceptions, have ignored the current major forecasting controversy: global warming and the role of climate modelling in resolving this challenging topic. In this paper, we take a forecaster’s perspective in reviewing established principles for validating the atmospheric-ocean general circulation models (AOGCMs) used in most climate forecasting, and in particular by the Intergovernmental Panel on Climate Change (IPCC). Such models should reproduce the behaviours characterising key model outputs, such as global and regional temperature changes. We develop various time series models and compare them with forecasts based on one well-established AOGCM from the UK Hadley Centre. Time series models perform strongly, and structural deficiencies in the AOGCM forecasts are identified using encompassing tests. Regional forecasts from various GCMs had even more deficiencies. We conclude that combining standard time series methods with the structure of AOGCMs may result in a higher forecasting accuracy. The methodology described here has implications for improving AOGCMs and for the effectiveness of environmental control policies which are focussed on carbon dioxide emissions alone. Critically, the forecast accuracy in decadal prediction has important consequences for environmental planning, so its improvement through this multiple modelling approach should be a priority.     

Improving demand forecasting in the air cargo handling industry: a case study

Air transportation plays a crucial role in the agile and dynamic environment of contemporary supply chains. This industry is characterised by high air cargo demand uncertainty, making forecasting extremely challenging. An in-depth case study has been undertaken in order to explore and untangle the factors influencing demand forecasting and consequently to improve the operational performance of an air cargo handling company. It has been identified that in practice, the demand forecasting process does not provide the necessary level of accuracy, to effectively cope with the high demand uncertainty. This has a negative impact on a whole range of air cargo operations, but especially on the management of the workforce, which is the most expensive resource in the air cargo handling industry. Besides forecast inaccuracy, a range of additional hidden factors that affect operations management have been identified. A number of recommendations have been made to improve demand forecasting and workforce management.     

Considerations of a retail forecasting practitioner

Forecasts can be used in an extraordinarily diverse range of ways across many domains in which forecasting practitioners work continuously towards improving their forecasts. Each of these domains may require the analysis of different kinds of inputs and special considerations. Even within a given domain, such as retail, there may be many similar use cases of the same kind of forecast, which can lead to practitioners making different decisions. This paper discusses several of the important decision points that practitioners must work through and uses item-level sales forecasting in the retail domain as leveraged by pricing and inventory management as examples of the different paths that may be taken. It considers how each use can lead to a different forecasting objective, and a corresponding focus on different error metrics. In addition, there are several tradeoffs in the forecasting methods that are used to meet each of the objectives best, including the kinds of models used, the running time speed, and forecast accuracy requirements.     

Density forecasting through disaggregation

In this paper, the revised expectations model (REM) is developed to incorporate economic agents’ price expectation formation effects. With this incorporation, two models, an aggregate one sector model and a disaggregated multi-sector model, are estimated and used in density forecasting of the US real GDP growth rate. The experiment shows that use of the disaggregated version of the model, which incorporates price expectation effects along with modern Bayesian MCMC estimation and prediction techniques, produces more precise density forecasts than those yielded by either an aggregate version or benchmark forecasting models.     

Holt’s exponential smoothing and neural network models for forecasting interval-valued time series

Interval-valued time series are interval-valued data that are collected in a chronological sequence over time. This paper introduces three approaches to forecasting interval-valued time series. The first two approaches are based on multilayer perceptron (MLP) neural networks and Holt’s exponential smoothing methods, respectively. In Holt’s method for interval-valued time series, the smoothing parameters are estimated by using techniques for non-linear optimization problems with bound constraints. The third approach is based on a hybrid methodology that combines the MLP and Holt models. The practicality of the methods is demonstrated through simulation studies and applications using real interval-valued stock market time series.     

Fiscal forecasting performance in an emerging economy: An empirical assessment of Brazil

This study makes use of Brazilian data to analyze government budget balance forecast errors. Besides the analysis of the quality and efficiency of budget balance forecasts, economic, political, and institutional and governance dimensions are explored. The findings show that the data forecasts have low quality and efficiency. Furthermore, it is observed that the budget forecast error is subject to a backward-looking effect, a bias in the economic growth forecasts, as well as cyclical fluctuations. Finally, electoral cycles represent a source of overestimated forecasts, and strong institutions and governance supported by the public are able to suppress opportunistic motivations in budget forecasts.     

Beyond point forecasting: Evaluation of alternative prediction intervals for tourist arrivals

This paper evaluates the performances of prediction intervals generated from alternative time series models, in the context of tourism forecasting. The forecasting methods considered include the autoregressive (AR) model, the AR model using the bias-corrected bootstrap, seasonal ARIMA models, innovations state space models for exponential smoothing, and Harvey’s structural time series models. We use thirteen monthly time series for the number of tourist arrivals to Hong Kong and Australia. The mean coverage rates and widths of the alternative prediction intervals are evaluated in an empirical setting. It is found that all models produce satisfactory prediction intervals, except for the autoregressive model. In particular, those based on the bias-corrected bootstrap perform best in general, providing tight intervals with accurate coverage rates, especially when the forecast horizon is long.     

Forecasting realized variance measures using time-varying coefficient models

This paper considers the problem of forecasting realized variance measures. These measures are highly persistent estimates of the underlying integrated variance, but are also noisy. Bollerslev, Patton and Quaedvlieg (2016), Journal of Econometrics 192(1), 1–18 exploited this so as to extend the commonly used heterogeneous autoregressive (HAR) by letting the model parameters vary over time depending on the estimated measurement error variances. We propose an alternative specification that allows the autoregressive parameters of HAR models to be driven by a latent Gaussian autoregressive process that may also depend on the estimated measurement error variance. The model parameters are estimated by maximum likelihood using the Kalman filter. Our empirical analysis considers the realized variances of 40 stocks from the S&P 500. Our model based on log variances shows the best overall performance and generates superior forecasts both in terms of a range of different loss functions and for various subsamples of the forecasting period.     

The use of multiple scenarios in sales forecasting: An empirical test

Nearly 500 annual sales forecasts were generated from the responses of 82 subjects who were presented with either a time-series plot of historical sales data by itself or with the same plus three scenarios, and were then asked to make forecasts. Sales forecasts were made in either a stable or an unstable environment. The findings did not support the claims made by scenario advocates. Scenarios did not make unexpected outcomes less surprising. Instead, scenarios were found to increase confidence in a favored forecast. Furthermore, no support was found for the contention that scenarios improved upon ‘eyeball’ extrapolations or made judgmental sales forecasts more accurate than quantitative extrapolations. Scenarios were found to be tainted by many of the same biases previously identified by cognitive psychologists.     

Short-term forecasting of crime

The major question investigated is whether it is possible to accurately forecast selected crimes 1 month ahead in small areas, such as police precincts. In a case study of Pittsburgh, PA, we contrast the forecast accuracy of univariate time series models with naïve methods commonly used by police. A major result, expected for the small-scale data of this problem, is that average crime count by precinct is the major determinant of forecast accuracy. A fixed-effects regression model of absolute percent forecast error shows that such counts need to be on the order of 30 or more to achieve accuracy of 20% absolute forecast error or less. A second major result is that practically any model-based forecasting approach is vastly more accurate than current police practices. Holt exponential smoothing with monthly seasonality estimated using city-wide data is the most accurate forecast model for precinct-level crime series.     

Forecasting tourist arrivals using time-varying parameter structural time series models

Empirical evidence has shown that seasonal patterns of tourism demand and the effects of various influencing factors on this demand tend to change over time. To forecast future tourism demand accurately requires appropriate modelling of these changes. Based on the structural time series model (STSM) and the time-varying parameter (TVP) regression approach, this study develops the causal STSM further by introducing TVP estimation of the explanatory variable coefficients, and therefore combines the merits of the STSM and TVP models. This new model, the TVP-STSM, is employed for modelling and forecasting quarterly tourist arrivals to Hong Kong from four key source markets: China, South Korea, the UK and the USA. The empirical results show that the TVP-STSM outperforms all seven competitors, including the basic and causal STSMs and the TVP model for one- to four-quarter-ahead ex post forecasts and one-quarter-ahead ex ante forecasts.     

Understanding models’ forecasting performance

We propose a new methodology to identify the sources of models’ forecasting performance. The methodology decomposes the models’ forecasting performance into asymptotically uncorrelated components that measure instabilities in the forecasting performance, predictive content, and over-fitting. The empirical application shows the usefulness of the new methodology for understanding the causes of the poor forecasting ability of economic models for exchange rate determination.