The role of judgment,assumptions, techniques,and confidence limits in forecasting population

This paper discusses the fundamental role judgment and assumptions play in forecasting population. It is suggested that so-called “projections” operationally are usually either forecasts or extrapolations. Specific projection methodologies and techniques are shown to embody assumptions. A simple typology of such assumptions is presented as a guide to evaluate forecasts. Tests of projection technique accuracy are cited and it is concluded that such tests cannot succeed due to the assumption factor. Finally, time series forecasting techniques are criticized because their terminology with respect to confidence limits about a forecast is misleading.     

Simple rules for combining forecasts: Some empirical results

This study is an empirical comparison of three rules for aggregating forecasts. The three combined forecasts evaluated are: a simple average forecast, a median forecast and a focus forecast. These combined forecasts are compared over four economic variables (housing starts, the index of industrial production, the unemployment rate and gross national product) using a set of previously published forecasts. The results indicate that an average forecast will not perform as well as previous studies indicate if all or most of the individual forecasts tend to over- or under-predict simultaneously. The median forecast also seems to be suspect in this case. There is little evidence to suggest that the median forecast is a viable alternative to the mean forecast. Focus forecasting, however, is found to perform well for all four variables. The evidence indicates that focus forecasting is a reasonable alternative to simple averaging.     

Evaluating quantile-bounded and expectile-bounded interval forecasts

In many different contexts, decision-making is improved by the availability of probabilistic predictions. The accuracy of probabilistic forecasting methods can be compared using scoring functions and insight provided by calibration tests. These tests evaluate the consistency of predictions with the observations. Our main agenda in this paper is interval forecasts and their evaluation. Such forecasts are usually bounded by two quantile forecasts. However, a limitation of quantiles is that they convey no information regarding the size of potential exceedances. By contrast, the location of an expectile is dictated by the whole distribution. This prompts us to propose expectile-bounded intervals. We provide interpretation, a consistent scoring function and a calibration test. Before doing this, we reflect on the evaluation of forecasts of quantile-bounded intervals and expectiles, and suggest extensions of previously proposed calibration tests in order to guard against strategic forecasting. We illustrate ideas using day-ahead electricity price forecasting.     

Forecasting macroeconomic time series: LASSO-based approaches and their forecast combinations with dynamic factor models

In a data-rich environment, forecasting economic variables amounts to extracting and organizing useful information from a large number of predictors. So far, the dynamic factor model and its variants have been the most successful models for such exercises. In this paper, we investigate a category of LASSO-based approaches and evaluate their predictive abilities for forecasting twenty important macroeconomic variables. These alternative models can handle hundreds of data series simultaneously, and extract useful information for forecasting. We also show, both analytically and empirically, that combing forecasts from LASSO-based models with those from dynamic factor models can reduce the mean square forecast error (MSFE) further. Our three main findings can be summarized as follows. First, for most of the variables under investigation, all of the LASSO-based models outperform dynamic factor models in the out-of-sample forecast evaluations. Second, by extracting information and formulating predictors at economically meaningful block levels, the new methods greatly enhance the interpretability of the models. Third, once forecasts from a LASSO-based approach are combined with those from a dynamic factor model by forecast combination techniques, the combined forecasts are significantly better than either dynamic factor model forecasts or the naïve random walk benchmark.     

A hybrid system-identification method for forecasting telecommunications product demands

A crucial challenge for telecommunications companies is how to forecast changes in demand for specific products over the next 6 to 18 months—the length of a typical short-range capacity-planning and capital-budgeting planning horizon. The problem is especially acute when only short histories of product sales are available. This paper presents a new two-level approach to forecasting demand from short-term data. The lower of the two levels consists of adaptive system-identification algorithms borrowed from signal processing, especially, Hidden Markov Model (HMM) methods [Hidden Markov Models: Estimation and Control (1995) Springer Verlag]. Although they have primarily been used in engineering applications such as automated speech recognition and seismic data processing, HMM techniques also appear to be very promising for predicting probabilities of individual customer behaviors from relatively short samples of recent product-purchasing histories. The upper level of our approach applies a classification tree algorithm to combine information from the lower-level forecasting algorithms. In contrast to other forecast-combination algorithms, such as weighted averaging or Bayesian aggregation formulas, the classification tree approach exploits high-order interactions among error patterns from different predictive systems. It creates a hybrid, forecasting algorithm that out-performs any of the individual algorithms on which it is based. This tree-based approach to hybridizing forecasts provides a new, general way to combine and improve individual forecasts, whether or not they are based on HMM algorithms. The paper concludes with the results of validation tests. These show the power of HMM methods to forecast what individual customers are likely to do next. They also show the gain from classification tree post-processing of the predictions from lower-level forecasts. In essence, these techniques enhance the limited techniques available for new product forecasting.     

Revenue forecasting, budget setting and risk

The budget-making process can be viewed as a problem of decision making under uncertainty since revenues are unknown at the time the budget is written. Revenue forecasts become information for describing the uncertainty in revenue receipts, thereby allowing the decision maker to trade off desire for a larger budget against the uncertainty in revenues. A chance constraint decision model is used to model this process. Application of the model to evaluate alternative forecasting techniques is then demonstrated for several revenue sources used in Kansas City, Missouri.     

A bootstrap-based non-parametric forecast density

Interest in density forecasts (as opposed to solely modeling the conditional mean) arises from the possibility of dynamics in higher moments of a time series, as well as in forecasting the probability of future events in some applications. By combining the idea of Markov bootstrapping with that of kernel density estimation, this paper presents a simple non-parametric method for estimating out-of-sample multi-step density forecasts. The paper also considers a host of evaluation tests for examining the dynamic misspecification of estimated density forecasts by targeting autocorrelation, heteroskedasticity and neglected non-linearity. These tests are useful, as a rejection of the tests gives insight into ways to improve a particular forecasting model. In an extensive Monte Carlo analysis involving a range of commonly used linear and non-linear time series processes, the non-parametric method is shown to work reasonably well across the simulated models for a suitable choice of the bandwidth (smoothing parameter). Furthermore, an application of the method to the U.S. Industrial Production series provides multi-step density forecasts that show no sign of dynamic misspecification.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Co-integration constraint and forecasting: An empirical examination

Does co-integration help long-term forecasts? In this paper, we use simulation, real data sets, and multi-step-ahead post-sample forecasts to study this question. Based on the square root of the trace of forecasting error-covariance matrix, we found that for simulated data imposing the ‘correct’ unit-root constraints implied by co-integration does improve the accuracy of forecasts. For real data sets, the answer is mixed. Imposing unit-root constraints suggested by co-integration tests produces better forecasts for some cases, but fares poorly for others. We give some explanations for the poor performance of co-integration in long-term forecasting and discuss the practical implications of the study. Finally, an adaptive forecasting procedure is found to perform well in one- to ten-step-ahead forecasts.     

Forecast encompassing tests and probability forecasts

We consider tests of forecast encompassing for probability forecasts, for both quadratic and logarithmic scoring rules. We propose test statistics for the null of forecast encompassing, present the limiting distributions of the test statistics, and investigate the impact of estimating the forecasting models' parameters on these distributions. The small‐sample performance is investigated, in terms of small numbers of forecasts and model estimation sample sizes. We show the usefulness of the tests for the evaluation of recession probability forecasts from logit models with different leading indicators as explanatory variables, and for evaluating survey‐based probability forecasts. Copyright © 2009 John Wiley & Sons, Ltd.     

The tourism forecasting competition

We evaluate the performances of various methods for forecasting tourism data. The data used include 366 monthly series, 427 quarterly series and 518 annual series, all supplied to us by either tourism bodies or academics who had used them in previous tourism forecasting studies. The forecasting methods implemented in the competition are univariate and multivariate time series approaches, and econometric models. This forecasting competition differs from previous competitions in several ways: (i) we concentrate on tourism data only; (ii) we include approaches with explanatory variables; (iii) we evaluate the forecast interval coverage as well as the point forecast accuracy; (iv) we observe the effect of temporal aggregation on the forecasting accuracy; and (v) we consider the mean absolute scaled error as an alternative forecasting accuracy measure. We find that pure time series approaches provide more accurate forecasts for tourism data than models with explanatory variables. For seasonal data we implement three fully automated pure time series algorithms that generate accurate point forecasts, and two of these also produce forecast coverage probabilities which are satisfactorily close to the nominal rates. For annual data we find that Naïve forecasts are hard to beat.     

The impossibility of accurate macro-economic forecasting

The macro-economic short-term forecasting record in the West over the past thirty years is very poor. Modern non-linear signal processing techniques can be used to show that such inaccuracy is a deep and inherent property of the data themselves. The forecasting record simply cannot be improved. Much economic policy still focuses on short-term intervention based on short-term forecasts. But such efforts are futile because forecasts of sufficient accuracy over time cannot be made.     

Individual versus group spot price forecasting in the international petroleum market: A case study

This paper utilizes the conventional statistical tests associated with the rational expectations hypothesis so as to compare the relative accuracy of individual versus group forecasting within the organization. In order to maintain comparability between forecasting regimens the study employs like information sets for the two prediction methods. Using the rational expectations tests as criteria, the statistical results show group forecasts inferior to individually produced predictions These findings imply that group-produced forecasting accuracy may be hampered by the psychological interaction associated with consensus behavior. Conversely, we find forecasting accuracy improves when predictions are elicited from individuals in an isolated laboratory-like setting.     

Forecasting inflation using time‐varying Bayesian model averaging

This paper presents a Bayesian model averaging regression framework for forecasting US inflation, in which the set of predictors included in the model is automatically selected from a large pool of potential predictors and the set of regressors is allowed to change over time. Using real‐time data on the 1960–2011 period, this model is applied to forecast personal consumption expenditures and gross domestic product deflator inflation. The results of this forecasting exercise show that, although it is not able to beat a simple random‐walk model in terms of point forecasts, it does produce superior density forecasts compared with a range of alternative forecasting models. Moreover, a sensitivity analysis shows that the forecasting results are relatively insensitive to prior choices and the forecasting performance is not affected by the inclusion of a very large set of potential predictors.     

Order-invariant tests for proper calibration of multivariate density forecasts

Established tests for proper calibration of multivariate density forecasts based on Rosenblatt probability integral transforms can be manipulated by changing the order of variables in the forecasting model. We derive order-invariant tests. The new tests are applicable to densities of arbitrary dimensions and can deal with parameter estimation uncertainty and dynamic misspecification. Monte Carlo simulations show that they often have superior power relative to established approaches. We use the tests to evaluate generalized autoregressive conditional heteroskedasticity-based multivariate density forecasts for a vector of stock market returns and macroeconomic forecasts from a Bayesian vector autoregression with time-varying parameters.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

Modelling and forecasting stock returns: exploiting the futures market,regime shifts and international spillovers

This paper proposes a vector equilibrium correction model of stock returns that exploits the information in the futures market, while allowing for both regime‐switching behaviour and international spillovers across stock market indices. Using data for three major stock market indices since 1989, we find that: (i) in sample, our model outperforms several alternative models on the basis of standard statistical criteria; (ii) in out‐of‐sample forecasting, our model does not produce significant gains in terms of point forecasts relative to more parsimonious alternative specifications, but it does so both in terms of market timing ability and in density forecasting performance. The economic value of the density forecasts is illustrated with an application to a simple risk management exercise. Copyright © 2005 John Wiley & Sons, Ltd.     

Tests of Predictive Ability for Vector Autoregressions Used for Conditional Forecasting

Many forecasts are conditional in nature. For example, a number of central banks routinely report forecasts conditional on particular paths of policy instruments. Even though conditional forecasting is common, there has been little work on methods for evaluating conditional forecasts. This paper provides analytical, Monte Carlo and empirical evidence on tests of predictive ability for conditional forecasts from estimated models. In the empirical analysis, we examine conditional forecasts obtained with a VAR in the variables included in the DSGE model of Smets and Wouters (American Economic Review 2007; 97 : 586–606). Throughout the analysis, we focus on tests of bias, efficiency and equal accuracy applied to conditional forecasts from VAR models. Copyright © 2016 John Wiley & Sons, Ltd.     

DeepAR: Probabilistic forecasting with autoregressive recurrent networks

Probabilistic forecasting, i.e., estimating a time series’ future probability distribution given its past, is a key enabler for optimizing business processes. In retail businesses, for example, probabilistic demand forecasts are crucial for having the right inventory available at the right time and in the right place. This paper proposes DeepAR, a methodology for producing accurate probabilistic forecasts, based on training an autoregressive recurrent neural network model on a large number of related time series. We demonstrate how the application of deep learning techniques to forecasting can overcome many of the challenges that are faced by widely-used classical approaches to the problem. By means of extensive empirical evaluations on several real-world forecasting datasets, we show that our methodology produces more accurate forecasts than other state-of-the-art methods, while requiring minimal manual work.