Stability in the inefficient use of forecasting systems: A case study in a supply chain company

Computer-based demand forecasting systems have been widely adopted in supply chain companies, but little research has studied how these systems are actually used in the forecasting process. We report the findings of a case study of demand forecasting in a pharmaceutical company over a 15-year period. At the start of the study, managers believed that they were making extensive use of their forecasting system that was marketed based on the accuracy of its advanced statistical methods. Yet most forecasts were obtained using the system’s facility for judgmentally overriding the automatic statistical forecasts. Carrying out the judgmental interventions involved considerable management effort as part of a sales & operations planning (S&OP) process, yet these often only served to reduce forecast accuracy. This study uses observations of the forecasting process, interviews with participants and data on the accuracy of forecasts to investigate why the managers continued to use non-normative forecasting practices for many years despite the potential economic benefits that could be achieved through change. The reasons for the longevity of these practices are examined both from the perspective of the individual forecaster and the organization as a whole.     

Forecasting from time series subject to sporadic perturbations: Effectiveness of different types of forecasting support

How effective are different approaches for the provision of forecasting support? Forecasts may be either unaided or made with the help of statistical forecasts. In practice, the latter are often crude forecasts that do not take sporadic perturbations into account. Most research considers forecasts based on series that have been cleansed of perturbation effects. This paper considers an experiment in which people made forecasts from time series that were disturbed by promotions. In all conditions, under-forecasting occurred during promotional periods and over-forecasting during normal ones. The relative sizes of these effects depended on the proportions of periods in the data series that contained promotions. The statistical forecasts improved the forecasting accuracy, not because they reduced these biases, but because they decreased the random error (scatter). The performance improvement did not depend on whether the forecasts were based on cleansed series. Thus, the effort invested in producing cleansed time series from which to forecast may not be warranted: companies may benefit from giving their forecasters even crude statistical forecasts. In a second experiment, forecasters received optimal statistical forecasts that took the effects of promotions into account fully. This increased the accuracy because the biases were almost eliminated and the random error was reduced by 20%. Thus, the additional effort required to produce forecasts that take promotional effects into account is worthwhile.     

Asymmetric loss functions and the rationality of expected stock returns

We combine the innovative approaches of Elliott, Komunjer, and Timmermann (2005) and Patton and Timmermann (2007) with a block bootstrap to analyze whether asymmetric loss functions can rationalize the S&P 500 return expectations of individual forecasters from the Livingston Surveys. Although the rationality of these forecasts has often been rejected, earlier studies have relied on the assumption that positive and negative forecast errors of identical magnitudes are equally important to forecasters. Allowing for homogenous asymmetric loss, our evidence still strongly rejects forecast rationality. However, if we allow for variation in asymmetric loss functions across forecasters, not only do we find significant differences in preferences, but also we can often no longer reject forecast rationality. Our conclusions raise serious doubts about the homogeneous expectations assumption often made in asset pricing, portfolio construction and corporate finance models.     

To tell the truth or the perceived truth: Structural estimation of peer effects in China’s macroeconomic forecast

This study investigates the strategic interactions among China’s professional macroeconomic forecasters in the context of a static game with incomplete information. Professional forecasters attempt to be more precise than their peers when they are uncertain about others’ ability to forecast, given their own ability to forecast macroeconomy. We then empirically estimate the peer effects using the two-step method proposed by Bajari et al. (2010). The results identify a pronounced peer effect among professional forecasters and specify the asymmetric peer effect exerted by prominent professional forecasters. The results remain valid through several robustness checks. The forecast customers must thus address the peer effects due to competition among professional forecasters when they use forecasting reports.     

What determines forecasters’ forecasting errors?

This paper contributes to the growing body of literature in macroeconomics and finance on expectation formation and information processing by analyzing the relationship between expectation formation at the individual level and the prediction of macroeconomic aggregates. Using information from business tendency surveys, we present a new approach of analyzing forecasters’ qualitative forecasting errors. Based on a quantal response approach with misclassification, we define forecasters’ qualitative mispredictions in terms of deviations from the qualitative rational expectation forecast, and relate them to the individual and macro factors that are driving these mispredictions. Our approach permits a detailed analysis of individual forecasting decisions, allowing for the introduction of individual and economy-wide determinants that affect the individual forecasting error process.     

Forecasting spare part demand with installed base information: A review

The classical spare part demand forecasting literature studies methods for forecasting intermittent demand. However, the majority of these methods do not consider the underlying demand-generating factors. The demand for spare parts originates from the replacement of parts in the installed base of machines, either preventively or upon breakdown of the part. This information from service operations, which we refer to as installed base information, can be used to forecast the future demand for spare parts. This paper reviews the literature on the use of such installed base information for spare part demand forecasting in order to asses (1) what type of installed base information can be useful; (2) how this information can be used to derive forecasts; (3) the value of using installed base information to improve forecasting; and (4) the limits of the existing methods. This serves as motivation for future research.     

The uncertainty track: Machine learning,statistical modeling,synthesis

This note provides an evaluation of the contributions of the M5 Competition to the construction of prediction intervals. We consider the choice of criteria used in the evaluations, the relative performance of designed and benchmark methods and the take-home lessons both for statistical forecasters and for those interested in forecasting retail sales.     

Forecasting inflation with online prices

Are survey-based forecasts unbeatable? They are not. This paper uses online price indices to forecast the Consumer Price Index. We find that online price indices anticipate changes in official inflation trends more than one month in advance. Our baseline one-month forecast outperforms Bloomberg surveys of forecasters, which only predict the contemporaneous inflation rate. Our baseline specification also outperforms statistical benchmark forecasts for Australia, Canada, France, Germany, Greece, Ireland, Italy, the Netherlands, the United Kingdom, and the United States. Similarly, our quarterly forecast for the US inflation rate substantially outperforms the Survey of Professional Forecasters.     

Trading and non-trading period realized market volatility: Does it matter for forecasting the volatility of US stocks?

We study the potential merits of using trading and non-trading period market volatilities to model and forecast the stock volatility over the next one to 22 days. We demonstrate the role of overnight volatility information by estimating heterogeneous autoregressive (HAR) model specifications with and without a trading period market risk factor using ten years of high-frequency data for the 431 constituents of the S&P 500 index. The stocks’ own overnight squared returns perform poorly across stocks and forecast horizons, as well as in the asset allocation exercise. In contrast, we find overwhelming evidence that the market-level volatility, proxied by S&P Mini futures, matters significantly for improving the model fit and volatility forecasting accuracy. The greatest model fit and forecast improvements are found for short-term forecast horizons of up to five trading days, and for the non-trading period market-level volatility. The documented increase in forecast accuracy is found to be associated with the stocks’ sensitivity to the market risk factor. Finally, we show that both the trading and non-trading period market realized volatilities are relevant in an asset allocation context, as they increase the average returns, Sharpe ratios and certainty equivalent returns of a mean–variance investor.     

Robust optimization of forecast combinations

We develop a methodology for constructing robust combinations of time series forecast models which improve upon a given benchmark specification for all symmetric and convex loss functions. Under standard regularity conditions, the optimal forecast combination asymptotically almost surely dominates the benchmark, and also optimizes the chosen goal function. The optimum in a given sample can be found by solving a convex optimization problem. An application to the forecasting of changes in the S&P 500 volatility index shows that robust optimized combinations improve significantly upon the out-of-sample forecasting accuracy of both simple averaging and unrestricted optimization.     

Predictability of bull and bear markets: A new look at forecasting stock market regimes (and returns) in the US

The empirical literature of stock market predictability mainly suffers from model uncertainty and parameter instability. To meet this challenge, we propose a novel approach that combines dimensionality reduction, regime-switching models, and forecast combination to predict excess returns on the S&P 500. First, we aggregate the weekly information of 146 popular macroeconomic and financial variables using different principal component analysis techniques. Second, we estimate Markov-switching models with time-varying transition probabilities using the principal components as predictors. Third, we pool the models in forecast clusters to hedge against model risk and to evaluate the usefulness of different specifications. Our weekly forecasts respond to regime changes in a timely manner to participate in recoveries or to prevent losses. This is also reflected in an improvement of risk-adjusted performance measures as compared to several benchmarks. However, when considering stock market returns, our forecasts do not outperform common benchmarks. Nevertheless, they do add statistical and, in particular, economic value during recessions or in declining markets.     

How reliable are forecasts of GDP growth and inflation for countries with limited coverage?

This paper analyses the performance of GDP growth and inflation forecasts for 25 transition countries between 1994 and 2007, as provided by 13 international institutions, including multilateral, private and academic forecasters. The empirical results show that there is a positive correlation between the number of forecasters covering a given country and the forecast accuracy. Simple combined forecasts are shown to be unbiased and more accurate than most of the individual forecasters, although also inefficient. However, only a few institutions provide efficient and unbiased forecasts, with just one out of 13 forecasters providing both unbiased and efficient forecasts of both GDP growth and inflation in the observed period. The directional analysis shows a correct forecast of the change in the forecast indicator in over two thirds of cases. However, the eventual outcome is within the range of available forecasts in less than half of the cases, with more than 40% of outcomes for GDP growth above the highest forecast. Encouragingly, forecasts are shown to be improving over time and becoming more accurate with the increase in the number of forecasting institutions – forecast accuracy measured by mean absolute error improves by 0.3 percentage points for growth and by 0.2 percentage points for inflation for each additional institution providing forecasts.     

Retail forecasting: Research and practice

This paper reviews the research literature on forecasting retail demand. We begin by introducing the forecasting problems that retailers face, from the strategic to the operational, as sales are aggregated over products to stores and to the company overall. Aggregated forecasting supports strategic decisions on location. Product-level forecasts usually relate to operational decisions at the store level. The factors that influence demand, and in particular promotional information, add considerable complexity, so that forecasters potentially face the dimensionality problem of too many variables and too little data. The paper goes on to evaluate evidence on comparative forecasting accuracy. Although causal models outperform simple benchmarks, adequate evidence on machine learning methods has not yet accumulated. Methods for forecasting new products are examined separately, with little evidence being found on the effectiveness of the various approaches. The paper concludes by describing company forecasting practices, offering conclusions as to both research gaps and barriers to improved practice.     

Informational efficiency tests of quarterly macroeconometric GNP forecasts from 1976 to 1985

The objective of the paper is to compare the informational efficiency of five macroeconometric and one statistical quarterly forecasting models. The results suggest that the forecasters inefficiently utilize readily available economic information. The qualitative effect for a particular information variable is the same across all forecasters exhibiting inefficiency. Further, the magnitude on coefficients of significant information variables are quite close. In particular, real GNP forecasts appear to not fully incorporate information about lagged M 1 growth and lagged changes in housing starts. Deflator forecasts can be improved by more fully specifying the degree of slackness in the economy as captured by capacity utilization and changes in the labor market.     

Economic Viewpoint: Economic Models and Economic Forecasting: Ptolemaic or Copernican?

Economic forecasters have come in for a lot of criticism in recent months. But this is not new. The failure to predict the boom in 1988 and ensuing inflation and balance of payments problems led to some serious re-examination of the structure of the main macroeconometric models in the UK. The speed with which the UK economy moved into sharp recession in the second half of 1990- also appeared to catch most forecasters on the hop. This Briefing Paper is not directly about forecasting; it is about macroeconometric models. However, forecasting provides by far the most extensive use of these models. Not all macroeconomic forecasters use a formal econometric model - some City economists find a spreadsheet more than adequate - but those that do find that they are an invaluable aid to clear thinking and provide an effective way of filtering all of the information that is available, in one form or another, about the economy. This paper was originally intended to provide a retrospective review of the development of UK macro-models since the late 1970s, coinciding broadly with my period at London Business School. However, it became clear that there was a need to argue more generally in favour of macro-modelling, given the numerous assaults that this activity has sustained over the last 15 years. I want to demonstrate that the conduct of research in this area - at the very least in the UK - has been, and continues to be, a progressive research strategy in the sense in which philosophers of science use this term. I believe that macro-modellers, because they have an obligation to forecast and to make the forecast public, are closer to how theoretical constructs in economics conflict with the observation of economic events, and provide a more robust testing ground for economic theories than the relatively narrow - though very important - confines of single equation statistical testing, that dominate academic journals. This should not mean that every new theoretical idea should be expected to be able to survive immediately the rigours of testing within an existing macro-model. One of the attractions of recent developments in macroeconomic theory has been an explicit attempt to seek to try to reconcile macroeconomics with micro-economic reasoning and to derive macroeconomic principles from how rational, maximizing individuals can be expected to behave in a market economy. There is always a need periodically to re-examine the basic postulates of any area of economics, especially one such as macroeconomics, which provides the basis for the conduct of national and international economic policy, and for providing explanations for economic cycles. Macro-econometric model building is a worthwhile exercise because it confronts theoretical models of how the macro-economy is supposed to work with the hard lessons of experience. The use of these models for forecasting is therefore crucial to their continued growth and development.     

Probabilistic forecasting of heterogeneous consumer transaction–sales time series

We present new Bayesian methodology for consumer sales forecasting. Focusing on the multi-step-ahead forecasting of daily sales of many supermarket items, we adapt dynamic count mixture models for forecasting individual customer transactions, and introduce novel dynamic binary cascade models for predicting counts of items per transaction. These transaction–sales models can incorporate time-varying trends, seasonality, price, promotion, random effects and other outlet-specific predictors for individual items. Sequential Bayesian analysis involves fast, parallel filtering on sets of decoupled items, and is adaptable across items that may exhibit widely-varying characteristics. A multi-scale approach enables information to be shared across items with related patterns over time in order to improve prediction, while maintaining the scalability to many items. A motivating case study in many-item, multi-period, multi-step-ahead supermarket sales forecasting provides examples that demonstrate an improved forecast accuracy on multiple metrics, and illustrates the benefits of full probabilistic models for forecast accuracy evaluation and comparison.     

The Predictive Ability of Geographic Segment Information at the Country, Continent, and Consolidated Levels

This study examines whether geographic information disclosed at an increasingly disaggregated level (specifically, consolidated vs. continent vs. country) results in increased predictive ability of company operations (specifically, sales, gross profit, and earnings). Multinational corporations (MNCs) are formed using a simulated merger approach by combining the annual operating results of six individual firms, one from each of six countries. This approach makes it possible to compare the forecasting accuracy of data disclosed at the country, continent, and consolidated levels, not possible using current geographic segment disclosures. Previous studies using year-ahead forecast models implicitly assume the predictive factors included in the models are significant in forecasting operating results. Using regression forecast models, this study tests whether the predictive factors included in the models are effective in forecasting operating results by examining the direction, size, and significance of the regression coefficient estimates. The coefficients provide evidence that exchange rate changes, inflation, and real GNP growth are useful in forecasting annual sales and gross profit. Whereas, at least for this sample and this time period, exchange rate changes, inflation, and real GNP growth are not significant variables in forecasting annual earnings. The results indicate that the accuracy of forecasts increase as sales and gross profit are disclosed at a more disaggregated geographic level. The hypothesized relationship between consolidated, continent, and country levels, while holding strongly under perfect foresight, holds to a lesser extent using forecasts of exchange rates, inflation, and real GNP.     

Effective judgmental forecasting in the context of fashion products

We study the conditions that influence judgmental forecasting effectiveness when predicting demand in the context of fashion products. Human judgment is of practical importance in this setting. Our goal is to investigate what type of decision support, in particular historical and/or contextual predictors, should be provided to human forecasters to improve their ability to detect and exploit linear and nonlinear cue–criterion relationships in the task environment. Using a field experiment on new product forecasts in the music industry, our analysis reveals that when forecasters are concerned with predictive accuracy and only managerial judgments are employed, providing both types of decision support data is beneficial. However, if judgmental forecasts are combined with a statistical forecast, restricting the decision support provided to human judges to contextual anchors is beneficial. We identify two novel interactions demonstrating that the exploitation of nonlinearities is easiest for human judgment if contextual data are present but historical data are absent. Thus, if the role of human judgment is to detect these nonlinearities (and the linearities are taken care of by some statistical model with which judgments are combined), then a restriction of the decision support provided makes sense. Implications for the theory and practice of building decision support models are discussed.     

Forecasting the volatility of asset returns: The informational gains from option prices

A new class of forecasting models is proposed that extends the realized GARCH class of models through the inclusion of option prices to forecast the variance of asset returns. The VIX is used to approximate option prices, resulting in a set of cross-equation restrictions on the model’s parameters. The full model is characterized by a nonlinear system of three equations containing asset returns, the realized variance, and the VIX, with estimation of the parameters based on maximum likelihood methods. The forecasting properties of the new class of forecasting models, as well as a number of special cases, are investigated and applied to forecasting the daily S&P500 index realized variance using intra-day and daily data from September 2001 to November 2017. The forecasting results provide strong support for including the realized variance and the VIX to improve variance forecasts, with linear conditional variance models performing well for short-term one-day-ahead forecasts, whereas log-linear conditional variance models tend to perform better for intermediate five-day-ahead forecasts.