Forecasting Errors: the Track Record

This Briefing Paper is the last of a series of three about forecasting. In this one we examine our forecasting record; it complements the February paper in which we analysed the properties of our forecasting model in terms of the error bands attached to the central forecast.
There are many ways of measuring forecasting errors, and in the first part of this Briefing Paper we describe briefing how we have tackled the problem. (A more detailed analysis can be found in the Appendix.) In Part II we report and comment upon the errors in our forecasts of annual growth rates and show how our forecasting performance has improved over the years. In Part III we focus on quarterly forecasts up to 8 quarters ahead, and compare our forecasting errors with measurement errors in the oficial statistics; with the estimation errors built into our forecast equations; and with the stochastic model errors we reported last February. A brief summary of the main conclusions is given below.     

Population forecasting theory,methods and assessments of accuracy. The magnitude of error due to different vital processes in population forecasts

The accuracy of population forecasts depends in part upon the method chosen for forecasting the vital rates of fertility, mortality, and migration. Methods for handling the stochastic propagation of error calculations in demographic forecasting are hard to do precisely. This paper discusses this obstacle in stochastic cohort-component population forecasts. The uncertainty of forecasts is due to uncertain estimates of the jump-off population and to errors in the forecasts of the vital rates. Empirically based of each source are presented and propagated through a simplified analytical model of population growth that allows assessment of the role of each component in the total error. Numerical estimates based on the errors of an actual vector ARIMA forecast of the US female population. These results broadly agree with those of the analytical model. This work especially uncertainty in the fertility forecasts to be so much higher than that in the other sources that the latter can be ignored in the propagation of error calculations for those cohorts that are born after the jump-off year of the forecast. A methodology is therefore presented which far simplifies the propagation of error calculations. It is noted, however, that the uncertainty of the jump-off population, migration, and mortality in the propagation of error for those alive at the jump-off time of the forecast must still be considered.     

Subjective evaluation of composite econometric policy inputs

Studies have suggested that a composite forecast may be preferred to a single forecast. In addition, forecasting objectives are often conflicting. For example, one forecast may have the smallest sum of absolute forecast errors, while another has the smallest maximum absolute error. This paper examines the appropriateness of using multiple objective linear programming to determine weighted linear combinations of forecasts to be used as inputs for policy analysis. An example is presented where the methodology is used to combine the forecasts for several policy variables. The forecasts are selected from large econometric, consensus, and univariate time series models.     

Integrated variance forecasting: Model based vs. reduced form

This paper compares model-based and reduced-form forecasts of financial volatility when high-frequency return data are available. We derived exact formulas for the forecast errors and analyzed the contribution of the “wrong” data modeling and errors in forecast inputs. The comparison is made for “feasible” forecasts, i.e., we assumed that the true data generating process, latent states and parameters are unknown. As an illustration, the same comparison is carried out empirically for spot 5 min returns of DM/USD exchange rates.It is shown that the comparison between feasible reduced-form and model-based forecasts is not always in favor of the latter in contrast to their infeasible versions. The reduced-form approach is generally better for long-horizon forecasting and for short-horizon forecasting in the presence of microstructure noise.     

Comparisons of aggregate variable forecasts using aggregate and disaggregate models

When forecasting aggregate variables, a choice must often be made to either add up individual forecasts made at a disaggregate level or to simply forecast at the aggregate level. The presence of heterogeneity introduces aggregation bias and makes the disaggregates approach more preferable, while the presence of data and specification errors introduces relatively large variances in the disaggregate forecasts, making the aggregate approach more preferable. It is suggested that the mean square error is useful in evaluating the combined effects of heterogeneity and specification and data errors, and in facilitating comparisons between aggregate and disaggregate approaches to aggregate variable forecasting.     

A comparison of the forecasting performance of WEFA and ARIMA time series methods

This paper examines the forecasting performance of the Wharton model (MARK III) over the period 1973 through 1975 and compares it with that of ARIMA models' performance over the same period. Despite strong intimation in the literature to the contrary, we find that this econometric model, at least, exhibits greater accuracy in every respect relative to ARIMA methods, in terms of its forecasts cum constant adjustments. When constant adjustments are disallowed then its forecasts are still more accurate than ARIMA forecasts over a 4- and 8-quarter forecasting horizon, but less accurate over a 1-quarter horizon. The comparison was carried out over twenty three macrovariables, under a slight handicap for the Wharton Model, in that the latter's parameters were estimated over a sample ending in 1969.3 while the ARIMA models were reidentified and reestimated as of the quarter immediately preceding the forecast.     

Forecasting automobile insurance paid claim costs using econometric and ARIMA models

Automobile insurance companies in the United States currently utilize simple exponential trend models to forecast paid claim costs, an important variable in ratemaking. This paper tests the performance of econometric and ARIMA models, as well as the current insurance industry method, in forecasting two paid claim cost series. The experiments encompass eight forecast periods ranging from 1974 through early 1983. The results indicate that automobile insurers could significantly improve their forecasts of property damage liability claim costs by adopting econometric models. For bodily injury liability claim costs, the accuracy of the econometric and insurance industry methods is approximately the same, and both outperform the ARIMA models. Overall, a net gain in accuracy could be achieved by adopting econometric models.     

Nonparametric and semiparametric regressions subject to monotonicity constraints: Estimation and forecasting

This paper considers nonparametric and semiparametric regression models subject to monotonicity constraint. We use bagging as an alternative approach to Hall and Huang (2001). Asymptotic properties of our proposed estimators and forecasts are established. Monte Carlo simulation is conducted to show their finite sample performance. An application to predicting equity premium is taken for illustration. We introduce a new forecasting evaluation criterion based on the second order stochastic dominance in the size of forecast errors and compare models over different sizes of forecast errors. Imposing monotonicity constraint can mitigate the chance of making large size forecast errors.     

Expert opinion versus expertise in forecasting

Expert opinion is an opinion given by an expert, and it can have significant value in forecasting key policy variables in economics and finance. Expert forecasts can either be expert opinions, or forecasts based on an econometric model. An expert forecast that is based on an econometric model is replicable, and can be defined as a replicable expert forecast (REF), whereas an expert opinion that is not based on an econometric model can be defined as a non-replicable expert forecast (Non-REF). Both REF and Non-REF may be made available by an expert regarding a policy variable of interest. In this paper, we develop a model to generate REF, and compare REF with Non-REF. A method is presented to compare REF and Non-REF using efficient estimation methods, and a direct test of expertise on expert opinion is given. The latter serves the purpose of investigating whether expert adjustment improves the model-based forecasts. Illustrations for forecasting pharmaceutical stock keeping unit (SKUs), where the econometric model is of (variations of) the autoregressive integrated moving average model (ARIMA) type, show the relevance of the new methodology proposed in the paper. In particular, experts possess significant expertise, and expert forecasts are significant in explaining actual sales.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Forecasting with an econometric model: Some recent results using the national institute model

It is widely believed that the large econometric models cannot be used for forecasting without considerable intervention on the part of the forecaster. In this paper we challenge this view by reproducing a number of recent forecasts published by the National Institute but without the ad hoc interventions used at the time. We show that in no case would the forecast, produced by the model used mechanically, have been radically different from that actually published. Further, in an ex-post comparison against actual out-turns, the mechanical model forecast is not obviously dominated by the published version.     

The tourism forecasting competition

We evaluate the performances of various methods for forecasting tourism data. The data used include 366 monthly series, 427 quarterly series and 518 annual series, all supplied to us by either tourism bodies or academics who had used them in previous tourism forecasting studies. The forecasting methods implemented in the competition are univariate and multivariate time series approaches, and econometric models. This forecasting competition differs from previous competitions in several ways: (i) we concentrate on tourism data only; (ii) we include approaches with explanatory variables; (iii) we evaluate the forecast interval coverage as well as the point forecast accuracy; (iv) we observe the effect of temporal aggregation on the forecasting accuracy; and (v) we consider the mean absolute scaled error as an alternative forecasting accuracy measure. We find that pure time series approaches provide more accurate forecasts for tourism data than models with explanatory variables. For seasonal data we implement three fully automated pure time series algorithms that generate accurate point forecasts, and two of these also produce forecast coverage probabilities which are satisfactorily close to the nominal rates. For annual data we find that Naïve forecasts are hard to beat.     

Forecasting and testing in co-integrated systems

This paper examines the behavior of forecasts made from a co-integrated system as introduced by Granger (1981), Granger and Weiss (1983) and Engle and Granger (1987). It is established that a multi-step forecast will satisfy the co-integrating relation exactly and that this particular linear combination of forecasts will have a finite limiting forecast error variance. A simulation study compares the multi-step forecast accuracy of unrestricted vector autoregression with the two-step estimation of the vector autoregression imposing the co-integration restriction.To test whether a system exhibits co-integration, the procedures introduced in Engle and Granger (1987) are extended to allow different sample sizes and numbers of variables.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

The performance of alternative forecasting methods for SETAR models

We compare a number of methods that have been proposed in the literature for obtaining h-step ahead minimum mean square error forecasts for self-exciting threshold autoregressive (SETAR) models. These forecasts are compared to those from an AR model. The comparison of forecasting methods is made using Monte Carlo simulation. The Monte-Carlo method of calculating SETAR forecasts is generally at least as good as that of the other methods we consider. An exception is when the disturbances in the SETAR model come from a highly asymmetric distribution, when a Bootstrap method is to be preferred.An empirical application calculates multi-period forecasts from a SETAR model of US gross national product using a number of the forecasting methods. We find that whether there are improvements in forecast performance relative to a linear AR model depends on the historical epoch we select, and whether forecasts are evaluated conditional on the regime the process was in at the time the forecast was made.     

Simple rules for combining forecasts: Some empirical results

This study is an empirical comparison of three rules for aggregating forecasts. The three combined forecasts evaluated are: a simple average forecast, a median forecast and a focus forecast. These combined forecasts are compared over four economic variables (housing starts, the index of industrial production, the unemployment rate and gross national product) using a set of previously published forecasts. The results indicate that an average forecast will not perform as well as previous studies indicate if all or most of the individual forecasts tend to over- or under-predict simultaneously. The median forecast also seems to be suspect in this case. There is little evidence to suggest that the median forecast is a viable alternative to the mean forecast. Focus forecasting, however, is found to perform well for all four variables. The evidence indicates that focus forecasting is a reasonable alternative to simple averaging.     

Modeling and forecasting the time series of US fertility: age distribution,range, and ultimate level

This discussion of modeling focuses on the difficulties in longterm, time-series forecasting of US fertility. Four possibilities are suggested. One difficulty with the traditional approach of using high or low bounds on fertility and mortality is that forecast errors are perfectly correlated over time, which means there are no cancellation of errors over time. The shape of future fertility intervals first increases, then stabilizes, and then decreases instead of remaining stable. This occurs because the number of terms being averaged increases with horizontal length. Alho and Spencer attempted to reduce these errors in time-series. Other difficulties are the idiosyncratic behavior of age specific fertility over time, biological bounds for total fertility rates (TFR) of 16 and zero, the integration of knowledge about fertility behavior that narrows the bounds, the unlikelihood of some probability outcomes of stochastic models with a normally distributed error term, the small relative change in TFR between years, a US fertility cycle of about 40 years, unimportant extrapolation of past trends in child and infant mortality, and the unlikelihood of reversals in mortality and contraceptive use trends. Another problem is the unsuitability of longterm forecasts. New methods include a model which estimates a one parameter family of fertility schedules and then forecasts that single parameter. Another method is a logistic transformation to account for prior information on the bounds on fertility estimates; this method is similar to Bayesian methods for ARMA models developed by Monahan. Models include information on the ultimate level of fertility and assume that the equilibrium level is a stochastic process trending over time. The horizon forecast method is preferred unless the effects of the outliers are known. Estimates of fertility are presented for the equilibrium constrained and logistic transformed model. Forecasts of age specific fertility rates can be calculated from forecasts of the fertility index (a single time varying parameter). The model of fertility fits poorly at older ages but captures some of the wide swings in the historical pattern. Age variations are not accounted for very well. Longterm forecasts tell a great deal about the uncertainty of forecast errors. Estimates are too sensitive to model specification for accuracy and ignore the biological and socioeconomic context.     

Value-added forecasting

Long ago, the emphasis shifted away from forecasting as a competitive weapon when it became apparent that forecast error could never be eliminated. Forecasts became a necessary evil that no one wanted to claim responsibility for. It's time to clear up some of the misconceptions about forecasts and to seize the opportunity inherent in the forecasting process. It is not forecast accuracy but rather improved understanding and use of forecasting as a tool for reducing both costs and lead times that will add real value to an enterprise and can improve the results from any and all other initiatives.     

Economic, organizational, and political influences on biases in forecasting state sales tax receipts

This paper investigates factors influencing fixed bias in forecasting state sales taxes revenues. By extending an existing model used to explain forecast accuracy to include a series of complex interactions related to the potential political and policy use of revenue forecasts, the paper extends our understanding of the forecasting process in government. Exploratory empirical analysis based on survey data is used to provide evidence that bias in forecasting results, at least in part, from political and policy manipulation. There is also evidence that institutional reforms associated with ‘good management’ practices affect forecast bias, but in complex ways depending upon the extent to which political competition exists within the state.     

On the forecasting performance of a small-scale DSGE model

Dynamic stochastic general equilibrium (DSGE) models have recently become standard tools for policy analysis. Nevertheless, their forecasting properties have still barely been explored. In this article, we address this problem by examining the quality of forecasts of the key U.S. economic variables: the three-month Treasury bill yield, the GDP growth rate and GDP price index inflation, from a small-size DSGE model, trivariate vector autoregression (VAR) models and the Philadelphia Fed Survey of Professional Forecasters (SPF). The ex post forecast errors are evaluated on the basis of the data from the period 1994–2006. We apply the Philadelphia Fed “Real-Time Data Set for Macroeconomists” to ensure that the data used in estimating the DSGE and VAR models was comparable to the information available to the SPF.Overall, the results are mixed. When comparing the root mean squared errors for some forecast horizons, it appears that the DSGE model outperforms the other methods in forecasting the GDP growth rate. However, this characteristic turned out to be statistically insignificant. Most of the SPF's forecasts of GDP price index inflation and the short-term interest rate are better than those from the DSGE and VAR models.