Evaluating density forecasts: forecast combinations,model mixtures,calibration and sharpness

This paper reviews current density forecast evaluation procedures, and considers a proposal that such procedures be augmented by an assessment of ‘sharpness’. This was motivated by an example in which some standard evaluation procedures using probability integral transforms cannot distinguish the ideal forecast from several competing forecasts. We show that this example has some unrealistic features from a time series forecasting perspective, and so provides insecure foundations for the argument that existing calibration procedures are inadequate in practice. Our alternative, more realistic example shows how relevant statistical methods, including information‐based methods, provide the required discrimination between competing forecasts. We introduce a new test of density forecast efficiency. Copyright © 2010 John Wiley & Sons, Ltd.     

Integrated variance forecasting: Model based vs. reduced form

This paper compares model-based and reduced-form forecasts of financial volatility when high-frequency return data are available. We derived exact formulas for the forecast errors and analyzed the contribution of the “wrong” data modeling and errors in forecast inputs. The comparison is made for “feasible” forecasts, i.e., we assumed that the true data generating process, latent states and parameters are unknown. As an illustration, the same comparison is carried out empirically for spot 5 min returns of DM/USD exchange rates.It is shown that the comparison between feasible reduced-form and model-based forecasts is not always in favor of the latter in contrast to their infeasible versions. The reduced-form approach is generally better for long-horizon forecasting and for short-horizon forecasting in the presence of microstructure noise.     

Combining forecasts: An application to elections

We summarize the literature on the effectiveness of combining forecasts by assessing the conditions under which combining is most valuable. Using data on the six US presidential elections from 1992 to 2012, we report the reductions in error obtained by averaging forecasts within and across four election forecasting methods: poll projections, expert judgment, quantitative models, and the Iowa Electronic Markets. Across the six elections, the resulting combined forecasts were more accurate than any individual component method, on average. The gains in accuracy from combining increased with the numbers of forecasts used, especially when these forecasts were based on different methods and different data, and in situations involving high levels of uncertainty. Such combining yielded error reductions of between 16% and 59%, compared to the average errors of the individual forecasts. This improvement is substantially greater than the 12% reduction in error that had been reported previously for combining forecasts.     

The performance of alternative forecasting methods for SETAR models

We compare a number of methods that have been proposed in the literature for obtaining h-step ahead minimum mean square error forecasts for self-exciting threshold autoregressive (SETAR) models. These forecasts are compared to those from an AR model. The comparison of forecasting methods is made using Monte Carlo simulation. The Monte-Carlo method of calculating SETAR forecasts is generally at least as good as that of the other methods we consider. An exception is when the disturbances in the SETAR model come from a highly asymmetric distribution, when a Bootstrap method is to be preferred.An empirical application calculates multi-period forecasts from a SETAR model of US gross national product using a number of the forecasting methods. We find that whether there are improvements in forecast performance relative to a linear AR model depends on the historical epoch we select, and whether forecasts are evaluated conditional on the regime the process was in at the time the forecast was made.     

Benchmarking robustness of load forecasting models under data integrity attacks

As the internet’s footprint continues to expand, cybersecurity is becoming a major concern for both governments and the private sector. One such cybersecurity issue relates to data integrity attacks. This paper focuses on the power industry, where the forecasting processes rely heavily on the quality of the data. Data integrity attacks are expected to harm the performances of forecasting systems, which will have a major impact on both the financial bottom line of power companies and the resilience of power grids. This paper reveals the effect of data integrity attacks on the accuracy of four representative load forecasting models (multiple linear regression, support vector regression, artificial neural networks, and fuzzy interaction regression). We begin by simulating some data integrity attacks through the random injection of some multipliers that follow a normal or uniform distribution into the load series. Then, the four aforementioned load forecasting models are used to generate one-year-ahead ex post point forecasts in order to provide a comparison of their forecast errors. The results show that the support vector regression model is most robust, followed closely by the multiple linear regression model, while the fuzzy interaction regression model is the least robust of the four. Nevertheless, all four models fail to provide satisfying forecasts when the scale of the data integrity attacks becomes large. This presents a serious challenge to both load forecasters and the broader forecasting community: the generation of accurate forecasts under data integrity attacks. We construct our case study using the publicly-available data from Global Energy Forecasting Competition 2012. At the end, we also offer an overview of potential research topics for future studies.     

Forecasts of US short-term interest rates: A flexible forecast combination approach

This paper develops a flexible approach to combine forecasts of future spot rates with forecasts from time-series models or macroeconomic variables. We find empirical evidence that, accounting for both regimes in interest rate dynamics, and combining forecasts from different models, helps improve the out-of-sample forecasting performance for US short-term rates. Imposing restrictions from the expectations hypothesis on the forecasting model are found to help at long forecasting horizons.     

Forecasting Errors: the Track Record

This Briefing Paper is the last of a series of three about forecasting. In this one we examine our forecasting record; it complements the February paper in which we analysed the properties of our forecasting model in terms of the error bands attached to the central forecast.
There are many ways of measuring forecasting errors, and in the first part of this Briefing Paper we describe briefing how we have tackled the problem. (A more detailed analysis can be found in the Appendix.) In Part II we report and comment upon the errors in our forecasts of annual growth rates and show how our forecasting performance has improved over the years. In Part III we focus on quarterly forecasts up to 8 quarters ahead, and compare our forecasting errors with measurement errors in the oficial statistics; with the estimation errors built into our forecast equations; and with the stochastic model errors we reported last February. A brief summary of the main conclusions is given below.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

BAYESIAN MODEL SELECTION AND FORECASTING IN NONCAUSAL AUTOREGRESSIVE MODELS

In this paper, we propose a Bayesian estimation and forecasting procedure for noncausal autoregressive (AR) models. Specifically, we derive the joint posterior density of the past and future errors and the parameters, yielding predictive densities as a by‐product. We show that the posterior model probabilities provide a convenient model selection criterion in discriminating between alternative causal and noncausal specifications. As an empirical application, we consider US inflation. The posterior probability of noncausality is found to be high—over 98%. Furthermore, the purely noncausal specifications yield more accurate inflation forecasts than alternative causal and noncausal AR models. Copyright © 2010 John Wiley & Sons, Ltd.     

Predicting recessions with a composite real-time dynamic probit model

In this paper we propose a composite indicator for real-time recession forecasting based on alternative dynamic probit models. For this purpose, we use a large set of monthly macroeconomic and financial leading indicators from the German and US economies. Alternative dynamic probit regressions are specified through automated general-to-specific and specific-to-general lag selection procedures on the basis of slightly different initial sets. The resulting recession probability forecasts are then combined in order to decrease the volatility of the forecast errors and increase their forecasting accuracy. This procedure features not only good in-sample forecast statistics, but also good out-of-sample performances, as is illustrated using a real-time evaluation exercise.     

A Bradley-Terry type model for forecasting tennis match results

The paper introduces a model for forecasting match results for the top tier of men’s professional tennis, the ATP tour. Employing a Bradley-Terry type model, and utilising the data available on players’ past results and the surface of the contest, we predict match winners for the coming week’s matches, having updated the model parameters to take the previous week’s results into account. We compare the model to two logit models: one using official rankings and another using the official ranking points of the two competing players. Our model provides superior forecasts according to each of five criteria measuring the predictive performance, two of which relate to betting returns.     

Population forecasting theory,methods and assessments of accuracy. The magnitude of error due to different vital processes in population forecasts

The accuracy of population forecasts depends in part upon the method chosen for forecasting the vital rates of fertility, mortality, and migration. Methods for handling the stochastic propagation of error calculations in demographic forecasting are hard to do precisely. This paper discusses this obstacle in stochastic cohort-component population forecasts. The uncertainty of forecasts is due to uncertain estimates of the jump-off population and to errors in the forecasts of the vital rates. Empirically based of each source are presented and propagated through a simplified analytical model of population growth that allows assessment of the role of each component in the total error. Numerical estimates based on the errors of an actual vector ARIMA forecast of the US female population. These results broadly agree with those of the analytical model. This work especially uncertainty in the fertility forecasts to be so much higher than that in the other sources that the latter can be ignored in the propagation of error calculations for those cohorts that are born after the jump-off year of the forecast. A methodology is therefore presented which far simplifies the propagation of error calculations. It is noted, however, that the uncertainty of the jump-off population, migration, and mortality in the propagation of error for those alive at the jump-off time of the forecast must still be considered.     

INCOME PERSISTENCE AND MACRO-POLICY FEEDBACKS IN THE US*

This paper analyses and forecasts annual time series of aggregate real income per head in the US. The approach integrates elements from recent univariate time series analyses with multi-equation macromodels in which policy feedback rules have been endogenized. The main conclusions are as follows. Firstly, aggregate real per capita income is subject to significant trend reversion. This conclusion comes through more clearly by examining the data at an annual rather than the more usual quarterly frequency and by incorporating multivariate economic content in the income process. Secondly, there is significant evidence for the Lucas (1976) or Haavelmo (1944) critique: in the US there appears to have been a shift in the structural macropolicy reaction function causing a corresponding shift in the reduced form income forecasting equation. This is associated with increased concern in the late 1980's over the size of US budget deficits. Thirdly, with the above proviso, useful real income forecasts can be made as far as three years ahead. Finally, the paper provides empirical evidence for the effectiveness of monetary policy on real output or income. The change in the short-term interest rate is highly significant in forecasting income growth up to three years after the change.     

Accuracy of German federal election forecasts, 2013 & 2017

The present study reviews the accuracy of four methods (polls, prediction markets, expert judgment, and quantitative models) for forecasting the two German federal elections in 2013 and 2017. On average across both elections, polls and prediction markets were most accurate, while experts and quantitative models were least accurate. However, the accuracy of individual forecasts did not correlate across elections. That is, the methods that were most accurate in 2013 did not perform particularly well in 2017. A combined forecast, calculated by averaging forecasts within and across methods, was more accurate than three of the four component forecasts. The results conform to prior research on US presidential elections in showing that combining is effective in generating accurate forecasts and avoiding large errors.     

Comparisons of aggregate variable forecasts using aggregate and disaggregate models

When forecasting aggregate variables, a choice must often be made to either add up individual forecasts made at a disaggregate level or to simply forecast at the aggregate level. The presence of heterogeneity introduces aggregation bias and makes the disaggregates approach more preferable, while the presence of data and specification errors introduces relatively large variances in the disaggregate forecasts, making the aggregate approach more preferable. It is suggested that the mean square error is useful in evaluating the combined effects of heterogeneity and specification and data errors, and in facilitating comparisons between aggregate and disaggregate approaches to aggregate variable forecasting.     

Validity of climate change forecasting for public policy decision making

Policymakers need to know whether prediction is possible and, if so, whether any proposed forecasting method will provide forecasts that are substantially more accurate than those from the relevant benchmark method. An inspection of global temperature data suggests that temperature is subject to irregular variations on all relevant time scales, and that variations during the late 1900s were not unusual. In such a situation, a “no change” extrapolation is an appropriate benchmark forecasting method. We used the UK Met Office Hadley Centre’s annual average thermometer data from 1850 through 2007 to examine the performance of the benchmark method. The accuracy of forecasts from the benchmark is such that even perfect forecasts would be unlikely to help policymakers. For example, mean absolute errors for the 20- and 50-year horizons were 0.18  C and 0.24  C respectively. We nevertheless demonstrate the use of benchmarking with the example of the Intergovernmental Panel on Climate Change’s 1992 linear projection of long-term warming at a rate of 0.03  C per year. The small sample of errors from ex ante projections at 0.03  C per year for 1992 through 2008 was practically indistinguishable from the benchmark errors. Validation for long-term forecasting, however, requires a much longer horizon. Again using the IPCC warming rate for our demonstration, we projected the rate successively over a period analogous to that envisaged in their scenario of exponential CO₂ growth—the years 1851 to 1975. The errors from the projections were more than seven times greater than the errors from the benchmark method. Relative errors were larger for longer forecast horizons. Our validation exercise illustrates the importance of determining whether it is possible to obtain forecasts that are more useful than those from a simple benchmark before making expensive policy decisions.     

The accuracy of long-term earnings forecasts in the electric utility industry

This paper examines the accuracy of various methods of forecasting long-term earnings growth for firms in the electric utility industry. In addition to a number of extrapolative techniques, Value Line analyst forecasts are also evaluated. Value Line analyst forecasts for a five-year time horizon are found to be superior to many of the extrapolative models. Among the extrapolative models examined, implied growth and historical book value per share growth rate models performed best. These results provide strong support for using Value Line growth forecasts in cost of capital estimates for electric utilities in the context of utility rate cases. Value Line forecast errors could be explained by changes in dividend payout ratios, the firm's regulatory environment and bond rating changes.     

VAR,ERROR CORRECTION AND PRETEST FORECASTS AT LONG HORIZONS

This paper focuses on the construction of forecasts over long horizons where a typical long-horizon forecast might span four years using 20 to 40 years’ data. It is argued that the presence of persistence in the form of unit or near-unit autoregressive roots poses substantial difficulties for long-horizon interval and point forecasting. These difficulties may not be overcome even by efficient pre-testing or model-selection procedures and might, in general, lead to point forecasts with large asymptotic root mean squared errors and undesirably wide prediction intervals.     

Density Forecasts With Midas Models

We propose a parametric block wild bootstrap approach to compute density forecasts for various types of mixed‐data sampling (MIDAS) regressions. First, Monte Carlo simulations show that predictive densities for the various MIDAS models derived from the block wild bootstrap approach are more accurate in terms of coverage rates than predictive densities derived from either a residual‐based bootstrap approach or by drawing errors from a normal distribution. This result holds whether the data‐generating errors are normally independently distributed, serially correlated, heteroskedastic or a mixture of normal distributions. Second, we evaluate density forecasts for quarterly US real output growth in an empirical exercise, exploiting information from typical monthly and weekly series. We show that the block wild bootstrapping approach, applied to the various MIDAS regressions, produces predictive densities for US real output growth that are well calibrated. Moreover, relative accuracy, measured in terms of the logarithmic score, improves for the various MIDAS specifications as more information becomes available. Copyright © 2016 John Wiley & Sons, Ltd.     

Leading Indicators for Euro‐area Inflation and GDP Growth*

In this paper, we evaluate the role of a set of variables as leading indicators for Euro‐area inflation and GDP growth. Our leading indicators are taken from the variables in the European Central Bank's (ECB) Euro‐area‐wide model database, plus a set of similar variables for the US. We compare the forecasting performance of each indicator ex post with that of purely autoregressive models. We also analyse three different approaches to combining the information from several indicators. First, ex post, we discuss the use as indicators of the estimated factors from a dynamic factor model for all the indicators. Secondly, within an ex ante framework, an automated model selection procedure is applied to models with a large set of indicators. No future information is used, future values of the regressors are forecast, and the choice of the indicators is based on their past forecasting records. Finally, we consider the forecasting performance of groups of indicators and factors and methods of pooling the ex ante single‐indicator or factor‐based forecasts. Some sensitivity analyses are also undertaken for different forecasting horizons and weighting schemes of forecasts to assess the robustness of the results.