Testing the value of directional forecasts in the presence of serial correlation

Common approaches to testing the economic value of directional forecasts are based on the classical χ²${\chi }^2$-test for independence, Fisher’s exact test or the Pesaran and Timmermann test for market timing. These tests are asymptotically valid for serially independent observations, but in the presence of serial correlation they are markedly oversized, as has been confirmed in a simulation study. We therefore summarize robust test procedures for serial correlation and propose a bootstrap approach, the relative merits of which we illustrate by means of a Monte Carlo study. Our evaluations of directional predictions of stock returns and changes in Euribor rates demonstrate the importance of accounting for serial correlation in economic time series when making such predictions.     

Some theoretical results on forecast combinations

This paper proposes a framework for the analysis of the theoretical properties of forecast combination, with the forecast performance being measured in terms of mean squared forecast errors (MSFE). Such a framework is useful for deriving all existing results with ease. In addition, it also provides insights into two forecast combination puzzles. Specifically, it investigates why a simple average of forecasts often outperforms forecasts from single models in terms of MSFEs, and why a more complicated weighting scheme does not always perform better than a simple average. In addition, this paper presents two new findings that are particularly relevant in practice. First, the MSFE of a forecast combination decreases as the number of models increases. Second, the conventional approach to the selection of optimal models, based on a simple comparison of MSFEs without further statistical testing, leads to a biased selection.     

A textual analysis of Bank of England growth forecasts

The Bank of England publishes a quarterly Inflation Report that provides numerical forecasts and a text discussion of its assessment of the UK economy. Previous research has evaluated the quantitative forecasts that are included in these reports, but we focus on the qualitative discussion of output growth, by using an in-sample textual analysis procedure to convert these qualitative assessments into a score for each report over the period 2005–2014. We also construct out-of-sample scores for reports before and after this period. We then compare the scores both to real-time output growth data and to the corresponding quantitative projections published by the bank. We find that overall developments in the UK economy were represented accurately in the text of the Inflation Report. Furthermore, efficiency regressions suggest that there is information in the text that could improve the Bank of England’s quantitative nowcasts and one-quarter-ahead forecasts.     

Medium term growth forecasts: Experts vs. simple models

We compare the medium-term GDP growth forecasts generated by experts to those generated by simple models. This study analyzes a large set of forecasts that covers 48 countries from 1997 to 2016. Out-of-sample exercises indicate that no noticeable difference in performance is observed for advanced economies. In contrast, in the case of emerging economies, model forecasts perform better than expert forecasts. In addition, similar patterns are found for a collection of forecasts from a different set of experts, which suggests that the reported regularity is prevalent. Further analyses suggest that the documented difference in performance can be explained by an optimism bias, excessive reactions to innovations in growth trajectories, and insufficient responses to the information contained in the current account balance.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

The uncertainty in extreme risk forecasts from covariate-augmented volatility models

For a GARCH-type volatility model with covariates, we derive asymptotically valid forecast intervals for risk measures, such as the Value-at-Risk or Expected Shortfall. To forecast these, we use estimators from extreme value theory. In the volatility model, we allow for leverage effects and the inclusion of exogenous variables, e.g., volatility indices or high-frequency volatility measures. In simulations, we find coverage of the forecast intervals to be adequate for sufficiently extreme risk levels and sufficiently large samples, which is consistent with theory. Finally, we investigate if covariate information from volatility indices or high-frequency data improves risk forecasts for major US stock indices. While—in our framework—volatility indices appear to be helpful in this regard, intra-day data are not.     

Comparing density forecasts in a risk management context

We compare multivariate and univariate approaches to assessing the accuracy of competing density forecasts of a portfolio return in the downside part of the support. We argue that the common practice of performing multivariate forecast comparisons can be problematic in the context of assessing portfolio risk, since better multivariate forecasts do not necessarily correspond to better aggregate portfolio return forecasts. This is illustrated by examples that involve (skew) elliptical distributions and an application to daily returns of a number of US stock prices. In addition, time-varying test statistics and Value-at-Risk forecasts provide empirical evidence of regime changes over the last decades.     

A new method to assess the degree of information rigidity using fixed-event forecasts

We propose a new method to explore the information content of fixed-event forecasts and estimate structural parameters that are keys to sticky and noisy information models. Estimation follows a regression-based framework in which estimated coefficients map one-to-one with parameters that measure the degree of information rigidity. The statistical characterization of regression errors explores the laws that govern expectation formation under sticky and noisy information, that is, they are coherent with the theory. This strategy is still unexplored in the literature and potentially enhances the reliability of inference results. The method also allows linking estimation results to the signal-to-noise ratio, an important parameter of noisy information models. This task cannot be accomplished if one adopts an “agnostic” characterization of regression errors. With regard to empirical results, they show a substantial degree of information rigidity in the countries studied. They also suggest that the theoretical characterization of regression errors yields a more conservative picture of the uncertainty surrounding parameter estimates.     

Does age structure forecast economic growth?

Increases in the proportion of the working age population can yield a “demographic dividend” that enhances the rate of economic growth. We estimate the parameters of an economic growth model using a cross section of countries over the period 1960 to 1980, and investigate whether the inclusion of age structure improves the model's forecasts for the period 1980 to 2000. We find that including the age structure improves the forecast, although there is evidence of parameter instability between periods with an unexplained growth slowdown in the second period. We use the model to generate growth forecasts for the period 2000 to 2020.     

Evaluating the conditionality of judgmental forecasts

We propose a framework for evaluating the conditionality of forecasts. The crux of our framework is the observation that a forecast is conditional if revisions to the conditioning factor are incorporated faithfully into the remainder of the forecast. We consider whether the Greenbook, Blue Chip survey and Survey of Professional Forecasters exhibit systematic biases in the manner in which they incorporate interest rate projections into the forecasts of other macroeconomic variables. We do not find strong evidence of systematic biases in the three economic forecasts that we consider, as the interest rate projections in these forecasts appear to be incorporated efficiently into the forecasts of other economic variables.     

Forecast-error-based estimation of forecast uncertainty when the horizon is increased

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC.     

Diagnostics for evaluating the value and rationality of economic forecasts

A number of studies have sought to determine whether economic forecasts had predictive value. These analyses used a single statistical methodology based on the independence of the actual and predicted changes. This paper questions whether the observed results are robust if alternative statistical methodologies are used to analyze this question. Procedures suggested by Cumby and Modest as well as rationality tests were applied to two data sets. Sometimes the conclusions differ depending on the procedures that are used. The results yield a guideline for the diagnostics that should be employed in testing for the value of economic forecasts.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

Takeover prediction using forecast combinations

The ability to identify likely takeover targets at an early stage should provide investors with valuable information, enabling them to profit by investing in potential target firms. In this paper we contribute to the takeover forecasting literature by suggesting the combination of probability forecasts as an alternative method of improving the forecast accuracy in takeover prediction and realizing improved economic returns from portfolios made up of predicted targets. Forecasts from several non-linear forecasting models, such as logistic and neural network models and a combination of them, are used to determine the methodology that best reduces the out-of-sample misclassification error. We draw two general conclusions from our results. First, the forecast combination method outperforms the single models, and should therefore be used to improve the accuracy of takeover target predictions. Second, we demonstrate that an investment in a portfolio of the combined predicted targets results in significant abnormal returns being made by an investor, in the order of up to double the market benchmark return when using a portfolio of manageable size.     

Investigating the inefficiency of the CBO’s budgetary projections

This paper evaluates the efficiency of budgetary projections issued by the Congressional Budget Office (CBO) from 1984 to 2016 and investigates the cause of their inefficiency. The efficiency of both its budgetary and macroeconomic projections is rejected in many cases, especially for revenue projections. A comparison of forecast evaluations suggests that the inefficiency of revenue projections is likely due to the inefficiency of the underlying macroeconomic projections. By adjusting budgetary projections by the CBO in real time using macroeconomic forecasts by the Survey of Professional Forecasters, the accuracy of the former projections can be significantly improved in some cases, by up to 26% in terms of the root-mean-square prediction error.     

Efficient estimation of forecast uncertainty based on recent forecast errors

Multi-step-ahead forecasts of the forecast uncertainty of an individual forecaster are often based on the horizon-specific sample means of his recent squared forecast errors, where the number of past forecast errors available decreases one-to-one with the forecast horizon. In this paper, the efficiency gains from the joint estimation of forecast uncertainty for all horizons in such samples are investigated. If the forecast uncertainty is estimated by seemingly unrelated regressions, it turns out that the covariance matrix of the squared forecast errors does not have to be estimated, but simply needs to have a certain structure, which is a very useful property in small samples. Considering optimal and non-optimal forecasts, it is found that the efficiency gains can be substantial for longer horizons in small samples. The superior performance of the seemingly-unrelated-regressions approach is confirmed in several empirical applications.     

GDP forecasts: Informational asymmetry of the SPF and FOMC minutes

This paper emphasizes asymmetric information about the U.S. economy between the FOMC and SPF. Following Stekler and Symington (2016), it extends their text-based FOMC minutes index (FMI) of economic outlook to 1986-2016. Following Ericsson (2016), it employs truncation adjustment indicators and reinterprets the FMI calibrations as the policy-makers forecasts of the GDP growth, which carry information about the staff Greenbook forecasts prepared prior to the bi-quarterly FOMC meetings. Tests confirm unbiasedness and rationality of these forecasts. The encompassing tests indicate that both the FMI and SPF forecasts contain unique information beyond their alternative’s information set and can be weighted equally. The orthogonality tests suggest that the SPF efficiently use all their information set but could gain if the FOMC minutes were published without a lag, while the policy-makers rely mostly on their projections made earlier in the meetings, and could benefit from incorporating the already published SPF forecasts.     

Comparing the accuracy of multivariate density forecasts in selected regions of the copula support

This paper develops a testing framework for comparing the predictive accuracy of competing multivariate density forecasts with different predictive copulas, focusing on specific parts of the copula support. The tests are framed in the context of the Kullback–Leibler Information Criterion, using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties for realistic sample sizes. In an empirical application to daily changes of yields on government bonds of the G7 countries we obtain insights into why the Student-t and Clayton mixture copula outperforms the other copulas considered; mixing in the Clayton copula with the t-copula is of particular importance to obtain high forecast accuracy in periods of jointly falling yields.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.