Evaluating Point and Density Forecasts of DSGE Models

This paper investigates the accuracy of forecasts from four dynamic stochastic general equilibrium (DSGE) models for inflation, output growth and the federal funds rate using a real‐time dataset synchronized with the Fed's Greenbook projections. Conditioning the model forecasts on the Greenbook nowcasts leads to forecasts that are as accurate as the Greenbook projections for output growth and the federal funds rate. Only for inflation are the model forecasts dominated by the Greenbook projections. A comparison with forecasts from Bayesian vector autoregressions shows that the economic structure of the DSGE models which is useful for the interpretation of forecasts does not lower the accuracy of forecasts. Combining forecasts of several DSGE models increases precision in comparison to individual model forecasts. Comparing density forecasts with the actual distribution of observations shows that DSGE models overestimate uncertainty around point forecasts. Copyright © 2013 John Wiley & Sons, Ltd.     

Does judgment improve macroeconomic density forecasts?

This paper presents empirical evidence on how judgmental adjustments affect the accuracy of macroeconomic density forecasts. Judgment is defined as the difference between professional forecasters’ densities and the forecast densities from statistical models. Using entropic tilting, we evaluate whether judgments about the mean, variance and skew improve the accuracy of density forecasts for UK output growth and inflation. We find that not all judgmental adjustments help. Judgments about point forecasts tend to improve density forecast accuracy at short horizons and at times of heightened macroeconomic uncertainty. Judgments about the variance hinder at short horizons, but can improve tail risk forecasts at longer horizons. Judgments about skew in general take value away, with gains seen only for longer horizon output growth forecasts when statistical models took longer to learn that downside risks had reduced with the end of the Great Recession. Overall, density forecasts from statistical models prove hard to beat.     

CONSTRUCTING OPTIMAL DENSITY FORECASTS FROM POINT FORECAST COMBINATIONS

Decision makers often observe point forecasts of the same variable computed, for instance, by commercial banks, IMF and the World Bank, but the econometric models used by such institutions are frequently unknown. This paper shows how to use the information available on point forecasts to compute optimal density forecasts. Our idea builds upon the combination of point forecasts under general loss functions and unknown forecast error distributions. We use real‐time data to forecast the density of US inflation. The results indicate that the proposed method materially improves the real‐time accuracy of density forecasts vis‐à‐vis those from the (unknown) individual econometric models. Copyright © 2013 John Wiley & Sons, Ltd.     

Forecasting realized volatility: a Bayesian model‐averaging approach

How to measure and model volatility is an important issue in finance. Recent research uses high‐frequency intraday data to construct ex post measures of daily volatility. This paper uses a Bayesian model‐averaging approach to forecast realized volatility. Candidate models include autoregressive and heterogeneous autoregressive specifications based on the logarithm of realized volatility, realized power variation, realized bipower variation, a jump and an asymmetric term. Applied to equity and exchange rate volatility over several forecast horizons, Bayesian model averaging provides very competitive density forecasts and modest improvements in point forecasts compared to benchmark models. We discuss the reasons for this, including the importance of using realized power variation as a predictor. Bayesian model averaging provides further improvements to density forecasts when we move away from linear models and average over specifications that allow for GARCH effects in the innovations to log‐volatility. Copyright © 2009 John Wiley & Sons, Ltd.     

Evaluating DSGE model forecasts of comovements

This paper develops and applies tools to assess multivariate aspects of Bayesian Dynamic Stochastic General Equilibrium (DSGE) model forecasts and their ability to predict comovements among key macroeconomic variables. We construct posterior predictive checks to evaluate conditional and unconditional density forecasts, in addition to checks for root-mean-squared errors and event probabilities associated with these forecasts. The checks are implemented on a three-equation DSGE model as well as the Smets and Wouters (2007) model using real-time data. We find that the additional features incorporated into the Smets–Wouters model do not lead to a uniform improvement in the quality of density forecasts and prediction of comovements of output, inflation, and interest rates.     

A bootstrap-based non-parametric forecast density

Interest in density forecasts (as opposed to solely modeling the conditional mean) arises from the possibility of dynamics in higher moments of a time series, as well as in forecasting the probability of future events in some applications. By combining the idea of Markov bootstrapping with that of kernel density estimation, this paper presents a simple non-parametric method for estimating out-of-sample multi-step density forecasts. The paper also considers a host of evaluation tests for examining the dynamic misspecification of estimated density forecasts by targeting autocorrelation, heteroskedasticity and neglected non-linearity. These tests are useful, as a rejection of the tests gives insight into ways to improve a particular forecasting model. In an extensive Monte Carlo analysis involving a range of commonly used linear and non-linear time series processes, the non-parametric method is shown to work reasonably well across the simulated models for a suitable choice of the bandwidth (smoothing parameter). Furthermore, an application of the method to the U.S. Industrial Production series provides multi-step density forecasts that show no sign of dynamic misspecification.     

Marginalized Predictive Likelihood Comparisons of Linear Gaussian State‐Space Models with Applications to DSGE,DSGE‐VAR,and VAR Models

The predictive likelihood is useful for ranking models in forecast comparison exercises using Bayesian inference. We discuss how it can be estimated, by means of marzginalization, for any subset of the observables in linear Gaussian state‐space models. We compare macroeconomic density forecasts for the euro area of a DSGE model to those of a DSGE‐VAR, a BVAR and a multivariate random walk over 1999:Q1–2011:Q4. While the BVAR generally provides superior forecasts, its performance deteriorates substantially with the onset of the Great Recession. This is particularly notable for longer‐horizon real GDP forecasts, where the DSGE and DSGE‐VAR models perform better. Copyright © 2016 John Wiley & Sons, Ltd.     

Do precious metal prices help in forecasting South African inflation?

In this paper we test whether the key metals prices of gold and platinum significantly improve inflation forecasts for the South African economy. We also test whether controlling for conditional correlations in a dynamic setup, using bivariate Bayesian-Dynamic Conditional Correlation (B-DCC) models, improves inflation forecasts. To achieve this we compare out-of-sample forecast estimates of the B-DCC model to Random Walk, Autoregressive and Bayesian VAR models. We find that for both the BVAR and BDCC models, improving point forecasts of the Autoregressive model of inflation remains an elusive exercise. This, we argue, is of less importance relative to the more informative density forecasts. For this we find improved forecasts of inflation for the B-DCC models at all forecasting horizons tested. We thus conclude that including metals price series as inputs to inflation models leads to improved density forecasts, while controlling for the dynamic relationship between the included price series and inflation similarly leads to significantly improved density forecasts.     

Can the random walk model be beaten in out-of-sample density forecasts? Evidence from intraday foreign exchange rates

It has been documented that random walk outperforms most economic structural and time series models in out-of-sample forecasts of the conditional mean dynamics of exchange rates. In this paper, we study whether random walk has similar dominance in out-of-sample forecasts of the conditional probability density of exchange rates given that the probability density forecasts are often needed in many applications in economics and finance. We first develop a nonparametric portmanteau test for optimal density forecasts of univariate time series models in an out-of-sample setting and provide simulation evidence on its finite sample performance. Then we conduct a comprehensive empirical analysis on the out-of-sample performances of a wide variety of nonlinear time series models in forecasting the intraday probability densities of two major exchange rates—Euro/Dollar and Yen/Dollar. It is found that some sophisticated time series models that capture time-varying higher order conditional moments, such as Markov regime-switching models, have better density forecasts for exchange rates than random walk or modified random walk with GARCH and Student-t innovations. This finding dramatically differs from that on mean forecasts and suggests that sophisticated time series models could be useful in out-of-sample applications involving the probability density.     

Measuring and adjusting for overconfidence

To evaluate density forecasts, the applied scoring rule is often arbitrarily chosen. The selection of the scoring rule strongly influences the ranking of forecasts. This paper identifies overconfidence as the main driver for scoring differences. A novel approach to measure overconfidence is proposed. Based on a non-proper scoring rule, the forecasts can be individually adjusted toward a calibrated forecast. Applying the adjustment procedure to the survey of professional forecasters, it can be shown that out-of-sample forecasts can be significantly improved. Also the ranking of the adjusted forecasts becomes less sensitive to the selection of scoring rules.     

Probabilistic access forecasting for improved offshore operations

Improving access is a priority in the offshore wind sector, driven by the opportunity to increase revenues, reduce costs, and improve safety at operational wind farms. This paper describes a novel method for producing probabilistic forecasts of safety-critical access conditions during crew transfers. Methods of generating density forecasts of significant wave height and peak wave period are developed and evaluated. It is found that boosted semi-parametric models outperform those estimated via maximum likelihood, as well as a non-parametric approach. Scenario forecasts of sea-state variables are generated and used as inputs to a data-driven vessel motion model, based on telemetry recorded during 700 crew transfers. This enables the production of probabilistic access forecasts of vessel motion during crew transfer up to 5 days ahead. The above methodology is implemented on a case study at a wind farm off the east coast of the UK.     

Nonparametric likelihood ratio model selection tests between parametric likelihood and moment condition models

We propose a nonparametric likelihood ratio testing procedure for choosing between a parametric (likelihood) model and a moment condition model when both models could be misspecified. Our procedure is based on comparing the Kullback–Leibler Information Criterion (KLIC) between the parametric model and moment condition model. We construct the KLIC for the parametric model using the difference between the parametric log likelihood and a sieve nonparametric estimate of population entropy, and obtain the KLIC for the moment model using the empirical likelihood statistic. We also consider multiple (>2)$(>2)$ model comparison tests, when all the competing models could be misspecified, and some models are parametric while others are moment-based. We evaluate the performance of our tests in a Monte Carlo study, and apply the tests to an example from industrial organization.     

Stress-testing US bank holding companies: A dynamic panel quantile regression approach

We propose an econometric framework for estimating capital shortfalls of bank holding companies (BHCs) under pre-specified macroeconomic scenarios. To capture the nonlinear dynamics of bank losses and revenues during periods of financial stress, we use a fixed effects quantile autoregressive (FE-QAR) model with exogenous macroeconomic covariates, an approach that delivers a superior out-of-sample forecasting performance relative to the standard linear framework. According to the out-of-sample forecasts, the realized net charge-offs during the 2007–09 crisis fall within the multi-step-ahead density forecasts implied by the FE-QAR model, but are frequently outside the density forecasts generated using the corresponding linear model. This difference reflects the fact that the linear specification substantially underestimates loan losses, especially for real estate loan portfolios. Employing the macroeconomic stress scenario used in CCAR 2012, we use the density forecasts generated by the FE-QAR model to simulate capital shortfalls for a panel of large BHCs. For almost all institutions in the sample, the FE-QAR model generates capital shortfalls that are considerably higher than those implied by its linear counterpart, which suggests that our approach has the potential to detect emerging vulnerabilities in the financial system.     

Stability in the inefficient use of forecasting systems: A case study in a supply chain company

Computer-based demand forecasting systems have been widely adopted in supply chain companies, but little research has studied how these systems are actually used in the forecasting process. We report the findings of a case study of demand forecasting in a pharmaceutical company over a 15-year period. At the start of the study, managers believed that they were making extensive use of their forecasting system that was marketed based on the accuracy of its advanced statistical methods. Yet most forecasts were obtained using the system’s facility for judgmentally overriding the automatic statistical forecasts. Carrying out the judgmental interventions involved considerable management effort as part of a sales & operations planning (S&OP) process, yet these often only served to reduce forecast accuracy. This study uses observations of the forecasting process, interviews with participants and data on the accuracy of forecasts to investigate why the managers continued to use non-normative forecasting practices for many years despite the potential economic benefits that could be achieved through change. The reasons for the longevity of these practices are examined both from the perspective of the individual forecaster and the organization as a whole.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Mixtures of t-distributions for finance and forecasting

We explore convenient analytic properties of distributions constructed as mixtures of scaled and shifted t-distributions. Particularly desirable for econometric applications are closed-form expressions for antiderivatives (e.g., the cumulative density function). We illustrate the usefulness of these distributions in two applications. In the first application, we produce density forecasts of U.S. inflation and show that these forecasts are more accurate, out-of-sample, than density forecasts obtained using normal or standard t-distributions. In the second application, we replicate the option-pricing exercise of Abadir and Rockinger [Density functionals, with an option-pricing application. Econometric Theory 19, 778–811.] and obtain comparably good results, while gaining analytical tractability.     

Dynamic quantile models

This paper introduces the Dynamic Additive Quantile (DAQ) model that ensures the monotonicity of conditional quantile estimates. The DAQ model is easily estimable and can be used for computation and updating of the Value-at-Risk. An asymptotically efficient estimator of the DAQ is obtained by maximizing an objective function based on the inverse KLIC measure. An alternative estimator proposed in the paper is the Method of L-Moments estimator (MLM). The MLM estimator is consistent, but generally not fully efficient. Goodness-of-fit tests and diagnostic tools for the assessment of the model are also provided. For illustration, the DAQ model is estimated from a series of returns on the Toronto Stock Exchange (TSX) market index.     

Modelling and forecasting stock returns: exploiting the futures market,regime shifts and international spillovers

This paper proposes a vector equilibrium correction model of stock returns that exploits the information in the futures market, while allowing for both regime‐switching behaviour and international spillovers across stock market indices. Using data for three major stock market indices since 1989, we find that: (i) in sample, our model outperforms several alternative models on the basis of standard statistical criteria; (ii) in out‐of‐sample forecasting, our model does not produce significant gains in terms of point forecasts relative to more parsimonious alternative specifications, but it does so both in terms of market timing ability and in density forecasting performance. The economic value of the density forecasts is illustrated with an application to a simple risk management exercise. Copyright © 2005 John Wiley & Sons, Ltd.     

Some observations on forecasting and policy

This paper offers some thoughts on the use of macroeconomic and financial forecasts in monetary and fiscal policy. It stresses the role of nowcasting in constructing good forecasts: most of the value added in macreoeconomic forecasts comes from getting a good approximation to the jumping-off point. Some specific applications are discussed: long-range debt/GDP projections and forecasting recessions using asset prices. I also discuss the construction and use of density forecasts.     

Diagnostics for evaluating the value and rationality of economic forecasts

A number of studies have sought to determine whether economic forecasts had predictive value. These analyses used a single statistical methodology based on the independence of the actual and predicted changes. This paper questions whether the observed results are robust if alternative statistical methodologies are used to analyze this question. Procedures suggested by Cumby and Modest as well as rationality tests were applied to two data sets. Sometimes the conclusions differ depending on the procedures that are used. The results yield a guideline for the diagnostics that should be employed in testing for the value of economic forecasts.