DSGE forecasts of the lost recovery

The years following the Great Recession were challenging for forecasters. Unlike other deep downturns, this recession was not followed by a swift recovery, but instead generated a sizable and persistent output gap that was not accompanied by deflation as a traditional Phillips curve relationship would have predicted. Moreover, the zero lower bound and unconventional monetary policy generated an unprecedented policy environment. We document the actual real-time forecasting performance of the New York Fed dynamic stochastic general equilibrium (DSGE) model during this period and explain the results using the pseudo real-time forecasting performance results from a battery of DSGE models. We find the New York Fed DSGE model’s forecasting accuracy to be comparable to that of private forecasters, and notably better for output growth than the median forecasts from the FOMC’s Summary of Economic Projections. The model’s financial frictions were key in obtaining these results, as they implied a slow recovery following the financial crisis.     

Evaluating FOMC forecasts

Monetary policy outcomes have improved since the early 1980s. One factor contributing to the improvement is that Federal Reserve policymakers began reporting economic forecasts to Congress in 1979. These forecasts indicate what the Federal Open Market Committee (FOMC) members think will be the likely consequence of their policies. We evaluate the accuracy of the FOMC forecasts relative to private sector forecasts, the forecasts of the Research Staff at the Board of Governors, and a naïve alternative. We find that the FOMC output forecasts were better than the naïve model and at least as good as those of the private sector and the Fed staff. The FOMC inflation forecasts were more accurate than the private sector forecasts and the naïve model; for the period ending in 1996, however, they were not as accurate as Fed staff inflation forecasts.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.     

Evaluating Point and Density Forecasts of DSGE Models

This paper investigates the accuracy of forecasts from four dynamic stochastic general equilibrium (DSGE) models for inflation, output growth and the federal funds rate using a real‐time dataset synchronized with the Fed's Greenbook projections. Conditioning the model forecasts on the Greenbook nowcasts leads to forecasts that are as accurate as the Greenbook projections for output growth and the federal funds rate. Only for inflation are the model forecasts dominated by the Greenbook projections. A comparison with forecasts from Bayesian vector autoregressions shows that the economic structure of the DSGE models which is useful for the interpretation of forecasts does not lower the accuracy of forecasts. Combining forecasts of several DSGE models increases precision in comparison to individual model forecasts. Comparing density forecasts with the actual distribution of observations shows that DSGE models overestimate uncertainty around point forecasts. Copyright © 2013 John Wiley & Sons, Ltd.     

Evaluating forecasts of political conflict dynamics

There is considerable interest today in the forecasting of conflict dynamics. Commonly, the root mean square error and other point metrics are used to evaluate the forecasts from such models. However, conflict processes are non-linear, so these point metrics often do not produce adequate evaluations of the calibration and sharpness of the forecast models. Forecast density evaluation improves the model evaluation. We review tools for density evaluation, including continuous rank probability scores, verification rank histograms, and sharpness plots. The usefulness of these tools for evaluating conflict forecasting models is explained. We illustrate this, first, in a comparison of several time series models’ forecasts of simulated data from a Markov-switching process, and second, in a comparison of several models’ abilities to forecast conflict dynamics in the Cross Straits. These applications show the pitfalls of relying on point metrics alone for evaluating the quality of conflict forecasting models. As in other fields, it is more useful to employ a suite of tools. A non-linear vector autoregressive model emerges as the model which is best able to forecast conflict dynamics between China and Taiwan.     

Evaluating the conditionality of judgmental forecasts

We propose a framework for evaluating the conditionality of forecasts. The crux of our framework is the observation that a forecast is conditional if revisions to the conditioning factor are incorporated faithfully into the remainder of the forecast. We consider whether the Greenbook, Blue Chip survey and Survey of Professional Forecasters exhibit systematic biases in the manner in which they incorporate interest rate projections into the forecasts of other macroeconomic variables. We do not find strong evidence of systematic biases in the three economic forecasts that we consider, as the interest rate projections in these forecasts appear to be incorporated efficiently into the forecasts of other economic variables.     

Assessing DSGE model nonlinearities

We develop a new class of time series models to identify nonlinearities in the data and to evaluate DSGE models. U.S. output growth and the federal funds rate display nonlinear conditional mean dynamics, while inflation and nominal wage growth feature conditional heteroskedasticity. We estimate a DSGE model with asymmetric wage and price adjustment costs and use predictive checks to assess its ability to account for these nonlinearities. While it is able to match the nonlinear inflation and wage dynamics, thanks to the estimated downward wage and price rigidities, these do not spill over to output growth or the interest rate.     

Evaluating quantile-bounded and expectile-bounded interval forecasts

In many different contexts, decision-making is improved by the availability of probabilistic predictions. The accuracy of probabilistic forecasting methods can be compared using scoring functions and insight provided by calibration tests. These tests evaluate the consistency of predictions with the observations. Our main agenda in this paper is interval forecasts and their evaluation. Such forecasts are usually bounded by two quantile forecasts. However, a limitation of quantiles is that they convey no information regarding the size of potential exceedances. By contrast, the location of an expectile is dictated by the whole distribution. This prompts us to propose expectile-bounded intervals. We provide interpretation, a consistent scoring function and a calibration test. Before doing this, we reflect on the evaluation of forecasts of quantile-bounded intervals and expectiles, and suggest extensions of previously proposed calibration tests in order to guard against strategic forecasting. We illustrate ideas using day-ahead electricity price forecasting.     

Evaluating density forecasts: forecast combinations,model mixtures,calibration and sharpness

This paper reviews current density forecast evaluation procedures, and considers a proposal that such procedures be augmented by an assessment of ‘sharpness’. This was motivated by an example in which some standard evaluation procedures using probability integral transforms cannot distinguish the ideal forecast from several competing forecasts. We show that this example has some unrealistic features from a time series forecasting perspective, and so provides insecure foundations for the argument that existing calibration procedures are inadequate in practice. Our alternative, more realistic example shows how relevant statistical methods, including information‐based methods, provide the required discrimination between competing forecasts. We introduce a new test of density forecast efficiency. Copyright © 2010 John Wiley & Sons, Ltd.     

Learning in an estimated medium-scale DSGE model

We evaluate the empirical relevance of learning by private agents in an estimated medium-scale DSGE model. We replace the standard rational expectations assumption in the Smets and Wouters (2007) model by a constant-gain learning mechanism. If agents know the correct structure of the model and only learn about the parameters, both expectation mechanisms produce very similar results, and only the transition dynamics that are generated by specific initial beliefs seem to improve the fit. If, instead, agents use only a reduced information set in forming the perceived law of motion, the implied model dynamics change and, depending on the specification of the initial beliefs, the marginal likelihood of the model can improve significantly. These best-fitting models add additional persistence to the dynamics and this reduces the gap between the IRFs of the DSGE model and the more data-driven DSGE-VAR model. However, the learning dynamics do not systematically alter the estimated structural parameters related to the nominal and real frictions in the DSGE model.     

Evaluating quantile forecasts in the M5 uncertainty competition

Probabilistic forecasts are necessary for robust decisions in the face of uncertainty. The M5 Uncertainty competition required participating teams to forecast nine quantiles for unit sales of various products at various aggregation levels and for different time horizons. This paper evaluates the forecasting performance of the quantile forecasts at different aggregation levels and at different quantile levels. We contrast this with some theoretical predictions, and discuss potential implications and promising future research directions for the practice of probabilistic forecasting.     

Evaluating interval forecasts of high‐frequency financial data

A number of methods of evaluating the validity of interval forecasts of financial data are analysed, and illustrated using intraday FTSE100 index futures returns. Some existing interval forecast evaluation techniques, such as the Markov chain approach of Christoffersen ( 1998 ), are shown to be inappropriate in the presence of periodic heteroscedasticity. Instead, we consider a regression‐based test, and a modified version of Christoffersen's Markov chain test for independence, and analyse their properties when the financial time series exhibit periodic volatility. These approaches lead to different conclusions when interval forecasts of FTSE100 index futures returns generated by various GARCH(1,1) and periodic GARCH(1,1) models are evaluated. Copyright © 2003 John Wiley & Sons, Ltd.     

Evaluating the accuracy of value-at-risk forecasts: New multilevel tests

We propose independence and conditional coverage tests which are aimed at evaluating the accuracy of Value-at-Risk (VaR) forecasts from the same model at different confidence levels. The proposed procedures are multilevel tests, i.e., joint tests of several quantiles corresponding to different confidence levels. In a comprehensive Monte Carlo exercise, we document the superiority of the proposed tests with respect to existing multilevel tests. In an empirical application, we illustrate the implementation of the tests using several VaR models and daily data for 15 MSCI world indices.     

On the forecasting performance of a small-scale DSGE model

Dynamic stochastic general equilibrium (DSGE) models have recently become standard tools for policy analysis. Nevertheless, their forecasting properties have still barely been explored. In this article, we address this problem by examining the quality of forecasts of the key U.S. economic variables: the three-month Treasury bill yield, the GDP growth rate and GDP price index inflation, from a small-size DSGE model, trivariate vector autoregression (VAR) models and the Philadelphia Fed Survey of Professional Forecasters (SPF). The ex post forecast errors are evaluated on the basis of the data from the period 1994–2006. We apply the Philadelphia Fed “Real-Time Data Set for Macroeconomists” to ensure that the data used in estimating the DSGE and VAR models was comparable to the information available to the SPF.Overall, the results are mixed. When comparing the root mean squared errors for some forecast horizons, it appears that the DSGE model outperforms the other methods in forecasting the GDP growth rate. However, this characteristic turned out to be statistically insignificant. Most of the SPF's forecasts of GDP price index inflation and the short-term interest rate are better than those from the DSGE and VAR models.     

Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions

Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception.In this paper, we define a formal framework for assessing the behavior of forecast evaluation procedures with respect to extreme events, which we use to demonstrate that assessment based on the expectation of a proper score is not suitable for extremes. Alternatively, we propose studying the properties of the CRPS as a random variable by using extreme value theory to address extreme event verification. An index is introduced to compare calibrated forecasts, which summarizes the ability of probabilistic forecasts for predicting extremes. The strengths and limitations of this method are discussed using both theoretical arguments and simulations.     

Professional forecasters and real-time forecasting with a DSGE model

This paper analyses the real-time forecasting performance of the New Keynesian DSGE model of Galí, Smets and Wouters (2012), estimated on euro area data. It investigates the extent to which the inclusion of forecasts of inflation, GDP growth and unemployment by professional forecasters improve the forecasting performance. We consider two approaches for conditioning on such information. Under the “noise” approach, the mean professional forecasts are assumed to be noisy indicators of the rational expectations forecasts implied by the DSGE model. Under the “news” approach, it is assumed that the forecasts reveal the presence of expected future structural shocks in line with those estimated in the past. The forecasts of the DSGE model are compared with those from a Bayesian VAR model, an AR(1) model, a sample mean and a random walk.     

Monetary policy trade-offs in an estimated open-economy DSGE model

This paper studies the trade-offs between stabilizing CPI inflation and alternative measures of the output gap in Ramses, the Riksbank׳s estimated dynamic stochastic general equilibrium (DSGE) model of a small open economy. Our main finding is that the trade-off between stabilizing CPI inflation and the output gap strongly depends on which concept of potential output in the output gap between output and potential output is used in the loss function. If potential output is defined as a smooth trend this trade-off is much more pronounced compared to the case when potential output is defined as the output level that would prevail if prices and wages were flexible.     

Evaluating factor forecasts for the UK: The role of asset prices

This paper applies a large data set, consisting of 167 monthly time series for the UK, both economic and financial, to simulate out-of-sample predictions of industrial production, inflation, 3-month Treasury Bills, and other variables. Fifteen dynamic factor models that allow forecasting based on large panels of time series are considered. The performances of these factor models are then compared to the following competing models: a simple univariate autoregressive, a vector autoregressive, a leading indicator, and a Phillips curve models. The results show that the best dynamic factor models outperform the competing models in forecasting at 6-, 12-, and 24-month horizons. Thus, the financial markets may have predictive power for the economic activity. This can be a useful tool for central banks and financial institutions, which may use the factor models to construct leading indicators of the economic conditions. In addition, researchers can see a strategic application of factor models.     

DSGE pileups

The sampling distribution of estimators for DSGE structural parameters tends to be non-normal and/or pile up on the boundary of the theoretically admissible parameter space. This calls into question both the reliability of asymptotic approximations and the presumption of correct specification. This paper seeks to develop a conceptual framework for understanding how these phenomena arise, and to provide pragmatic methods for dealing with them in practice. The results are presented in three examples and a medium scale DSGE model.     

Endogenous risk in a DSGE model with capital-constrained financial intermediaries

This paper proposes a perturbation-based approach to implement the idea of endogenous financial risk in a standard DSGE macro-model. Recent papers, such as Mendoza (2010), Brunnermeier and Sannikov (2012) and He and Krishnamurthy (2012), that have stimulated the research field on endogenous risk in a macroeconomic context, are based on sophisticated solution methods that are not easily applicable in larger models. We propose an approximation method that allows us to capture some of the basic insights of this literature in a standard macro-model. We are able to identify an important risk-channel that derives from the risk aversion of constrained intermediaries and that contributes significantly to the overall financial and macroeconomic volatility.