Raiders of the lost high-frequency forecasts: New data and evidence on the efficiency of the Fed's forecasting

We introduce a new dataset of real gross domestic product (GDP) growth and core personal consumption expenditures (PCE) inflation forecasts produced by the staff of the Board of Governors of the Federal Reserve System. In contrast to the eight Greenbook forecasts a year the staff produce for Federal Open Market Committee (FOMC) meetings, our dataset has roughly weekly forecasts. We use these data to study whether the staff forecasts efficiently. Prespecified regressions of forecast errors on forecast revisions show the staff's GDP forecasts exhibit time-varying inefficiency between FOMC meetings, and also show some evidence for inefficient inflation forecasts.     

Evaluating FOMC forecasts

Monetary policy outcomes have improved since the early 1980s. One factor contributing to the improvement is that Federal Reserve policymakers began reporting economic forecasts to Congress in 1979. These forecasts indicate what the Federal Open Market Committee (FOMC) members think will be the likely consequence of their policies. We evaluate the accuracy of the FOMC forecasts relative to private sector forecasts, the forecasts of the Research Staff at the Board of Governors, and a naïve alternative. We find that the FOMC output forecasts were better than the naïve model and at least as good as those of the private sector and the Fed staff. The FOMC inflation forecasts were more accurate than the private sector forecasts and the naïve model; for the period ending in 1996, however, they were not as accurate as Fed staff inflation forecasts.     

How accurate are government forecasts of economic fundamentals? The case of Taiwan

A government’s ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.In this paper, we develop a methodology for evaluating non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that the undocumented knowledge of the Taiwanese government reduces forecast errors substantially.     

Analyzing fixed-event forecast revisions

It is common practice to evaluate fixed-event forecast revisions in macroeconomics by regressing current forecast revisions on one-period lagged forecast revisions. Under weak-form (forecast) efficiency, the correlation between the current and one-period lagged revisions should be zero. The empirical findings in the literature suggest that this null hypothesis of zero correlation is rejected frequently, and the correlation can be either positive (which is widely interpreted in the literature as “smoothing”) or negative (which is widely interpreted as “over-reacting”). We propose a methodology for interpreting such non-zero correlations in a straightforward and clear manner. Our approach is based on the assumption that numerical forecasts can be decomposed into both an econometric model and random expert intuition. We show that the interpretation of the sign of the correlation between the current and one-period lagged revisions depends on the process governing intuition, and the current and lagged correlations between intuition and news (or shocks to the numerical forecasts). It follows that the estimated non-zero correlation cannot be given a direct interpretation in terms of either smoothing or over-reaction.     

The Influence and Policy Signalling Role of FOMC Forecasts

Federal Open Market Committee (FOMC) policymakers have published macroeconomic forecasts since 1979 and we examine the effects of FOMC inflation forecasts using a structural VAR model. First, we assess whether they influence private inflation expectations. Second, we investigate the underlying mechanism at work and whether they convey policy signals. We provide original evidence that FOMC inflation forecasts influence private ones. We also find that the influencing effect of FOMC forecasts does not come through current Fed rate changes, that FOMC forecasts affect private expectations in a different way than current policy decisions, and that FOMC forecasts are informative about future Fed rate movements.     

Forecasting the daily federal funds rate

This paper compares two alternative one-day-ahead forecasts of tomorrow's federal funds rate. The first forecast is a simple random walk forecast in which the forecast of tomorrow's federal funds rate is taken to be today's federal funds rate. The second forecast is an ARIMA model forecast that was allowed to vary with changes in the Federal Reserve System's operating procedures. These two forecasts are compared in terms of their general forecast accuracy and the decision support they provide to a financial institution hypothesized to be borrowing $7 million a week in the federal funds market. Even in cases felt to be most favorable to the ARIMA forecasts, the degree of forecast accuracy and decision support superiority of the ARIMA forecasts is found to be quite small.     

Gauging the uncertainty of the economic outlook using historical forecasting errors: The Federal Reserve’s approach

The Federal Open Market Committee (FOMC) of the U.S. Federal Reserve regularly publishes participants’ qualitative assessments of forecast uncertainty, expressed relative to that seen on average in the past. The benchmarks used for these historical comparisons are the average root mean squared forecast errors (RMSEs) made by various private and government forecasters over the past twenty years. This paper documents how these benchmarks are constructed and discusses some of their properties. We draw several conclusions. First, if past performance is a reasonable guide to future accuracy, considerable uncertainty surrounds macroeconomic projections. Second, different forecasters have similar accuracy. Third, estimates of uncertainty about future real activity and interest rates are now considerably greater than prior to the financial crisis; in contrast, estimates of inflation accuracy have changed little. Finally, fan charts, constructed under certain assumptions and viewed in conjunction with the FOMC’s qualitative assessments, provide a reasonable approximation to future uncertainty.     

Forecasting United States–Asia international message telephone service

This study compares forecasts of US international message telephone service (IMTS) traffic using several relative mean squared error statistics. The forecasts are obtained from time-series extrapolation, univariate autoregressive integrated moving average (ARIMA), error correction and vector autoregressive models. The models are estimated on annual US IMTS outgoing traffic data for six US–Asia bilateral markets for the period 1964 to 1993. No single approach provides best forecasts. However, forecast evaluation statistics indicate that econometric models generally outperform the alternatives.     

Forecasting macroeconomic time series: LASSO-based approaches and their forecast combinations with dynamic factor models

In a data-rich environment, forecasting economic variables amounts to extracting and organizing useful information from a large number of predictors. So far, the dynamic factor model and its variants have been the most successful models for such exercises. In this paper, we investigate a category of LASSO-based approaches and evaluate their predictive abilities for forecasting twenty important macroeconomic variables. These alternative models can handle hundreds of data series simultaneously, and extract useful information for forecasting. We also show, both analytically and empirically, that combing forecasts from LASSO-based models with those from dynamic factor models can reduce the mean square forecast error (MSFE) further. Our three main findings can be summarized as follows. First, for most of the variables under investigation, all of the LASSO-based models outperform dynamic factor models in the out-of-sample forecast evaluations. Second, by extracting information and formulating predictors at economically meaningful block levels, the new methods greatly enhance the interpretability of the models. Third, once forecasts from a LASSO-based approach are combined with those from a dynamic factor model by forecast combination techniques, the combined forecasts are significantly better than either dynamic factor model forecasts or the naïve random walk benchmark.     

One model and various experts: Evaluating Dutch macroeconomic forecasts

The Netherlands Bureau for Economic Policy Analysis (CPB) uses a large macroeconomic model to create forecasts of various important macroeconomic variables. The outcomes of this model are usually filtered by experts, and it is the expert forecasts that are made available to the general public. In this paper we re-create the model forecasts for the period 1997-2008 and compare the expert forecasts with the pure model forecasts. Our key findings from the first time that this unique database has been analyzed are that (i) experts adjust upwards more often; (ii) expert adjustments are not autocorrelated, but their sizes do depend on the value of the model forecast; (iii) the CPB model forecasts are biased for a range of variables, but (iv) at the same time, the associated expert forecasts are more often unbiased; and that (v) expert forecasts are far more accurate than the model forecasts, particularly when the forecast horizon is short. In summary, the final CPB forecasts de-bias the model forecasts and lead to higher accuracies than the initial model forecasts.     

Expert opinion versus expertise in forecasting

Expert opinion is an opinion given by an expert, and it can have significant value in forecasting key policy variables in economics and finance. Expert forecasts can either be expert opinions, or forecasts based on an econometric model. An expert forecast that is based on an econometric model is replicable, and can be defined as a replicable expert forecast (REF), whereas an expert opinion that is not based on an econometric model can be defined as a non-replicable expert forecast (Non-REF). Both REF and Non-REF may be made available by an expert regarding a policy variable of interest. In this paper, we develop a model to generate REF, and compare REF with Non-REF. A method is presented to compare REF and Non-REF using efficient estimation methods, and a direct test of expertise on expert opinion is given. The latter serves the purpose of investigating whether expert adjustment improves the model-based forecasts. Illustrations for forecasting pharmaceutical stock keeping unit (SKUs), where the econometric model is of (variations of) the autoregressive integrated moving average model (ARIMA) type, show the relevance of the new methodology proposed in the paper. In particular, experts possess significant expertise, and expert forecasts are significant in explaining actual sales.     

Linear models, smooth transition autoregressions, and neural networks for forecasting macroeconomic time series: A re-examination

In this paper, we examine the forecast accuracy of linear autoregressive, smooth transition autoregressive (STAR), and neural network (NN) time series models for 47 monthly macroeconomic variables of the G7 economies. Unlike previous studies that typically consider multiple but fixed model specifications, we use a single but dynamic specification for each model class. The point forecast results indicate that the STAR model generally outperforms linear autoregressive models. It also improves upon several fixed STAR models, demonstrating that careful specification of nonlinear time series models is of crucial importance. The results for neural network models are mixed in the sense that at long forecast horizons, an NN model obtained using Bayesian regularization produces more accurate forecasts than a corresponding model specified using the specific-to-general approach. Reasons for this outcome are discussed.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Assessing the uncertainty in central banks’ inflation outlooks

Recent research has found that macroeconomic survey forecasts of uncertainty exhibit several deficiencies, such as horizon-dependent biases and lower levels of accuracy than simple unconditional uncertainty forecasts. We examine the inflation uncertainty forecasts from the Bank of England, the Banco Central do Brasil, the Magyar Nemzeti Bank and the Sveriges Riksbank to assess whether central banks’ uncertainty forecasts might be subject to similar problems. We find that, while most central banks’ uncertainty forecasts also tend to be underconfident at short horizons and overconfident at longer horizons, they are mostly not significantly biased. Moreover, they tend to be at least as precise as unconditional uncertainty forecasts from two different approaches.     

Asymmetry in unemployment rate forecast errors

Asymmetries in unemployment dynamics have been observed in the time series of a number of countries, including the United States. This paper studies asymmetries in unemployment rate forecast errors. We consider conditions under which optimal forecasts will display asymmetrically-distributed errors and how the degree of asymmetry might vary with the forecast horizon. Using data from the U.S. Survey of Professional Forecasters and the Federal Reserve Greenbook, we find substantial evidence of forecast error asymmetry, which tends to increase with the forecast horizon; we also find noteworthy differences in forecasts from these two sources. The results give insight into the abilities of professional forecasters to adapt their forecasts to asymmetry in underlying processes.     

Predicting economic contractions and expansions with the aid of professional forecasts

Traditional econometric models of economic contractions typically perform poorly in forecasting exercises. This criticism is also frequently levelled at professional forecast probabilities of contractions. This paper addresses the problem of incorporating the entire distribution of professional forecasts into an econometric model for forecasting contractions and expansions. A new augmented probit approach is proposed, involving the transformation of the distribution of professional forecasts into a ‘professional forecast’ prior for the economic data underlying the probit model. Since the object of interest is the relationship between the distribution of professional forecasts and the probit model’s economic-data dependent parameters, the solution avoids criticisms levelled at the accuracy of professional forecast based point estimates of contractions. An application to US real GDP data shows that the model yields significant forecast improvements relative to alternative approaches.     

Combining forecasts: Performance and coherence

There is general agreement in many forecasting contexts that combining individual predictions leads to better final forecasts. However, the relative error reduction in a combined forecast depends upon the extent to which the component forecasts contain unique/independent information. Unfortunately, obtaining independent predictions is difficult in many situations, as these forecasts may be based on similar statistical models and/or overlapping information. The current study addresses this problem by incorporating a measure of coherence into an analytic evaluation framework so that the degree of independence between sets of forecasts can be identified easily. The framework also decomposes the performance and coherence measures in order to illustrate the underlying aspects that are responsible for error reduction. The framework is demonstrated using UK retail prices index inflation forecasts for the period 1998–2014, and implications for forecast users are discussed.     

Using forecast evaluation to improve the accuracy of the Greenbook forecast

Recently, Patton and Timmermann (2012) proposed a more powerful kind of forecast efficiency regression at multiple horizons, and showed that it provides evidence against the efficiency of the Fed’s Greenbook forecasts. I use their forecast efficiency evaluation to propose a method for adjusting the Greenbook forecasts. Using this method in a real-time out-of-sample forecasting exercise, I find that it provides modest improvements in the accuracies of the forecasts for the GDP deflator and CPI, but not for other variables. The improvements are statistically significant in some cases, with magnitudes of up to 18% in root mean square prediction error.     

Forecast-error-based estimation of forecast uncertainty when the horizon is increased

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC.     

Forecast combination through dimension reduction techniques

This paper considers several methods of producing a single forecast from several individual ones. We compare “standard” but hard to beat combination schemes (such as the average of forecasts at each period, or consensus forecast and OLS-based combination schemes) with more sophisticated alternatives that involve dimension reduction techniques. Specifically, we consider principal components, dynamic factor models, partial least squares and sliced inverse regression.Our source of forecasts is the Survey of Professional Forecasters, which provides forecasts for the main US macroeconomic aggregates. The forecasting results show that partial least squares, principal component regression and factor analysis have similar performances (better than the usual benchmark models), but sliced inverse regression shows an extreme behavior (performs either very well or very poorly).