Measuring forecast uncertainty: A review with evaluation based on a macro model of the French economy

Five alternative techniques have been applied to measure the degree of uncertainty associated with the forecasts produced by a macro-model of the French economy, the Mini-DMS developed at INSEE. They are bootstrap, analytic simulation on coefficients, Monte Carlo on coefficients, parametric stochastic simulation and re-estimation, a residual-based procedure. Due to the complexity and the size of the model (nonlinear and with more than 200 equations), several associated technical problems had to be solved. The remarkable convergence of results which has been obtained for all the main endogenous variables suggests that forecast confidence intervals are likely to be quite reliable for this model.     

Population forecasting theory,methods and assessments of accuracy. The magnitude of error due to different vital processes in population forecasts

The accuracy of population forecasts depends in part upon the method chosen for forecasting the vital rates of fertility, mortality, and migration. Methods for handling the stochastic propagation of error calculations in demographic forecasting are hard to do precisely. This paper discusses this obstacle in stochastic cohort-component population forecasts. The uncertainty of forecasts is due to uncertain estimates of the jump-off population and to errors in the forecasts of the vital rates. Empirically based of each source are presented and propagated through a simplified analytical model of population growth that allows assessment of the role of each component in the total error. Numerical estimates based on the errors of an actual vector ARIMA forecast of the US female population. These results broadly agree with those of the analytical model. This work especially uncertainty in the fertility forecasts to be so much higher than that in the other sources that the latter can be ignored in the propagation of error calculations for those cohorts that are born after the jump-off year of the forecast. A methodology is therefore presented which far simplifies the propagation of error calculations. It is noted, however, that the uncertainty of the jump-off population, migration, and mortality in the propagation of error for those alive at the jump-off time of the forecast must still be considered.     

Analysts' utilization of historical earnings information

This study examines whether security analysts (in)efficiently utilize the information contained in past series of annual and quarterly earnings in producing earnings forecasts. To do so, it investigates whether equal-weighted combinations of security analysts' forecasts with forecasts from statistical models based on historical earnings are superior, both in terms of being a better surrogate for the market's expectations of earnings and of accuracy, to forecasts from either one of these two sources. The empirical findings indicate that, although analysts' forecasts are superior to forecasts from statistical models, performance can be improved—both in terms of accuracy and also of being a better surrogate for market earnings expectations—by combining analysts' forecasts with forecasts from statistical models based on past quarterly earnings. Improvements in proxying for market earnings expectations were obtained even when analysts' forecasts made in June of the forecast year were used in the combinations. An implication of these findings is that investors can improve their investment decisions by using an average of the mean analysts' forecasts and the forecast produced by a time-series model of quarterly earnings in their investment decisions.     

Forecasting with High‐Dimensional Panel VARs

This paper develops methods for estimating and forecasting in Bayesian panel vector autoregressions of large dimensions with time‐varying parameters and stochastic volatility. We exploit a hierarchical prior that takes into account possible pooling restrictions involving both VAR coefficients and the error covariance matrix, and propose a Bayesian dynamic learning procedure that controls for various sources of model uncertainty. We tackle computational concerns by means of a simulation‐free algorithm that relies on analytical approximations to the posterior. We use our methods to forecast inflation rates in the eurozone and show that these forecasts are superior to alternative methods for large vector autoregressions.     

How accurate are government forecasts of economic fundamentals? The case of Taiwan

A government’s ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.In this paper, we develop a methodology for evaluating non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that the undocumented knowledge of the Taiwanese government reduces forecast errors substantially.     

Stochastic population forecasts and their uses

"The properties and uses of stochastic forecasts are discussed here. For linear stochastic projections, we show how the computation of forecast moments and the statistical distribution of forecasts depend on the multiplicative and autoregressive structure of the dynamics. Both scalar and vector projection methods are discussed, and their similarities are explored. Next we discuss the uses of stochastic forecasts, arguing that it is important to relate forecasts to the specific decision-making criteria of particular forecast users. The example of [the U.S. system of] Social Security is used to show how a dynamic programming approach may be used to explore alternative decisions in a probabilistic context."     

Evaluating Point and Density Forecasts of DSGE Models

This paper investigates the accuracy of forecasts from four dynamic stochastic general equilibrium (DSGE) models for inflation, output growth and the federal funds rate using a real‐time dataset synchronized with the Fed's Greenbook projections. Conditioning the model forecasts on the Greenbook nowcasts leads to forecasts that are as accurate as the Greenbook projections for output growth and the federal funds rate. Only for inflation are the model forecasts dominated by the Greenbook projections. A comparison with forecasts from Bayesian vector autoregressions shows that the economic structure of the DSGE models which is useful for the interpretation of forecasts does not lower the accuracy of forecasts. Combining forecasts of several DSGE models increases precision in comparison to individual model forecasts. Comparing density forecasts with the actual distribution of observations shows that DSGE models overestimate uncertainty around point forecasts. Copyright © 2013 John Wiley & Sons, Ltd.     

Nowcasting GDP with a pool of factor models and a fast estimation algorithm

We propose a novel mixed-frequency dynamic factor model with time-varying parameters and stochastic volatility for macroeconomic nowcasting and develop a fast estimation algorithm. This enables us to generate forecast densities based on a large space of factor models. We apply our framework to nowcast US GDP growth in real time. Our results reveal that stochastic volatility seems to improve the accuracy of point forecasts the most, compared to the constant-parameter factor model. These gains are most prominent during unstable periods such as the Covid-19 pandemic. Finally, we highlight indicators driving the US GDP growth forecasts and associated downside risks in real time.     

Econometric Models and Economic Forecasting

This Briefing Paper is thejirst ofa series of three designeddiscussed is the process of making 'constant adjustments' in forecasts. This process involves modifying the results generated by the econometric model. For the first time we are publishing tables of the constant adjustments used in the current forecast. We explain in general why such adjustments are made and also explain the actual adjustments we have made for this forecast.
The second article of the series, to be published in our February 1983 edition, will describe the potential sources of error in forecasts. In particular it will describe the inevitable stochastic or random element involved in e tatistical attempts to quantify economic behaviour. As a completely new departure the article will report estimates of future errors based on stochastic simulations of the LBS. model and will provide statistical error bad for the main elements of the forecast.
The final article, to be published in our June 1983 edition, will contrast the measures of forecast error that e e obtain from the estimation process and our stochastic e imulationsp with the errors that we have actually made, as revealed by an examination of our forecasting 'track record'. It is hoped to draw, from this comparison, some e eneral conclusions about the scope and limits of econometric forecasting producers.     

Modeling and forecasting the time series of US fertility: age distribution,range, and ultimate level

This discussion of modeling focuses on the difficulties in longterm, time-series forecasting of US fertility. Four possibilities are suggested. One difficulty with the traditional approach of using high or low bounds on fertility and mortality is that forecast errors are perfectly correlated over time, which means there are no cancellation of errors over time. The shape of future fertility intervals first increases, then stabilizes, and then decreases instead of remaining stable. This occurs because the number of terms being averaged increases with horizontal length. Alho and Spencer attempted to reduce these errors in time-series. Other difficulties are the idiosyncratic behavior of age specific fertility over time, biological bounds for total fertility rates (TFR) of 16 and zero, the integration of knowledge about fertility behavior that narrows the bounds, the unlikelihood of some probability outcomes of stochastic models with a normally distributed error term, the small relative change in TFR between years, a US fertility cycle of about 40 years, unimportant extrapolation of past trends in child and infant mortality, and the unlikelihood of reversals in mortality and contraceptive use trends. Another problem is the unsuitability of longterm forecasts. New methods include a model which estimates a one parameter family of fertility schedules and then forecasts that single parameter. Another method is a logistic transformation to account for prior information on the bounds on fertility estimates; this method is similar to Bayesian methods for ARMA models developed by Monahan. Models include information on the ultimate level of fertility and assume that the equilibrium level is a stochastic process trending over time. The horizon forecast method is preferred unless the effects of the outliers are known. Estimates of fertility are presented for the equilibrium constrained and logistic transformed model. Forecasts of age specific fertility rates can be calculated from forecasts of the fertility index (a single time varying parameter). The model of fertility fits poorly at older ages but captures some of the wide swings in the historical pattern. Age variations are not accounted for very well. Longterm forecasts tell a great deal about the uncertainty of forecast errors. Estimates are too sensitive to model specification for accuracy and ignore the biological and socioeconomic context.     

Forecasts in times of crises

Financial crises pose unique challenges for forecast accuracy. Using the IMF’s Monitoring of Fund Arrangements (MONA) database, we conduct the most comprehensive evaluation of IMF forecasts to date for countries in times of crises. We examine 29 macroeconomic variables in terms of bias, efficiency, and information content to find that IMF forecasts add substantial informational value, as they consistently outperform naive forecast approaches. However, we also document that there is room for improvement: two-thirds of the key macroeconomic variables that we examine are forecast inefficiently, and six variables (growth of nominal GDP, public investment, private investment, the current account, net transfers, and government expenditures) exhibit significant forecast biases. The forecasts for low-income countries are the main drivers of forecast biases and inefficiency, perhaps reflecting larger shocks and lower data quality. When we decompose the forecast errors into their sources, we find that forecast errors for private consumption growth are the key contributor to GDP growth forecast errors. Similarly, forecast errors for non-interest expenditure growth and tax revenue growth are crucial determinants of the forecast errors in the growth of fiscal budgets. Forecast errors for balance of payments growth are influenced significantly by forecast errors in goods import growth. The results highlight which macroeconomic aggregates require further attention in future forecast models for countries in crises.     

Forecasting Errors: the Track Record

This Briefing Paper is the last of a series of three about forecasting. In this one we examine our forecasting record; it complements the February paper in which we analysed the properties of our forecasting model in terms of the error bands attached to the central forecast.
There are many ways of measuring forecasting errors, and in the first part of this Briefing Paper we describe briefing how we have tackled the problem. (A more detailed analysis can be found in the Appendix.) In Part II we report and comment upon the errors in our forecasts of annual growth rates and show how our forecasting performance has improved over the years. In Part III we focus on quarterly forecasts up to 8 quarters ahead, and compare our forecasting errors with measurement errors in the oficial statistics; with the estimation errors built into our forecast equations; and with the stochastic model errors we reported last February. A brief summary of the main conclusions is given below.     

Forecasting with Global Vector Autoregressive Models: a Bayesian Approach

This paper develops a Bayesian variant of global vector autoregressive (B‐GVAR) models to forecast an international set of macroeconomic and financial variables. We propose a set of hierarchical priors and compare the predictive performance of B‐GVAR models in terms of point and density forecasts for one‐quarter‐ahead and four‐quarter‐ahead forecast horizons. We find that forecasts can be improved by employing a global framework and hierarchical priors which induce country‐specific degrees of shrinkage on the coefficients of the GVAR model. Forecasts from various B‐GVAR specifications tend to outperform forecasts from a naive univariate model, a global model without shrinkage on the parameters and country‐specific vector autoregressions. Copyright © 2016 John Wiley & Sons, Ltd.     

Assumptions on Fertility in Stochastic Population Forecasts*

In recent years Statistics Netherlands has published several stochastic population forecasts. The degree of uncertainty of the future population is assessed on the basis of assumptions about the probability distribution of future fertility, mortality and migration. The assumptions on fertility are based on an analysis of historic forecasts of the total fertility rate (TFR), on time‐series models of observations of the TFR, and on expert knowledge. This latter argument‐based approach refers to the TFR distinguished by birth order. In the most recent Dutch forecast the 95% forecast interval of the total fertility rate in 2050 is assumed to range from 1.2 to 2.3 children per woman.     

Stochastic demographic forecasting

"This paper describes a particular approach to stochastic population forecasting, which is implemented for the U.S.A. through 2065. Statistical time series methods are combined with demographic models to produce plausible long run forecasts of vital rates, with probability distributions. The resulting mortality forecasts imply gains in future life expectancy that are roughly twice as large as those forecast by the Office of the Social Security Actuary....Resulting stochastic forecasts of the elderly population, elderly dependency ratios, and payroll tax rates for health, education and pensions are presented."     

Uncertainty and forecast precision

Macroeconomic forecasters typically report a single estimate per time period for each macroeconomic variable. But they rarely provide consumers of forecasts with information about the degree of confidence there is in the forecast or the likely range of dispersion of the actual outcome relative to the conditional forecast. Partly this is due to the non-linearity of the model and thus the cost of producing standard error bands. But there is also the problem that a mechanical stochastic simulation may misrepresent the degree of forecast uncertainty because forecasters use non-model information to produce a forecast which is more precise, at least for the immediate future, than the model alone. In this paper we propose a method for generating standard error bands which gives a truer reflection of the forecaster's uncertainty.     

Currency forecasting: an investigation of extrapolative judgement

This paper aims to explore the potential effects of trend type, noise and forecast horizon on experts' and novices' probabilistic forecasts. The subjects made forecasts over six time horizons from simulated monthly currency series based on a random walk, with zero, constant and stochastic drift, at two noise levels. The difference between the Mean Absolute Probability Score of each participant and an AR(1) model was used to evaluate performance. The results showed that the experts performed better than the novices, although worse than the model except in the case of zero drift series. No clear expertise effects occurred over horizons, albeit subjects' performance relative to the model improved as the horizon increased. Possible explanations are offered and some suggestions for future research are outlined.     

Response to updated mortality forecasts in life cycle saving and labor supply

Historical evidence shows that demographic forecasts, including mortality forecasts, have often been grossly in error. One consequence of this is that forecasts are updated frequently. How should individuals or institutions react to updates, given that these are likewise expected to be uncertain? We discuss this problem in the context of a life cycle saving and labor supply problem, in which a cohort of workers decides how much to work and how much to save for mutual pensions. Mortality is stochastic and point forecasts are updated regularly. A Markovian approximation for the predictive distribution of mortality is derived. This renders the model computationally tractable, and allows us to compare a theoretically optimal rational expectations solution to a strategy in which the cohort merely updates the life cycle plan to match each updated mortality forecast. The implications of the analyses for overlapping generations modeling of pension systems are pointed out.     

Forecasting sales response for multiple time horizons and temporally aggregated data: A comparison of constant and stochastic coefficient models

Past research on time-varying sales-response models emphasized the application of different estimation techniques in examining variation in advertising effectiveness over time. This study focuses on comparing sales forecasts using constant and stochastic coefficients sales-response models. Selected constant and stochastic coefficient models are applied to six sets of bimonthly and one set of annual advertising and sales data to assess forecasting accuracy for time horizons of various lengths. Results show improved forecasting accuracy for a first-order autoregressive stochastic coefficient model, particularly in short-run forecasting applications.     

How quickly do forecasters incorporate news? Evidence from cross‐country surveys

Using forecasts from Consensus Economics Inc., we provide evidence on the efficiency of real GDP growth forecasts by testing whether forecast revisions are uncorrelated. As the forecast data used are multi‐dimensional—18 countries, 24 monthly forecasts for the current and the following year and 16 target years—the panel estimation takes into account the complex structure of the variance–covariance matrix due to propagation of shocks across countries and economic linkages among them. Efficiency is rejected for all 18 countries: forecast revisions show a high degree of serial correlation. We then develop a framework for characterizing the nature of the inefficiency in forecasts. For a smaller set of countries, the G‐7, we estimate a VAR model on forecast revisions. The degree of inefficiency, as manifested in the serial correlation of forecast revisions, tends to be smaller in forecasts of the USA than in forecasts for European countries. Our framework also shows that one of the sources of the inefficiency in a country's forecasts is resistance to utilizing foreign news. Thus the quality of forecasts for many of these countries can be significantly improved if forecasters pay more attention to news originating from outside their respective countries. This is particularly the case for Canadian and French forecasts, which would gain by paying greater attention than they do to news from the USA and Germany, respectively. Copyright © 2006 John Wiley & Sons, Ltd.