How accurate are government forecasts of economic fundamentals? The case of Taiwan

A government’s ability to forecast key economic fundamentals accurately can affect business confidence, consumer sentiment, and foreign direct investment, among others. A government forecast based on an econometric model is replicable, whereas one that is not fully based on an econometric model is non-replicable. Governments typically provide non-replicable forecasts (or expert forecasts) of economic fundamentals, such as the inflation rate and real GDP growth rate.In this paper, we develop a methodology for evaluating non-replicable forecasts. We argue that in order to do so, one needs to retrieve from the non-replicable forecast its replicable component, and that it is the difference in accuracy between these two that matters. An empirical example to forecast economic fundamentals for Taiwan shows the relevance of the proposed methodological approach. Our main finding is that the undocumented knowledge of the Taiwanese government reduces forecast errors substantially.     

EVALUATING MACROECONOMIC FORECASTS: A CONCISE REVIEW OF SOME RECENT DEVELOPMENTS

Macroeconomic forecasts are frequently produced, widely published, intensively discussed, and comprehensively used. The formal evaluation of such forecasts has a long research history. Recently, a new angle to the evaluation of forecasts has been addressed, and in this review we analyze some recent developments from that perspective. The literature on forecast evaluation predominantly assumes that macroeconomic forecasts are generated from econometric models. In practice, however, most macroeconomic forecasts, such as those from the IMF, World Bank, OECD, Federal Reserve Board, Federal Open Market Committee (FOMC), and the ECB, are typically based on econometric model forecasts jointly with human intuition. This seemingly inevitable combination renders most of these forecasts biased and, as such, their evaluation becomes nonstandard. In this review, we consider the evaluation of two forecasts in which: (i) the two forecasts are generated from two distinct econometric models; (ii) one forecast is generated from an econometric model and the other is obtained as a combination of a model and intuition; and (iii) the two forecasts are generated from two distinct (but unknown) combinations of different models and intuition. It is shown that alternative tools are needed to compare and evaluate the forecasts in each of these three situations. These alternative techniques are illustrated by comparing the forecasts from the (econometric) Staff of the Federal Reserve Board and the FOMC on inflation, unemployment, and real GDP growth. It is shown that the FOMC does not forecast significantly better than the Staff, and that the intuition of the FOMC does not add significantly in forecasting the actual values of the economic fundamentals. This would seem to belie the purported expertise of the FOMC.     

Expert opinion versus expertise in forecasting

Expert opinion is an opinion given by an expert, and it can have significant value in forecasting key policy variables in economics and finance. Expert forecasts can either be expert opinions, or forecasts based on an econometric model. An expert forecast that is based on an econometric model is replicable, and can be defined as a replicable expert forecast (REF), whereas an expert opinion that is not based on an econometric model can be defined as a non-replicable expert forecast (Non-REF). Both REF and Non-REF may be made available by an expert regarding a policy variable of interest. In this paper, we develop a model to generate REF, and compare REF with Non-REF. A method is presented to compare REF and Non-REF using efficient estimation methods, and a direct test of expertise on expert opinion is given. The latter serves the purpose of investigating whether expert adjustment improves the model-based forecasts. Illustrations for forecasting pharmaceutical stock keeping unit (SKUs), where the econometric model is of (variations of) the autoregressive integrated moving average model (ARIMA) type, show the relevance of the new methodology proposed in the paper. In particular, experts possess significant expertise, and expert forecasts are significant in explaining actual sales.     

Econometric Models and Economic Forecasting

This Briefing Paper is thejirst ofa series of three designeddiscussed is the process of making 'constant adjustments' in forecasts. This process involves modifying the results generated by the econometric model. For the first time we are publishing tables of the constant adjustments used in the current forecast. We explain in general why such adjustments are made and also explain the actual adjustments we have made for this forecast.
The second article of the series, to be published in our February 1983 edition, will describe the potential sources of error in forecasts. In particular it will describe the inevitable stochastic or random element involved in e tatistical attempts to quantify economic behaviour. As a completely new departure the article will report estimates of future errors based on stochastic simulations of the LBS. model and will provide statistical error bad for the main elements of the forecast.
The final article, to be published in our June 1983 edition, will contrast the measures of forecast error that e e obtain from the estimation process and our stochastic e imulationsp with the errors that we have actually made, as revealed by an examination of our forecasting 'track record'. It is hoped to draw, from this comparison, some e eneral conclusions about the scope and limits of econometric forecasting producers.     

The tourism forecasting competition

We evaluate the performances of various methods for forecasting tourism data. The data used include 366 monthly series, 427 quarterly series and 518 annual series, all supplied to us by either tourism bodies or academics who had used them in previous tourism forecasting studies. The forecasting methods implemented in the competition are univariate and multivariate time series approaches, and econometric models. This forecasting competition differs from previous competitions in several ways: (i) we concentrate on tourism data only; (ii) we include approaches with explanatory variables; (iii) we evaluate the forecast interval coverage as well as the point forecast accuracy; (iv) we observe the effect of temporal aggregation on the forecasting accuracy; and (v) we consider the mean absolute scaled error as an alternative forecasting accuracy measure. We find that pure time series approaches provide more accurate forecasts for tourism data than models with explanatory variables. For seasonal data we implement three fully automated pure time series algorithms that generate accurate point forecasts, and two of these also produce forecast coverage probabilities which are satisfactorily close to the nominal rates. For annual data we find that Naïve forecasts are hard to beat.     

Forecasting television ratings

Despite the state of flux in media today, television remains the dominant player globally for advertising spending. Since television advertising time is purchased on the basis of projected future ratings, and ad costs have skyrocketed, there is increasingly pressure to forecast television ratings accurately. The forecasting methods that have been used in the past are not generally very reliable, and many have not been validated; also, even more distressingly, none have been tested in today’s multichannel environment. In this study we compare eight different forecasting models, ranging from a naïve empirical method to a state-of-the-art Bayesian model-averaging method. Our data come from a recent time period, namely 2004-2008, in a market with over 70 channels, making the data more typical of today’s viewing environment. The simple models that are commonly used in industry do not forecast as well as any econometric models. Furthermore, time series methods are not applicable, as many programs are broadcast only once. However, we find that a relatively straightforward random effects regression model often performs as well as more sophisticated Bayesian models in out-of-sample forecasting. Finally, we demonstrate that making improvements in ratings forecasts could save the television industry between $250 and $586 million per year.     

Forecasting temperature to price CME temperature derivatives

This paper seeks to forecast temperatures in US cities in order to price temperature derivatives on the Chicago Mercantile Exchange (CME). The CME defines the average daily temperature underlying its contracts as the average of the maximum and minimum daily temperatures, yet all published work on temperature forecasting for pricing purposes has ignored this peculiar definition of the average and sought to model the average temperature directly. This paper is the first to look at the average temperature forecasting problem as an analysis of extreme values. The theory of extreme values guides model selection for temperature maxima and minima, and a forecast distribution for the CME’s daily average temperature is found through convolution. While univariate time series AR-GARCH and regression models generally yield superior point forecasts of temperatures, our extreme-value-based model consistently outperforms these models in density forecasting, the most important risk management tool.     

Open economy forecasting with a DSGE-VAR: Head to head with the RBNZ published forecasts

We construct a DSGE-VAR model for competing head to head with the long history of published forecasts of the Reserve Bank of New Zealand. We also construct a Bayesian VAR model with a Minnesota prior for forecast comparison. The DSGE-VAR model combines a structural DSGE model with a statistical VAR model based on the in-sample fit over the majority of New Zealand’s inflation-targeting period. We evaluate the real-time out-of-sample forecasting performance of the DSGE-VAR model, and show that the forecasts from the DSGE-VAR are competitive with the Reserve Bank of New Zealand’s published, judgmentally-adjusted forecasts. The Bayesian VAR model with a Minnesota prior also provides a competitive forecasting performance, and generally, with a few exceptions, out-performs both the DSGE-VAR and the Reserve Bank’s own forecasts.     

Forecasting economic activity with mixed frequency BVARs

Mixed frequency Bayesian vector autoregressions (MF-BVARs) allow forecasters to incorporate large numbers of time series that are observed at different intervals into forecasts of economic activity. This paper benchmarks the performances of MF-BVARs for forecasting U.S. real gross domestic product growth against surveys of professional forecasters and documents the influences of certain specification choices. We find that a medium–large MF-BVAR provides an attractive alternative to surveys at the medium-term forecast horizons that are of interest to central bankers and private sector analysts. Furthermore, we demonstrate that certain specification choices influence its performance strongly, such as model size, prior selection mechanisms, and modeling in levels versus growth rates.     

A comparison of the forecasting performance of WEFA and ARIMA time series methods

This paper examines the forecasting performance of the Wharton model (MARK III) over the period 1973 through 1975 and compares it with that of ARIMA models' performance over the same period. Despite strong intimation in the literature to the contrary, we find that this econometric model, at least, exhibits greater accuracy in every respect relative to ARIMA methods, in terms of its forecasts cum constant adjustments. When constant adjustments are disallowed then its forecasts are still more accurate than ARIMA forecasts over a 4- and 8-quarter forecasting horizon, but less accurate over a 1-quarter horizon. The comparison was carried out over twenty three macrovariables, under a slight handicap for the Wharton Model, in that the latter's parameters were estimated over a sample ending in 1969.3 while the ARIMA models were reidentified and reestimated as of the quarter immediately preceding the forecast.     

Predicting recessions with a composite real-time dynamic probit model

In this paper we propose a composite indicator for real-time recession forecasting based on alternative dynamic probit models. For this purpose, we use a large set of monthly macroeconomic and financial leading indicators from the German and US economies. Alternative dynamic probit regressions are specified through automated general-to-specific and specific-to-general lag selection procedures on the basis of slightly different initial sets. The resulting recession probability forecasts are then combined in order to decrease the volatility of the forecast errors and increase their forecasting accuracy. This procedure features not only good in-sample forecast statistics, but also good out-of-sample performances, as is illustrated using a real-time evaluation exercise.     

Forecasting macroeconomic time series: LASSO-based approaches and their forecast combinations with dynamic factor models

In a data-rich environment, forecasting economic variables amounts to extracting and organizing useful information from a large number of predictors. So far, the dynamic factor model and its variants have been the most successful models for such exercises. In this paper, we investigate a category of LASSO-based approaches and evaluate their predictive abilities for forecasting twenty important macroeconomic variables. These alternative models can handle hundreds of data series simultaneously, and extract useful information for forecasting. We also show, both analytically and empirically, that combing forecasts from LASSO-based models with those from dynamic factor models can reduce the mean square forecast error (MSFE) further. Our three main findings can be summarized as follows. First, for most of the variables under investigation, all of the LASSO-based models outperform dynamic factor models in the out-of-sample forecast evaluations. Second, by extracting information and formulating predictors at economically meaningful block levels, the new methods greatly enhance the interpretability of the models. Third, once forecasts from a LASSO-based approach are combined with those from a dynamic factor model by forecast combination techniques, the combined forecasts are significantly better than either dynamic factor model forecasts or the naïve random walk benchmark.     

Forecasting with an econometric model: Some recent results using the national institute model

It is widely believed that the large econometric models cannot be used for forecasting without considerable intervention on the part of the forecaster. In this paper we challenge this view by reproducing a number of recent forecasts published by the National Institute but without the ad hoc interventions used at the time. We show that in no case would the forecast, produced by the model used mechanically, have been radically different from that actually published. Further, in an ex-post comparison against actual out-turns, the mechanical model forecast is not obviously dominated by the published version.     

A Bradley-Terry type model for forecasting tennis match results

The paper introduces a model for forecasting match results for the top tier of men’s professional tennis, the ATP tour. Employing a Bradley-Terry type model, and utilising the data available on players’ past results and the surface of the contest, we predict match winners for the coming week’s matches, having updated the model parameters to take the previous week’s results into account. We compare the model to two logit models: one using official rankings and another using the official ranking points of the two competing players. Our model provides superior forecasts according to each of five criteria measuring the predictive performance, two of which relate to betting returns.     

Macroeconomic forecast accuracy in a data‐rich environment

The performance of six classes of models in forecasting different types of economic series is evaluated in an extensive pseudo out‐of‐sample exercise. One of these forecasting models, regularized data‐rich model averaging (RDRMA), is new in the literature. The findings can be summarized in four points. First, RDRMA is difficult to beat in general and generates the best forecasts for real variables. This performance is attributed to the combination of regularization and model averaging, and it confirms that a smart handling of large data sets can lead to substantial improvements over univariate approaches. Second, the ARMA(1,1) model emerges as the best to forecast inflation changes in the short run, while RDRMA dominates at longer horizons. Third, the returns on the S&P 500 index are predictable by RDRMA at short horizons. Finally, the forecast accuracy and the optimal structure of the forecasting equations are quite unstable over time.     

The performance of alternative forecasting methods for SETAR models

We compare a number of methods that have been proposed in the literature for obtaining h-step ahead minimum mean square error forecasts for self-exciting threshold autoregressive (SETAR) models. These forecasts are compared to those from an AR model. The comparison of forecasting methods is made using Monte Carlo simulation. The Monte-Carlo method of calculating SETAR forecasts is generally at least as good as that of the other methods we consider. An exception is when the disturbances in the SETAR model come from a highly asymmetric distribution, when a Bootstrap method is to be preferred.An empirical application calculates multi-period forecasts from a SETAR model of US gross national product using a number of the forecasting methods. We find that whether there are improvements in forecast performance relative to a linear AR model depends on the historical epoch we select, and whether forecasts are evaluated conditional on the regime the process was in at the time the forecast was made.     

Real-time macroeconomic forecasting with leading indicators: An empirical comparison

This paper demonstrates that the Conference Board’s Composite Leading Index (CLI) has significant real-time predictive ability for Industrial Production (IP) growth rates at horizons from one to six months ahead over the period 1989-2009. A popular but unrealistic analysis, which combines real-time data for CLI and final vintage data for IP as predictor variables, obscures the actual predictive content of the CLI, in the sense that in that case, the improvements in forecast accuracy relative to a univariate AR model are not significant. The CLI appears to be less useful for forecasting growth rates of the Conference Board’s Composite Coincident Index (CCI) in real time, as a univariate AR model performs better. This result is mostly due to its disappointing performance during the first five years of the forecast period. The CLI may not be the best way of exploiting the information contained in the underlying individual leading indicator variables. The use of principal components instead of CLI leads to more accurate real-time forecasts for both IP and CCI growth rates.     

Forecasting economic time series using flexible versus fixed specification and linear versus nonlinear econometric models

Nine macroeconomic variables are forecast in a real-time scenario using a variety of flexible specification, fixed specification, linear, and nonlinear econometric models. All models are allowed to evolve through time, and our analysis focuses on model selection and performance. In the context of real-time forecasts, flexible specification models (including linear autoregressive models with exogenous variables and nonlinear artificial neural networks) appear to offer a useful and viable alternative to less flexible fixed specification linear models for a subset of the economic variables which we examine, particularly at forecast horizons greater than 1-step ahead. We speculate that one reason for this result is that the economy is evolving (rather slowly) over time. This feature cannot easily be captured by fixed specification linear models, however, and manifests itself in the form of evolving coefficient estimates. We also provide additional evidence supporting the claim that models which ‘win’ based on one model selection criterion (say a squared error measure) do not necessarily win when an alternative selection criterion is used (say a confusion rate measure), thus highlighting the importance of the particular cost function which is used by forecasters and ‘end-users’ to evaluate their models. A wide variety of different model selection criteria and statistical tests are used to illustrate our findings.     

Assessing the accuracy of electricity production forecasts in developing countries

This study assesses the accuracy of time series econometric methods for forecasting electricity production in developing countries. An analysis of the historical time series for 106 developing countries over the period 1960–2012 demonstrates that econometric forecasts are highly accurate for the majority of these countries. These forecasts have much smaller errors than the predictions of simple heuristic models, which assume that electricity production grows at an exogenous rate or is proportional to the real GDP growth. However, the quality of the forecasts diminishes for the countries and regions, where rapid economic and structural transformation makes it difficult to establish stable historical production trends.     

Benchmarking robustness of load forecasting models under data integrity attacks

As the internet’s footprint continues to expand, cybersecurity is becoming a major concern for both governments and the private sector. One such cybersecurity issue relates to data integrity attacks. This paper focuses on the power industry, where the forecasting processes rely heavily on the quality of the data. Data integrity attacks are expected to harm the performances of forecasting systems, which will have a major impact on both the financial bottom line of power companies and the resilience of power grids. This paper reveals the effect of data integrity attacks on the accuracy of four representative load forecasting models (multiple linear regression, support vector regression, artificial neural networks, and fuzzy interaction regression). We begin by simulating some data integrity attacks through the random injection of some multipliers that follow a normal or uniform distribution into the load series. Then, the four aforementioned load forecasting models are used to generate one-year-ahead ex post point forecasts in order to provide a comparison of their forecast errors. The results show that the support vector regression model is most robust, followed closely by the multiple linear regression model, while the fuzzy interaction regression model is the least robust of the four. Nevertheless, all four models fail to provide satisfying forecasts when the scale of the data integrity attacks becomes large. This presents a serious challenge to both load forecasters and the broader forecasting community: the generation of accurate forecasts under data integrity attacks. We construct our case study using the publicly-available data from Global Energy Forecasting Competition 2012. At the end, we also offer an overview of potential research topics for future studies.